Subject: Re: PATCH: logical_work_mem and logical streaming of large
 in-progress transactions
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>, Erik Rijkers <er@xs4all.nl>
References: <688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com>
 <bbe1333f-17ba-1156-0713-7cc0bfe2d842@2ndquadrant.com>
 <bc83b847-2fc7-6561-4ef1-a3c9d154cc31@2ndquadrant.com>
 <47223121-222d-8e04-4928-a28ecdc5327d@2ndquadrant.com>
 <CAD21AoBa2mhbhZW6hnDRKs1ggycJOBE69eHk3KgCJ5Cwj9-ZkA@mail.gmail.com>
 <b9b306a3-c0a2-7eb9-92eb-a3ca111bd1bc@2ndquadrant.com>
 <4762f6bd-b147-cd06-f286-22f54790b77d@2ndquadrant.com>
 <cb2530b102afad854e051cca44577c6a@xs4all.nl>
 <d807dc44-55f9-5c23-97e5-a5881f4135e2@2ndquadrant.com>
Cc: Masahiko Sawada <sawada.mshk@gmail.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Organization: 2ndQuadrant
Message-ID: <5eddec05-404f-3fc4-bafd-15e9e4787705@2ndquadrant.com>
Date: Fri, 9 Mar 2018 11:07:55 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0)
 Gecko/20100101 Thunderbird/52.6.0
MIME-Version: 1.0
In-Reply-To: <d807dc44-55f9-5c23-97e5-a5881f4135e2@2ndquadrant.com>
Content-Type: multipart/mixed;
 boundary="------------0317EC59C834BCB2CEB17D29"
Content-Language: en-US
Precedence: bulk

This is a multi-part message in MIME format.
--------------0317EC59C834BCB2CEB17D29
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: 8bit

I think this patch is not going to be ready for PG11.

- It depends on some work in the thread "logical decoding of two-phase
transactions", which is still in progress.

- Various details in the logical_work_mem patch (0001) are unresolved.

- This being partially a performance feature, we haven't seen any
performance tests (e.g., which settings result in which latencies under
which workloads).

That said, the feature seems useful and desirable, and the
implementation makes sense.  There are documentation and tests.  But
there is a significant amount of design and coding work still necessary.

Attached is a fixup patch that I needed to make it compile.

The last two patches in your series (0008, 0009) are labeled as bug
fixes.  Would you like to argue that they should be applied
independently of the rest of the feature?

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

--------------0317EC59C834BCB2CEB17D29
Content-Type: text/plain; charset=UTF-8; x-mac-type="0"; x-mac-creator="0";
 name="0001-fixup-Track-statistics-for-streaming-spilling.patch"
Content-Transfer-Encoding: quoted-printable
Content-Disposition: attachment;
 filename*0="0001-fixup-Track-statistics-for-streaming-spilling.patch"

=46rom 7ac3c2b16f9976c75a0feea3131a36bdf50da2f8 Mon Sep 17 00:00:00 2001
From: Peter Eisentraut <peter_e@gmx.net>
Date: Fri, 9 Mar 2018 10:50:33 -0500
Subject: [PATCH] fixup! Track statistics for streaming/spilling

---
 src/include/catalog/pg_proc.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/src/include/catalog/pg_proc.h b/src/include/catalog/pg_proc.=
h
index 9d6c88f0c1..f1cea24379 100644
--- a/src/include/catalog/pg_proc.h
+++ b/src/include/catalog/pg_proc.h
@@ -2901,7 +2901,7 @@ DATA(insert OID =3D 2022 (  pg_stat_get_activity			=
PGNSP PGUID 12 1 100 0 0 f f f
 DESCR("statistics: information about currently active backends");
 DATA(insert OID =3D 3318 (  pg_stat_get_progress_info			  PGNSP PGUID 12=
 1 100 0 0 f f f t t s r 1 0 2249 "25" "{25,23,26,26,20,20,20,20,20,20,20=
,20,20,20}" "{i,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{cmdtype,pid,datid,relid,par=
am1,param2,param3,param4,param5,param6,param7,param8,param9,param10}" _nu=
ll_ _null_ pg_stat_get_progress_info _null_ _null_ _null_ ));
 DESCR("statistics: information about progress of backends running mainte=
nance command");
-DATA(insert OID =3D 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 =
0 0 f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,=
23,25,20,20,20,20,20,20}" "{o,o,o,o,o,o,o,o,o,o,o}" "{pid,state,sent_lsn,=
write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag,sync_priori=
ty,sync_state,spill_txns,spill_count,spill_bytes,stream_txns,stream_count=
,stream_bytes}" _null_ _null_ pg_stat_get_wal_senders _null_ _null_ _null=
_ ));
+DATA(insert OID =3D 3099 (  pg_stat_get_wal_senders	PGNSP PGUID 12 1 10 =
0 0 f f f f t s r 0 0 2249 "" "{23,25,3220,3220,3220,3220,1186,1186,1186,=
23,25,20,20,20,20,20,20}" "{o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,sta=
te,sent_lsn,write_lsn,flush_lsn,replay_lsn,write_lag,flush_lag,replay_lag=
,sync_priority,sync_state,spill_txns,spill_count,spill_bytes,stream_txns,=
stream_count,stream_bytes}" _null_ _null_ pg_stat_get_wal_senders _null_ =
_null_ _null_ ));
 DESCR("statistics: information about currently active replication");
 DATA(insert OID =3D 3317 (  pg_stat_get_wal_receiver	PGNSP PGUID 12 1 0 =
0 0 f f f f f s r 0 0 2249 "" "{23,25,3220,23,3220,23,1184,1184,3220,1184=
,25,25}" "{o,o,o,o,o,o,o,o,o,o,o,o}" "{pid,status,receive_start_lsn,recei=
ve_start_tli,received_lsn,received_tli,last_msg_send_time,last_msg_receip=
t_time,latest_end_lsn,latest_end_time,slot_name,conninfo}" _null_ _null_ =
pg_stat_get_wal_receiver _null_ _null_ _null_ ));
 DESCR("statistics: information about WAL receiver");
--=20
2.16.2


--------------0317EC59C834BCB2CEB17D29--