Subject: Re: PATCH: logical_work_mem and logical streaming of large
 in-progress transactions
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
References: <688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com>
 <bbe1333f-17ba-1156-0713-7cc0bfe2d842@2ndquadrant.com>
Message-ID: <ca1e792e-6b59-bb9b-10ea-bf1d30c52548@2ndquadrant.com>
Date: Wed, 3 Jan 2018 21:13:29 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.4.0
MIME-Version: 1.0
In-Reply-To: <bbe1333f-17ba-1156-0713-7cc0bfe2d842@2ndquadrant.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 01/03/2018 09:06 PM, Tomas Vondra wrote:
> Hi,
> 
> attached is v4 of the patch series, with a couple of changes:
> 
> 1) Fixes a bunch of bugs I discovered during stress testing.
> 
> I'm not going to go into details, but the main fixes are related to
> properly updating progress from the worker, and not streaming when
> creating the logical replication slot.
> 
> 2) Introduces columns into pg_stat_replication.
> 
> The new columns track various kinds of statistics (number of xacts,
> bytes, ...) about spill-to-disk/streaming. This will be useful when
> tuning the GUC memory limit.
> 
> 3) Two temporary bugfixes that make the patch series work.
> 

Forgot to mention that the v4 also extends the CREATE SUBSCRIPTION to
allow customizing the streaming and memory limit. So you can do

    CREATE SUBSCRIPTION ... WITH (streaming=on, work_mem=1024)

and this subscription will allow streaming, and the logica_work_mem (on
provider) will be set to 1MB.

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services