Subject: Re: PATCH: logical_work_mem and logical streaming of large
 in-progress transactions
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>, Greg Stark <stark@mit.edu>
References: <688b0b7f-2f6c-d827-c27b-216a8e3ea700@2ndquadrant.com>
 <ec3fbad0-06f6-08cb-7f0e-edd3fb0c2785@2ndquadrant.com>
 <CAM-w4HNWQMnE8j9=Fa1QdwfKYs=XQanx03PL23yULbh1hPYSGQ@mail.gmail.com>
 <5d00346a-b565-7fde-c6da-35f39c72936a@2ndquadrant.com>
 <d72caaf1-a2da-081c-57ea-d30e5f47fd1e@2ndquadrant.com>
Cc: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
From: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
Organization: 2ndQuadrant
Message-ID: <4d7ed7b8-df43-9b79-b04e-ad4c43ef519c@2ndquadrant.com>
Date: Thu, 18 Jan 2018 08:24:06 -0500
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:52.0)
 Gecko/20100101 Thunderbird/52.5.2
MIME-Version: 1.0
In-Reply-To: <d72caaf1-a2da-081c-57ea-d30e5f47fd1e@2ndquadrant.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk

On 1/12/18 23:19, Tomas Vondra wrote:
> Wouldn't the 'toplevel_by_lsn' be suitable for this? Subtransactions
> don't really commit independently, but as part of the toplevel xact. And
> that list is ordered by LSN, which is pretty much exactly the order in
> which we see the transactions.

Yes indeed.  There is even ReorderBufferGetOldestTXN().

> Another somewhat non-intuitive detail is that because ReorderBuffer
> switched to Generation allocator for changes (which usually represent
> 99% of the memory used during decoding), it does not reuse memory the
> way AllocSet does. Actually, it does not reuse memory at all, aiming to
> eventually give the memory back to libc (which AllocSet can't do).
> 
> Because of this evicting the youngest transactions seems like a quite
> bad idea, because those chunks will not be reused and there may be other
> chunks on the blocks, preventing their release.

Right.  But this raises the question whether we are doing the memory
accounting on the right level.  If we are doing all this tracking based
on ReorderBufferChanges, but then serializing changes possibly doesn't
actually free any memory in the operating system, that's no good.  Can
we get some usage statistics out of the memory context?  It seems like
we need to keep serializing transactions until we actually see the
memory context size drop.

-- 
Peter Eisentraut              http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services