pg.ddx.io pgsql-hackers@postgresql.org mailing list archive help / color / mirror / Atom feed From: Michael Paquier <michael@paquier.xyz>
To: gkokolatos@pm.me
Cc: Magnus Hagander <magnus@hagander.net>
Cc: Dilip Kumar <dilipbalaut@gmail.com>
Cc: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Subject: Re: Teach pg_receivewal to use lz4 compression
Date: Fri, 29 Oct 2021 20:38:33 +0900
Message-ID: <YXvdOb6yo/xxfVce@paquier.xyz> (raw )
In-Reply-To: <8CDVNs7db0aITUJjQ8TkK6pX_mO3vTVdpr0niTKodjcNw8DnUIK1fUWHeTYf9oYQP3YfizJT5XevtKdp7xJ04LU_yRI7MY8pPutua3eQAUo=@pm.me >
References: <CAFiTN-urjKKCwu+YA5VYLG40_RLst-B=BxZNKXGpuTgh8hoFtA@mail.gmail.com >
<PZnyf3PE6s8jQBf-CTQjAvjSRvw8dWWxYrLWR2NdqTrhm7wHfrQ2gUgwdlmGevfIxindJ3KooyuWQ5MRU9NTyMRiNMfRBkyVnqkyPH7nP_k=@pm.me >
<YOe5LouYuxXOrL0H@paquier.xyz >
<rYyZ3Fj9qayyY9-egNsV_kkLbL_BSWcOEdi3Mb6M9eQRTkcA2jrqFEHglLUEYnzWR_wttCqn7VI94MZ2p7mwNje51lHTvWYnJ1jHdOceen4=@pm.me >
<YUGW2mwx0eNnffoR@paquier.xyz >
<6GriwrD-LWnn-8G-AdhMyJQh_Dk9uU22mfJS5C_3knfhbL9I35i9aGnBxiY4tvePz-hCdZ45Le0zUSaslCoPEBf3h8zLoB3nQcns1Z5MHrM=@pm.me >
<YURGPe8FLJReiabx@paquier.xyz >
<2sv1K4rAd0hZl65M4_N_yavaqHXnYQJRZOJFK8zMYiHRZsR1BJvXzprxTqi3IzangO5hyuZC5ES-VulHvIU-8B7zJGr15wgF4Dlfbfp2QRk=@pm.me >
<YUWEqIe4Bx6Q4lcm@paquier.xyz >
<8CDVNs7db0aITUJjQ8TkK6pX_mO3vTVdpr0niTKodjcNw8DnUIK1fUWHeTYf9oYQP3YfizJT5XevtKdp7xJ04LU_yRI7MY8pPutua3eQAUo=@pm.me >
On Fri, Oct 29, 2021 at 09:45:41AM +0000, gkokolatos@pm.me wrote:
> On Saturday, September 18th, 2021 at 8:18 AM, Michael Paquier <michael@paquier.xyz> wrote:
>> We don't really care about contentSize as long as a segment is not
>> completed. Rather than filling contentSize all the time we write
>> something, we'd better update frameInfo once the segment is
>> completed and closed. That would also take take of the error as this
>> is not checked if contentSize is 0. It seems to me that we should
>> fill in the information when doing a CLOSE_NORMAL.
>
> Thank you for the comment. I think that the opposite should be done. At the time
> that the file is closed, the header is already written to disk. We have no way
> to know that is not. If we need to go back to refill the information, we will
> have to ask for the API to produce a new header. There is little guarantee that
> the header size will be the same and as a consequence we will have to shift
> the actual data around.
Why would the header size change between the moment the segment is
begun and it is finished? We could store it in memory and write it
again when the segment is closed instead, even if it means to fseek()
back to the beginning of the file once the segment is completed.
Storing WalSegSz from the moment a segment is opened makes the code
weaker to SIGINTs and the kind, so this does not fix the problem I
mentioned previously :/
> In the attached, the header is rewritten only when closing an incomplete
> segment. For all intents and purposes that segment is not usable. However there
> might be custom scripts that might want to attempt to parse even an otherwise
> unusable file.
>
> A different and easier approach would be to simply prepare the LZ4 context for
> future actions and simply ignore the file.
I am not sure what you mean by "ignore" here. Do you mean to store 0
in contentSize when opening the segment and rewriting again the header
once the segment is completed?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
view thread (55+ messages) latest in thread
Message-ID: <YXvdOb6yo/xxfVce@paquier.xyz>
Permalink: ../YXvdOb6yo%2FxxfVce@paquier.xyz/
Also on: postgresql.org/message-id/YXvdOb6yo/xxfVce@paquier.xyz
copy link · copy postgr.es
reply Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: michael@paquier.xyz, gkokolatos@pm.me, magnus@hagander.net, dilipbalaut@gmail.com, pgsql-hackers@lists.postgresql.org
Subject: Re: Teach pg_receivewal to use lz4 compression
In-Reply-To: <YXvdOb6yo/xxfVce@paquier.xyz>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox