Content-Type: multipart/alternative;
 boundary="------------sNv4qgqDXOBjcwhi49XES5eZ"
Message-ID: <69ae171b-87b5-463e-8f60-671a083d2e63@dunslane.net>
Date: Mon, 6 Jan 2025 11:22:53 -0500
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: Windows pg_basebackup unable to create >2GB pg_wal.tar tarballs
 ("could not close file: Invalid argument" when creating pg_wal.tar of size ~
 2^31 bytes)
To: Thomas Munro <thomas.munro@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>,
 Jakub Wartak <jakub.wartak@enterprisedb.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
 Dilip Kumar <dilip.kumar@enterprisedb.com>
References: <CAKZiRmyM4YnokK6Oenw5JKwAQ3rhP0YTz2T-tiw5dAQjGRXE3Q@mail.gmail.com>
 <CA+hUKG+oo_g4-dcsZkGsmKHad4sOmkMOBctDPV1CEGWPFeeTwQ@mail.gmail.com>
 <CAKZiRmxL_0cKA1W1v8Qxs+tRy44NacL9gnxVaa-APzPvBF40Yw@mail.gmail.com>
 <CA+hUKGLpkcNRkrwMCXuJu1jc5-38bvN-yEq5B0FpSV91U9ZwMQ@mail.gmail.com>
 <CA+TgmoYVy3Z7F2bfBNmRch2gx1cL46-PjjBUVbc8EjdAvmfoAw@mail.gmail.com>
 <CA+hUKG++NtrryUcSE5SwUi98SG+X9Q+w=mYo0Rx-DgSMPALr2A@mail.gmail.com>
 <CA+hUKG+6ZPD_X5ADMwX2uUtXqe_wv8+KQ5xFeAR2zbcodjNZvw@mail.gmail.com>
 <CA+TgmobCEWj2XDDrqbw2xqu91FStrusHaXXJtczLfpS7Tq55mQ@mail.gmail.com>
 <CA+hUKGK0+5PAfW9q_4z9XEiVfjgcCi2JgcNBmiRxTPCpxdFMdQ@mail.gmail.com>
 <CAEbAQ-VgeghebE+xgJFOF9dppYX0OS-7N_vq=u7fuhz5oBWyyQ@mail.gmail.com>
From: Andrew Dunstan <andrew@dunslane.net>
Content-Language: en-US
Autocrypt: addr=andrew@dunslane.net; keydata=
 xsBNBE7KWFkBCAClridxur2AIc7eW2AR7izbfp3EnNefie2HbLF0izW5Ik5UjX2HBXBx4syI
 gY6b0ugohXrr274+baoAlvSbq6cAoQuEVrk5IZFzt20b1Xkx65FwGSEj526yiKLocqkJceSq
 Xr9xcA5SGY+FZv441chh5SU92v4q6z+6LPpoHOh97ptAVXZYNTtU0LevyvD5lja0TzbvJm6C
 eFXitJfnm1pLEr0DGJCR/iUOl/N62Kh4855zZC7NHIjQHPOvV5Stz/l5ilDhvGVk+xkXFPys
 SjZoUr1rXhYLpiyi5sR0X9FHXT0KnGuz1F5ERO7ZTLSSQ6fJwPj6gOk9K+vvoKvoeql5ABEB
 AAHNJEFuZHJldyBEdW5zdGFuIDxhbmRyZXdAZHVuc2xhbmUubmV0PsLAmwQTAQgARQIbAwIX
 gAIZAQULCQgHAgMiAgEGFQoJCAsCBBYCAwECHgcWIQTkPlhGHfx8v0RpFaWZ+n/LWfw7gQUC
 ZFlxxwUJGVGAbgAKCRCZ+n/LWfw7gXikB/9ZdcUy6CTBFIIuL/bVsc1eLEW/gJBjJBF6HxNY
 xgEkAgXAp4Lg4A5U+QB9GouFr7+GYxF0BU4hzoGhNPUWltxnHdMWP8nC/38LAqgMi8L/bbsm
 HW5YPBdWYaAZAPJQVfOAgjTbRUb26KSprpyrrJKW0ZmrZfjhNPcQ72jpWzoPLQqx2X6B0fru
 1jq+cBh8lb6r1mJTim1T3JIn+F/v5VpdQS+EL8xqsHkfzKjIPsW3CIXpkypSk6saA55Rkkbl
 26AW8ftPVB0Q6Lnn6FLt9CP0MGNixBQ55yq8r1K+nCBvCCjvQjM8RDm0UUum0WNl+ifQgTLO
 E8TWEnwVtkBf+3QWzsBNBE7KWFkBCADRnOM0FCzsYW6jtncg+dWIagjUZpvaClmqn/sJluLa
 Q3v1VXMQJzYs3eC1gh386W+XBwLRpDj3jzH81lX+p73Re3d3oJW7X+ffsxuzu5ZVdMUkqBYo
 nkAbKxr6gyJ12F/+JkUVzLcoTN+d/7YsQvUVi7NaKH8mJgjz112O4fUe3p9wfAaFa0RXHc5S
 GPzRTYRRlv/XZBIho4J2tkZOnteZJZ+GbxQVlINt6fd8P6al3MWOvpP/ExJPguEfjOsO6Njy
 xjo3WfpD4lHMOR/Oc3/8mScEF84rF2jXbsFgelWnbPWAvXY+pD0dXOFRkagGmC/viwBDqq5b
 5tk76kKmUbZxABEBAAHCwHwEGAEIACYCGwwWIQTkPlhGHfx8v0RpFaWZ+n/LWfw7gQUCZFlx
 5wUJGVGAjgAKCRCZ+n/LWfw7gf+iB/4g8CPY5jihf5r/8EsoIGe2H+dpVmpPF8YGBzTIvCz/
 fQoOq8AX/pE76QEuFnFZWfjw+wgBXgCVmkox2Eflkk6z4ND3pcwGZ6CfCxTQCDk/dij+2DQ4
 6bmDCy/sBgcbz9mTpoLC11HLoPae6YN9nBNQRZDcEFEu54OaVOqlIdbA6m+POIBCXZdHOFc0
 WoDTgxHRzC1jgQNidyd6tKqcsVJs0dzF0oKTmFFmUAqTdJO12LBuNA1rlqrR3EtpYk8B/wtS
 5dIMD7Q8hwQpL+4C6GNpb6ZKnPkLi47pDOLhz2qBrqN+rqUEsT3YnExYpzj5yOBi+FlmV1Hw
 49QYe1sn2ZPs
In-Reply-To: <CAEbAQ-VgeghebE+xgJFOF9dppYX0OS-7N_vq=u7fuhz5oBWyyQ@mail.gmail.com>
Archived-At: <https://www.postgresql.org/message-id/69ae171b-87b5-463e-8f60-671a083d2e63%40dunslane.net>
Precedence: bulk

This is a multi-part message in MIME format.
--------------sNv4qgqDXOBjcwhi49XES5eZ
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit


On 2024-12-07 Sa 2:42 AM, Davinder Singh wrote:
>
> On Fri, Dec 6, 2024 at 11:31 AM Thomas Munro <thomas.munro@gmail.com> 
> wrote:
>
>     Some better new:
>
>     1.  _chsize_s does in fact seem to work in an msvcrt.dll build.
>     Thanks to Andres for testing that for me on Windows with a standalone
>     2 line program on ucrt and msvcrt runtimes.
>
>     2.  MinGW always has _chsize_s, but it does more or less what I had
>     been proposing already if it can't find the function by the Windows
>     thing like dlsym, so there's no point in writing another thing
>     like that.
>     The test was with a 3G file though so it was the real function.
>
>     https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-crt/secapi/_chsize_s.c
>
>     3.  After a long time trying various ways to make it tidy and getting
>     incomprehensible results I eventually realised that precompiled
>     headers were causing strange results on MinGW/Meson builds (on CI),
>     due to fighting over macros with this guy:
>
>     https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/unistd.h
>
>     I found it easiest to completely stop it from getting in our way
>     completely, with the attached.  Thoughts, anyone?
>
>     Davinder, Jakub, are you in a position repro the issue and make
>     versions of the patch for the 13, 15 and master branches on top of
>     that?
>
>
> Yes, Thomas, the issue is reproducible.
> After applying your patch and replacing off_t with pgoff_t, the 
> problem has been resolved. I have attached the patches for the 13, 15, 
> and master. These patches include only the changes related to pgoff_t, 
> intended for use with the v3-0001 patch--
>
>


Hi, Thomas,


Do you have a plan for moving ahead with this?


cheers


andrew

--
Andrew Dunstan
EDB:https://www.enterprisedb.com

--------------sNv4qgqDXOBjcwhi49XES5eZ
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
  </head>
  <body>
    <p><br>
    </p>
    <div class="moz-cite-prefix">On 2024-12-07 Sa 2:42 AM, Davinder
      Singh wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAEbAQ-VgeghebE+xgJFOF9dppYX0OS-7N_vq=u7fuhz5oBWyyQ@mail.gmail.com">
      <meta http-equiv="content-type" content="text/html; charset=UTF-8">
      <div dir="ltr">
        <div dir="ltr">
          <div dir="ltr">
            <div><br>
            </div>
          </div>
          <div class="gmail_quote">
            <div dir="ltr" class="gmail_attr">On Fri, Dec 6, 2024 at
              11:31 AM Thomas Munro &lt;<a
                href="mailto:thomas.munro@gmail.com" target="_blank"
                moz-do-not-send="true" class="moz-txt-link-freetext">thomas.munro@gmail.com</a>&gt;
              wrote:<br>
            </div>
            <blockquote class="gmail_quote"
style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">Some
              better new:<br>
              <br>
              1.  _chsize_s does in fact seem to work in an msvcrt.dll
              build.<br>
              Thanks to Andres for testing that for me on Windows with a
              standalone<br>
              2 line program on ucrt and msvcrt runtimes.<br>
              <br>
              2.  MinGW always has _chsize_s, but it does more or less
              what I had<br>
              been proposing already if it can't find the function by
              the Windows<br>
              thing like dlsym, so there's no point in writing another
              thing like that.<br>
              The test was with a 3G file though so it was the real
              function.<br>
              <br>
              <a
href="https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-crt/secapi/_chsize_s.c"
                rel="noreferrer" target="_blank" moz-do-not-send="true"
                class="moz-txt-link-freetext">https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-crt/secapi/_chsize_s.c</a><br>
              <br>
              3.  After a long time trying various ways to make it tidy
              and getting<br>
              incomprehensible results I eventually realised that
              precompiled<br>
              headers were causing strange results on MinGW/Meson builds
              (on CI),<br>
              due to fighting over macros with this guy:<br>
              <br>
              <a
href="https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/unistd.h"
                rel="noreferrer" target="_blank" moz-do-not-send="true"
                class="moz-txt-link-freetext">https://github.com/mingw-w64/mingw-w64/blob/master/mingw-w64-headers/crt/unistd.h</a><br>
              <br>
              I found it easiest to completely stop it from getting in
              our way<br>
              completely, with the attached.  Thoughts, anyone?<br>
              <br>
              Davinder, Jakub, are you in a position repro the issue and
              make<br>
              versions of the patch for the 13, 15 and master branches
              on top of<br>
              that?<br>
            </blockquote>
          </div>
          <div><br clear="all">
          </div>
          <div>Yes, Thomas, the issue is reproducible.</div>
          <div>
            <div>After applying your patch and replacing off_t with
              pgoff_t, the problem has been resolved. I have
              attached the patches for the 13, 15, and master. These
              patches include only the changes related to pgoff_t,
              intended for use with the v3-0001 patch<span
                class="gmail_signature_prefix">--</span></div>
          </div>
          <div><span class="gmail_signature_prefix"><br>
            </span></div>
          <div><span class="gmail_signature_prefix"><br>
            </span></div>
        </div>
      </div>
    </blockquote>
    <p><br>
    </p>
    <p><br>
    </p>
    <p>Hi, Thomas,</p>
    <p><br>
    </p>
    <p>Do you have a plan for moving ahead with this? <br>
    </p>
    <p><br>
    </p>
    <p>cheers</p>
    <p><br>
    </p>
    <p>andrew<br>
    </p>
    <pre class="moz-signature"
    signature-switch-id="d0437855-2267-4610-80b3-83167ec45b0b" cols="72">--
Andrew Dunstan
EDB: <a class="moz-txt-link-freetext" href="https://www.enterprisedb.com">https://www.enterprisedb.com</a>
</pre>
  </body>
</html>

--------------sNv4qgqDXOBjcwhi49XES5eZ--