Message-ID: <6d18ff30-00bf-42a4-9cad-25be682a43db@dunslane.net>
Date: Thu, 18 Jun 2026 08:17:47 -0400
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: BackgroundPsql swallowing errors on windows
From: Andrew Dunstan <andrew@dunslane.net>
To: Andres Freund <andres@anarazel.de>
Cc: pgsql-hackers@postgresql.org, Noah Misch <noah@leadboat.com>
References: <wmovm6xcbwh7twdtymxuboaoarbvwj2haasd3sikzlb3dkgz76@n45rzycluzft>
 <e5d8d598-bf23-4966-91fb-b4043e451d2c@dunslane.net>
 <gcjjfm54opgd4kohazc5sowpieta77sg7oywooaf75rux2hgky@qt6mqpfquuhj>
 <762648ad-99b5-4871-beaa-ca275e46ef6e@dunslane.net>
 <5dgt6vir7n664qruw3ndywpmfxsmfuu4oeh43syocgo3g6v72n@hjxahrqk4w7t>
 <1779b4d6-768f-4edb-8cd6-62706d321f32@dunslane.net>
 <tir2xsciqfiypagzortygjrwaomwdflbw3tjpz7iji2pefxye7@wkgxlxm7e2bg>
 <1880b789-cb54-4162-ae19-9eb29fabc8da@dunslane.net>
 <cdd68e37-e2b0-4e58-88aa-b4f09923bf00@dunslane.net>
 <3510c901-b760-4534-9371-8a921d955549@dunslane.net>
 <506de8d3-e2b1-4929-88d8-96253d5e4121@dunslane.net>
 <455b56c8-436d-45fe-b70b-314e11ca65fc@dunslane.net>
 <aa1bfc94-bd50-4b40-aec0-77eb533260a0@dunslane.net>
Content-Language: en-US
Autocrypt: addr=andrew@dunslane.net; keydata=
 xsBNBE7KWFkBCAClridxur2AIc7eW2AR7izbfp3EnNefie2HbLF0izW5Ik5UjX2HBXBx4syI
 gY6b0ugohXrr274+baoAlvSbq6cAoQuEVrk5IZFzt20b1Xkx65FwGSEj526yiKLocqkJceSq
 Xr9xcA5SGY+FZv441chh5SU92v4q6z+6LPpoHOh97ptAVXZYNTtU0LevyvD5lja0TzbvJm6C
 eFXitJfnm1pLEr0DGJCR/iUOl/N62Kh4855zZC7NHIjQHPOvV5Stz/l5ilDhvGVk+xkXFPys
 SjZoUr1rXhYLpiyi5sR0X9FHXT0KnGuz1F5ERO7ZTLSSQ6fJwPj6gOk9K+vvoKvoeql5ABEB
 AAHNJEFuZHJldyBEdW5zdGFuIDxhbmRyZXdAZHVuc2xhbmUubmV0PsLAlwQTAQgAQQIbAwIX
 gAIZAQULCQgHAwUVCgkICwUWAgMBAAIeBRYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNGd
 BQkdEO8nAAoJEJn6f8tZ/DuBq74H/jkTR4Zi3stbw+xC7v2u3QozssK7MYPL2AsVfh7OealS
 h182fiWXpfvmmAB7WUHbhk9GC2RAOnHI/2d2jgKaMLAHsGYOT0YopTVIwRY43fCw/mK67yxc
 wmDcX+zyKfLaivNbf5A7QPLNwda98bEAMSJ8Sn652Uc6cA8t3uKGsVzbRBQOoYzjgvBCfSrE
 9ql3PDNg0l4BfAqabd2f70ZUm9VAMEPrgv/v2xI7M2XiL4g5BVmqLCOwxLM8RMCotCuoweUr
 VO43DeBCIDwLxotMJKvGWDjBzQYlU1NPUAtNcz/gN9ITUe1VUGjyvGj4u1lxBOcQQUw7l1+T
 5moZ4iZxXzvOwE0ETspYWQEIANGc4zQULOxhbqO2dyD51YhqCNRmm9oKWaqf+wmW4tpDe/VV
 cxAnNizd4LWCHfzpb5cHAtGkOPePMfzWVf6nvdF7d3eglbtf59+zG7O7llV0xSSoFiieQBsr
 GvqDInXYX/4mRRXMtyhM353/tixC9RWLs1oofyYmCPPXXY7h9R7en3B8BoVrRFcdzlIY/NFN
 hFGW/9dkEiGjgna2Rk6e15kln4ZvFBWUg23p93w/pqXcxY6+k/8TEk+C4R+M6w7o2PLGOjdZ
 +kPiUcw5H85zf/yZJwQXzisXaNduwWB6Vads9YC9dj6kPR1c4VGRqAaYL++LAEOqrlvm2Tvq
 QqZRtnEAEQEAAcLAfAQYAQgAJgIbDBYhBOQ+WEYd/Hy/RGkVpZn6f8tZ/DuBBQJoGNI2BQkd
 EODdAAoJEJn6f8tZ/DuBfw0IAKTsfD40teP/pp+bsLLMSxPXUYrrprTj7WFB5v61p6dkpSr/
 qXmMlyahdxQFaPmfVgVirB1Vk/kHiWNnnGjfUV9nB2Zg9LI0Xb9/ts3LsUiRWXzG3tkMY6XL
 vsVOxW4XFRND9l2q+WW93aZ1DZl+fqWfYgMvsusFRhmGFOKTRfKPta2Pkv+AhA24N4+PrR5p
 bU4k2MO8PAGiK8eaYKGFG1bHKuAvoDoF7WXJ3FHxuWqLnKEt4dfOLm5pAe3zq1Lt6q8azT9i
 QWGpSAK5vQUWQHBHpiDjdPeqKZ6HiAXIIKfSmb+jrvXBqoP+D6/K7rUjG2aXiRtTIAXms9sm
 VRu7cmw=
In-Reply-To: <aa1bfc94-bd50-4b40-aec0-77eb533260a0@dunslane.net>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
Archived-At: 
 <https://www.postgresql.org/message-id/6d18ff30-00bf-42a4-9cad-25be682a43db%40dunslane.net>
Precedence: bulk


On 2026-06-15 Mo 9:43 AM, Andrew Dunstan wrote:
>
> On 2026-06-12 Fr 6:39 PM, Andrew Dunstan wrote:
>>
>> On 2026-06-10 We 4:23 PM, Andrew Dunstan wrote:
>>>
>>> On 2026-06-03 We 5:23 PM, Andrew Dunstan wrote:
>>>>
>>>> On 2026-06-02 Tu 3:03 PM, Andrew Dunstan wrote:
>>>>>
>>>>> On 2026-02-18 We 2:41 PM, Andrew Dunstan wrote:
>>>>>>
>>>>>> On 2026-02-17 Tu 4:56 PM, Andres Freund wrote:
>>>>>>> Hi,
>>>>>>>
>>>>>>> On 2026-02-17 16:31:02 -0500, Andrew Dunstan wrote:
>>>>>>>> On 2026-02-16 Mo 7:17 PM, Andres Freund wrote:
>>>>>>>>> I briefly tried this out. The overall resource usage of the 
>>>>>>>>> test is noticeably
>>>>>>>>> reduced - and that's on linux with fast fork, so it should be 
>>>>>>>>> considerably
>>>>>>>>> better on windows.  However, the tests take a lot longer than 
>>>>>>>>> before, I think
>>>>>>>>> mostly due to polling for results rather than waiting for them 
>>>>>>>>> to be ready
>>>>>>>>> using PQsocketPoll() or such.
>>>>>>>>>
>>>>>>>>> E.g. bloom/001_wal takes about 15s on HEAD for me, but 138s 
>>>>>>>>> with the patch. I
>>>>>>>>> think that's just due to the various usleep(100_000);
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> FWIW, oauth_validator/001_server fails with the patch at the 
>>>>>>>>> moment.
>>>>>>>>>
>>>>>>>> Try this version. On my machine it's now a few percent faster. 
>>>>>>>> I fixed the
>>>>>>>> polling. I also added pipeline support for large sets of 
>>>>>>>> commands, to
>>>>>>>> minimize roundtrips.
>>>>>>> Nice!  Will try it out.
>>>>>>>
>>>>>>>
>>>>>>> Have you tried it on windows already? That's where we pay by far 
>>>>>>> the biggest
>>>>>>> price due to all the unnecessary process creations...
>>>>>>>
>>>>>>> It looks like strawberry perl has FFI::Platypus, but not 
>>>>>>> FFI::C.  There is
>>>>>>> perl/vendor/lib/FFI/Platypus/Lang/C.pm, but that just seems like 
>>>>>>> it's
>>>>>>> documentation.  There is however FFI::Platypus::Record, which 
>>>>>>> maybe could
>>>>>>> suffice?
>>>>>>>
>>>>>>> Do we actually need FFI::C, or can we work around not having it? 
>>>>>>> Looks like
>>>>>>> it's just used for notify related stuff.
>>>>>>>
>>>>>>> It looks like mingw doesn't have packages for FFI::Platypus, but 
>>>>>>> it'll
>>>>>>> probably be a lot easier to build that than when using msvc.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> I replaced the use of FFI::C with FFI::Platypus::Record. That 
>>>>>> comes for free with FFI::Platypus, so there would be no extra 
>>>>>> dependency. It means a little extra housekeeping so we don't lose 
>>>>>> track of the pointer for later use with PQfreemem, but it's not 
>>>>>> too bad.
>>>>>>
>>>>>> I have tried it out with Windows, seemed to work OK although the 
>>>>>> xid_wraparound tests 2 and 3 timed out.
>>>>>>
>>>>>> Latest is attached.
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>> Here is v12. I removed the XS variant in this version, which makes 
>>>>> things simpler. We can restore it if necessary.
>>>>>
>>>>> Patch 1 adds the new framework
>>>>>
>>>>> Patch 2 adapts Cluster.pm to it, as well as handling some 
>>>>> instability at global destruction time that was exacerbated by 
>>>>> using FFI::Platypus.
>>>>>
>>>>> Patch 3 makes improvements in the individual TAP tests using the 
>>>>> framework, including removing every one of the calls to 
>>>>> background_psql().
>>>>>
>>>>>
>>>>> I'm going to add this to the CF and will start testing (again) on 
>>>>> Windows.
>>>>>
>>>>>
>>>>>
>>>>
>>>> v13 attached now passes all tests on my Windows machine.
>>>>
>>>>
>>>>
>>>
>>> rebased, including porting a new use of background_psql.
>>>
>>>
>>>
>>
>> v15 including a check for FFI::Platypus at setup time, and CI 
>> modifications to allow tests to pass.
>>
>>
>>
>
>
> v16 follows several rounds of review, and tightens up a lot of things, 
> so now errors don't silently disappear, or failing tests hang where 
> previously they would time out. There has also been some code cleanup, 
> removal of magic numbers (you can now check for CONNECTION_OK for 
> example), removal of some dead code.
>


In view of discussions on 
https://www.postgresql.org/message-id/cdaaf722-4529-435b-9340-cedf1a3a277f%40dunslane.net 
I have withdrawn the CF item.


cheers


andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com