Sender: Lars Kanis <larskanis@gmail.com>
Message-ID: <68cb19fc-4a1c-473e-99e6-d83b99c2e245@greiz-reinsdorf.de>
Date: Wed, 11 Sep 2024 21:08:45 +0200
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird
Subject: Re: libpq: Process buffered SSL read bytes to support records >8kB on
 async API
To: Jacob Champion <jacob.champion@enterprisedb.com>
Cc: pgsql-hackers@lists.postgresql.org
References: <2039ac58-d3e0-434b-ac1a-2a987f3b4cb1@greiz-reinsdorf.de>
 <CAOYmi+=jZNU9mZ4Z83aR0RiqZ2pF35u5040gCxBgcFqoA4oNbQ@mail.gmail.com>
Content-Language: en-US
From: Lars Kanis <lars@greiz-reinsdorf.de>
Autocrypt: addr=lars@greiz-reinsdorf.de; keydata=
 xsDNBFrzVfABDADpT2a8KyYxT21gCDT+4EShLP+FpTh3Din+Wepj8nkSjPjLWlcMtKP03egA
 EK4kCWRAi2HpKHDPC+KZZkgi+6hRaEUEZpapPVBmsVoDFSVAeBUwwYRYJajKm5diV/ajG4Ab
 rrxbYHzjZnn+LQQhAKpMtHrpeY0uIdR7+qSpzpmVeLB4oBDfWDeviUm+Fy8cCI7uwITkg/uI
 pEchcxX/w4zsdrPs3oO4ghQJ3WEUlJkyfGG4mVXHOKcIW85Nw3U5m09leRzmDY946E45JpF9
 rQfM/veLIxppm+KUdRQYSmthhBBM4mqnGBtxsbuWOGJaSdbPQEr+6Zmc+v+hEzLOZ62nYoX3
 sA4elAYPp+I5j3pepbHGA2Xikiv/7dMCfoBXb/6b4/bdydMwc5tL+wKLcl7bMuA6xgc3se6l
 VCreW4fGWchT1sqTCiiqhm46+Gn4jx3TE1GWfK9xr8LM9bvChs1pnVWN+HV0WJDH4HXX2G4E
 /PAvvAxQSkefsINugKt5SlEAEQEAAc0ZPGxhcnNAZ3JlaXotcmVpbnNkb3JmLmRlPsLBEAQT
 AQgAOgIbAwYLCQcDAgEGFQgJCgsCAhYCAh4BBQkSz/lQFiEEtf+RSflaWb+8/a8FQ7UmuKrj
 K6cFAlrzWPcACgkQQ7UmuKrjK6dCDAv8Cwm3ZEBgbZSB7aEDO0YsM2NljkWcdX8MFoRLPde4
 L18aOe2fMMIf64x8FOE5lNzqoJ6nGzftPJeBKHvJSAx2VuCRHGwVA/wgxvfjjnzAWnYEfbPT
 KyMkv8fHvdfstzW7XPCzX9C7WgwJgwE4LX+zyWm4K4gOkfI/8YujPDenWaL5ZKwVJzeDHeDm
 MAZ3I6isADCL1SmBkuceM/d6KC35TPhGdm/+punR9IkkobuNCBo57T0/H3QV4sSsIcJvvaXf
 Mhs+MEcq9PPIJxQJ2QPSvR/5kZxIYPd+h0yL/ms12HMUQqgkELqqYDbKkDIdPzLORBtCJQ5a
 IhADQDhaHRu4HboH7ScmrKVHrhlGCbSu7VGl3IxSZOynlIWpmpCDFUpePuj75CjQHEW0JIOg
 GcVNHJSFy6Ywb429MiX9NxeKmkekAH2OGZ7QP4BA9nCFEPaMvw6i9Jzkt2n41OhkozoIazVz
 Ijq+pe3Np6Jp052SSOL4eJMveleyg1ATSw63bxASzsDNBFrzVfUBDAC4O72/eetf7mYTPJ4B
 pIeJm/DFVRN1nBGeKka3xITN3Y8bPYRmZf2hVRhE3BiQo3/1HCKjhKs+o0yC7aYWm9ejEeJ/
 tO5CMg/u6/pVu74A6TTrlgnfdzv9D8cQA2AvbcA8l0tbDWQuM9j+PFVn4vxGharNI68j5u5/
 cnEkgpey8cvPw5hGmmcBvKvIHnUzOnR/Ez+JULpRgSsdp9QgNYa6d35EVaAWLXQj9+LptNBZ
 KNSxNIFYSG1+W/Ean1gjhn3+jcHqZlFqI5L0e3pBkIyZku6oHlsmDXAP66Y2uQgcst4rwK9V
 ytsE9cEZDNIDDHClMX5I0CYKkz11jVdjmaIw+z8fjRgpVSg7GTOGFobL54XWn+h7cyWhDssg
 yM5U7AR1jjvyRUTa/1ug2OQayiMz0/hsaunBoWYYbDil7E9rbrFE3bfnKIxaGIwjRCjI4/ny
 jluqtwdhuB4mLSmzDlOXgFvFCNox3jRazqPQxi9BEjO27jDs2MCR9HPeNEQutaUAEQEAAcLA
 5QQYAQgADwUCWvNV9wUJAAAAAAIbDAAKCRBDtSa4quMrp4l6C/0cXg+z54FVG1H54OdI99ad
 RSO2QhSpNPyIY1AQbVXc7BCsBiWWh1+YkClOTUPNI1cmq/TAc51UpU4YUvz2W6BQBQNFJVMA
 wLFKfh+ZNDcKiTFq8Vc/RBfNVdbqRO6sGSjBO5ydtM4TdrasnpDOG+yd319k+yNyARegEiBR
 P9JfWwiouM3y3VFnMYXPScymyUcE1ClawgTZz3PTlTq26lELZTD2OURMA69/LNei0x/EzPY9
 LjNDGbN/G5mXO2O0z8MesiA3uH00FSTuo+y7poPAFkyCL+Z74KUPL8LFoIFnRP12kT5J+oFj
 Cn3CqccBKRaNiV2kT1Kef+3Wvvnm9SERjApz2AjBV9TichaG+S6bgluPB2MMbm9TUXbij7Yt
 wOtdYQullDWdrrvjHWSpHEBd/LL4q4CPsMni7/Q+b+lUARbSN3KaJ1EAzwOQnJu4s7ga79fX
 NtbZ1a9D+RfKrs6CD9CutTOHHVcLce/Xzenr1Sisa7YpgrM2Vy9v2eEXgXk=
In-Reply-To: <CAOYmi+=jZNU9mZ4Z83aR0RiqZ2pF35u5040gCxBgcFqoA4oNbQ@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit
Archived-At: <https://www.postgresql.org/message-id/68cb19fc-4a1c-473e-99e6-d83b99c2e245%40greiz-reinsdorf.de>
Precedence: bulk

Thank you Jacob for verifying this issue!

> Gory details of the packet sizes, if it's helpful:
> - max TLS record size is 12k, because it made the math easier
> - server sends DataRow of 32006 bytes, followed by DataRow of 806
> bytes, followed by CommandComplete/ReadyForQuery
> - so there are three TLS records on the wire containing
>    1) DataRow 1 fragment 1 (12k bytes)
>    2) DataRow 1 fragment 2 (12k bytes)
>    3) DataRow 1 fragment 3 (7430 bytes) + DataRow 2 (806 bytes)
>         + CommandComplete + ReadyForQuery

How did you verify the issue on the server side - with YugabyteDB or 
with a modified Postgres server? I'd like to verify the GSSAPI part and 
I'm familiar with the Postgres server only.


> I agree that PQconsumeInput() needs to ensure that the transport
> buffers are all drained. But I'm not sure this is a complete solution;
> doesn't GSS have the same problem? And are there any other sites that
> need to make the same guarantee before returning?

Which other sites do you mean? The synchronous transfer already works, 
since the select() is short-circuit in case of pending bytes: [1]


> I need to switch away from this for a bit. Would you mind adding this
> to the next Commitfest as a placeholder?

No problem; registered: https://commitfest.postgresql.org/50/5251/

--

Regards, Lars

[1] 
https://github.com/postgres/postgres/blob/77761ee5dddc0518235a51c533893e81e5f375b9/src/interfaces/libpq/fe-misc.c#L1070