Subject: Re: Improving connection scalability: GetSnapshotData()
To: Andres Freund <andres@anarazel.de>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Cc: Peter Geoghegan <pg@bowt.ie>,
 Alexander Korotkov <a.korotkov@postgrespro.ru>,
 Alvaro Herrera <alvherre@2ndquadrant.com>, Bruce Momjian <bruce@momjian.us>,
 Thomas Munro <thomas.munro@gmail.com>, David Rowley <dgrowleyml@gmail.com>
References: <20200301083601.ews6hz5dduc3w2se@alap3.anarazel.de>
 <CAH2-WzmNpBm0vqOaxzJhd-31SZVU904xrZvWPp1RyGOEFZQTug@mail.gmail.com>
 <CAPpHfdsJUWeZ=kCUZuA5oADyustFQfUrWMimpD4YL-n8tqShhQ@mail.gmail.com>
 <F0E7D264-FEA3-47D3-815E-C834569EBCDA@anarazel.de>
 <CAPpHfdthCaCt09Nnj6oCYq93_PUfJSw5QOLGgnid3qJsEWXuGA@mail.gmail.com>
 <20200406133959.viql5fqecog6mppj@alap3.anarazel.de>
 <20200406235228.lnxcau2wgocnnn22@alap3.anarazel.de>
 <20200407121503.zltbpqmdesurflnm@alap3.anarazel.de>
From: "Jonathan S. Katz" <jkatz@postgresql.org>
Autocrypt: addr=jkatz@postgresql.org; keydata=
 mQINBFtShwABEAC7PNHDUOTYuifpUCk23KqfxdQQkn4nkoxOXRK0+rAj36FiwqGB4TJFuOVZ
 sDFAEiSlC8Jt4y5Cs7B5tetT8JNd2cs6zp/udMJJDz9d65O9PDpdlMgAmIiTzpLlSdx8FG56
 DTksaDv1d8j3cTJPSE4/fWSxqzA7o3Y9UuL7atZPrzfImgpRKs0of7elIHwOa8GucjyhYqcR
 h60wFBJc2KXqQdDYRTZy43DSnY/0VNc0omiH355fustvpm+m5HjD3w7qZyfN3fpKJpnX1LCF
 f3MnPHaDGITIYGRCBXvf0UqUtD6OEVWPv2C2gyqWMIpWmZTOgDufltKyIByKBoS9x0PlFkij
 04X3KODCngt+N8Ssc9OICc6QSxhjoP48PYPdmiTmkrGuf0LX084wj1xeo1NX7XxZK39F6dTJ
 DhsIiW0sNS0xMxQHLHG9VLbPjx3SANQBh6BuryPz5ZupW9/TIDmkvprtU/oXfKgtfYm3fxmk
 EctxbWrEPsFTFPyuMqQu6l+xyQv0s1VLZfjNWaua6H1/gGoIt6kRnn5qMXDVVpijuWkHbv7G
 ngaQMd258UrrOEHnnjzhQ7jxMWV9D+emxbAtlIxnYvCWlV4IwAQhEHfvudqYaIY3hNWrvQ6H
 GB2KXoTZYN9g5djm14/5nj1IU5zOcovkjJnKhoo9iStnpFF2cwARAQABtCdKb25hdGhhbiBT
 LiBLYXR6IDxqa2F0ekBwb3N0Z3Jlc3FsLm9yZz6JAlQEEwEIAD4CGwMFCQeGH4AFCwkIBwIG
 FQoJCAsCBBYCAwECHgECF4AWIQT6hLaVryv3miBkP/HxBJxynxxlJwUCW1KJJQAKCRDxBJxy
 nxxlJwjrD/kBgqsW4QpNpTFw7ifRokZV08CCX4huPBJQ91rrv+UEWlEcotFBHVkYyHnpzARl
 tcZxhJ9CbFxjniH9cOTty5T/O1yolbOHtZSW8Z8aWV6BVEbjMb+BFxSSLm7RnvJdzQbGCZq2
 ZZvfVpB6z3EHYph4KDdVKvMFjoLskxmdS1DE0tE3zTxvoQsi24Q+HOS07kUjs6fsu/WICMfz
 mgO++AWG9Y0CvN0mm4TkujESzyKM9E5irD+leEMIcddl51Aa2c/VMfBXQbRmpHIgUFTmuHQD
 CnQih+9i3OJAksDg66SP8a7yiXv5mwvyDi1EfTGVKYR2j+pwyjwnC3oIbvDMmB3uTn2JIjnT
 iZKPVtAcAylXjubFltihQgNyuShdP4W+kBwZizhUFqUVL8Anx+KoytYmJPfMRFLGuK4obXKq
 a2ZS3k9KB+H+isOx2nFJOsc7V360Zp1DVaNmuiK10TT6QndShSPaqkJqFtCb6r92rZ9sZM/L
 3vtCI4Rrl3Pt1MgtENXupS8gZpJnAYS0j5A1PAZ09r6ANoaeMHspF+5J5fOHeEvqphXr36mm
 a83Vl1t4orPb0+QmmijmlpseDU63M88Aw5p3c4qj7t8Qr2EZ5zrn7/sFn5wOfbs8Nymxafif
 QCnlV2vg9p0m7vSk/yLJ4PFZvs52FgqAGRCdRn0s2EC99bkCDQRbUocAARAAv8ho/toQ9DG3
 j4f9h9n1aRHr2FlviN2Utpy6L8+dfDggO0geilmkGQOolZ2E60gGfye/kUtF9W3NByO4hxDR
 9u6qbOXcdqnuA+cc68EfqlWFJrVtYFxt0h4ElWYOYnIezKthriWch/FY70FGrxs3z8UHOHq5
 0wBW433eTvZm90WixBiXEt2v1DgW4Vr3ymfO7Aap/IYyPuE4JzgudAuAl0HKPyNEHWHG1dAb
 jX1RiCw9gknIDWQOF0B4UAaJctWGVcnZ3A2ULwNGMa1P9ZJlBWf1vcj01aiHMU0yQ7JjJiSp
 vfm9eM0uSLwRdDrJjyl5ZZqVumjdv2SMNQ8GvYRbEMys3GGDSt9zXgfCSUnPnJfYxjzBHRI6
 x44Wfsx8S6hWxepOogCJJ/g67Bk9mY8YV4klWIXDJVOL5jnBC09DbsZG81JaE2QxB8Y7W36Z
 Mroi9XMxg3s805hQAQUvdG/poU8hN8BWdrnTm/+4eQQp7gDY1ePDmGM6bJC+OHOSnFtR/f+7
 0zpKJ10cc7cBygGnl1yR3KjhFyAWUFvP4ZGziKCcpMwXZfe9PGuyA/YOubMphxIn3YsK2wrd
 faKZYX2GMZCZhMMcvx9IpQrxIJgU+VlwXu/O+Lk10VIPcxPJJwmpdI6HzcS8ZgG6IMcC444X
 XTuLaP8j2mgcMvYak3ScCykAEQEAAYkCPAQYAQgAJhYhBPqEtpWvK/eaIGQ/8fEEnHKfHGUn
 BQJbUocAAhsMBQkHhh+AAAoJEPEEnHKfHGUnReMP/RA2UhGQj+G4uBshkRLjpRysabdPqgQB
 dEBk6wYbio88Wg/2/hgY7UzmDDEwX3sZfQDcrI6+vIobI8uqstZID+WgAAa1JLfChMyVQnSy
 0zfWMOABXscc5tGuvFRZvJklTissMFjXUwaREEKp4ZikTvJ/62MCjSdtrUhnPLvoTsHTKRKD
 ichE+b5A54alwsubTijw12O6N22r5IjZiiSZV0u7dsShyKw+7wCSax9fuBoE43NMYf+dnjMK
 nerAQYUcZWYMnk+EC8RaqYAxv6XZ2tKx1AkGGktwkQIBwrz9IlDSvJ3LWJ2UIIuLRTdngNgg
 GIL4zzuUa2F56FqskQIuYMaNETk6LYfalBDQ6TVLAcgCPQxp4k4i/PRsZ3lZ2ZhRHRYciOvm
 kp+I6EfHwllQpYrWs1thluBGqlJSVJgKl0IOFvKLsQ6KKfqzAwh6FxrO5qajp4viNIgtWoFw
 O5Bp0jgFTbH2OrMWIRfUdUCH1Djbuo65svhNj7FNsQVYzHDI+Nd1I/LOBoPc3UorRMF2M4JV
 kUR7skOHWUbPTSNUr7qc14NSMY3PKjGeVGAsBVHBPvmRx/Ss2tW/TpJWpxg4pmquFQSXuaYh
 Yf4FN8Sxy320pcr+FqN8AhZkYJcNY82OTtR5VEKRC/mYcyq61qXIKqngydn79bJjGxHipxyF FmEC
Message-ID: <6be8c321-68ea-a865-d8d0-50a3af616463@postgresql.org>
Date: Tue, 7 Apr 2020 10:27:11 -0400
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:68.0)
 Gecko/20100101 Thunderbird/68.6.0
MIME-Version: 1.0
In-Reply-To: <20200407121503.zltbpqmdesurflnm@alap3.anarazel.de>
Content-Type: multipart/signed; micalg=pgp-sha256;
 protocol="application/pgp-signature";
 boundary="PeVGliQ8V0pbr54HZgdPnVxfHn1LiwGWz"
Precedence: bulk

This is an OpenPGP/MIME signed message (RFC 4880 and 3156)
--PeVGliQ8V0pbr54HZgdPnVxfHn1LiwGWz
Content-Type: multipart/mixed; boundary="H9KANdPcUbRpMAphiH1yl3tArnO9n6JFp"

--H9KANdPcUbRpMAphiH1yl3tArnO9n6JFp
Content-Type: text/plain; charset=windows-1252
Content-Language: en-US
Content-Transfer-Encoding: quoted-printable

On 4/7/20 8:15 AM, Andres Freund wrote:

> I think this is pretty close to being committable.
>=20
>=20
> But: This patch came in very late for v13, and it took me much longer t=
o
> polish it up than I had hoped (partially distraction due to various bug=
s
> I found (in particular snapshot_too_old), partially covid19, partially
> "hell if I know"). The patchset touches core parts of the system. While=

> both Thomas and David have done some review, they haven't for the lates=
t
> version (mea culpa).
>=20
> In many other instances I would say that the above suggests slipping to=

> v14, given the timing.
>=20
> The main reason I am considering pushing is that I think this patcheset=

> addresses one of the most common critiques of postgres, as well as very=

> common, hard to fix, real-world production issues. GetSnapshotData() ha=
s
> been a major bottleneck for about as long as I have been using postgres=
,
> and this addresses that to a significant degree.
>=20
> A second reason I am considering it is that, in my opinion, the changes=

> are not all that complicated and not even that large. At least not for =
a
> change to a problem that we've long tried to improve.

Even as recently as earlier this week there was a blog post making the
rounds about the pain points running PostgreSQL with many simultaneous
connections. Anything to help with that would go a long way, and looking
at the benchmarks you ran (at least with a quick, nonthorough glance)
this could and should be very positively impactful to a *lot* of
PostgreSQL users.

I can't comment on the "close to committable" aspect (at least not with
an informed, confident opinion) but if it is indeed close to committable
and you can put the work to finish polishing (read: "bug fixing" :-) and
we have a plan both of testing and, if need be, to revert, I would be
okay with including it, for whatever my vote is worth. Is the timing /
situation ideal? No, but the way you describe it, it sounds like there
is enough that can be done to ensure it's ready for Beta 1.

=46rom a RMT standpoint, perhaps this is one of the "Recheck at Mid-Beta"=

items, as well.

Thanks,

Jonathan


--H9KANdPcUbRpMAphiH1yl3tArnO9n6JFp--

--PeVGliQ8V0pbr54HZgdPnVxfHn1LiwGWz
Content-Type: application/pgp-signature; name="signature.asc"
Content-Description: OpenPGP digital signature
Content-Disposition: attachment; filename="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAEBCAAdFiEE+oS2la8r95ogZD/x8QSccp8cZScFAl6MjcAACgkQ8QSccp8c
ZSdQ6A/+JlCCm9c6FxDaZol1oHVVUU38MTYyDmMP3PXPrK0tHuy3rMBAviMFJFEW
s1+vpGuKDMX5naSNDDEp0HOKvaZ0sWkyFgYO87l36c/oPzxL620BJeU/vE3fsiSB
w81kvxXdalvfnCz5dgj/jX69WIlvdPwVIE8i8QW5/RwaV8XryGKjQ3m5PXhMgcsU
FvrBV/Ry4A+NeQzTe4i7nJ8UrFRA2wIujLOxu0OW0iod9+7aPowoSCBeUwa/nowy
LEu06gqPcXgei0aOGPmQPLz0kmTzUgocsJYK5g/hNWY/vK18+ttQRp8XpseurgJX
loD0TwCChuBgSnff2iQElTIlatOtO13BlCrxnNsmQV86wvJgCEJ+ph18zUC2DndO
JxnICW26b9q9vjmqjGnF8m5T5NQV6Xy4Zl8xtha+c+eiX/7do6VysuCDtpY5nNDa
L9iMLpmu79aqhUibElWw0D4JSk7JavJdtzx+pCeMeHr3qMw/T70pN1KV+VYkNmTI
0IoJ6bht4tRRtzvsvnv5qjx23MxqrB639eR8kgrMNZ8ourwcAuPX11ortdxifWcV
tfDUopNuYG2t2/wuYb/QI1xO/h/p7iiEjS7d65tvCyjAHkG64bRtmYIQVO+inq+i
VDvW7g3q/wbED4Y6nwvjmARm73Ok1JQ2m7rlI2Txong3YmXxPG8=
=CwUR
-----END PGP SIGNATURE-----

--PeVGliQ8V0pbr54HZgdPnVxfHn1LiwGWz--