Date: Wed, 8 Jul 2020 10:00:37 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Alvaro Herrera <alvherre@2ndquadrant.com>
Cc: Andres Freund <andres@anarazel.de>, Bruce Momjian <bruce@momjian.us>,
	Jeff Davis <pgsql@j-davis.com>, Robert Haas <robertmhaas@gmail.com>,
	David Rowley <dgrowleyml@gmail.com>,
	Justin Pryzby <pryzby@telsasoft.com>,
	Melanie Plageman <melanieplageman@gmail.com>,
	Tomas Vondra <tomas.vondra@2ndquadrant.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: Default setting for enable_hashagg_disk
Message-ID: <20200708140037.GI3125@tamriel.snowman.net>
References: <85F9F4F0-918A-457F-BC9E-F2AC2EFC1F86@anarazel.de>
 <20200625225853.GA11137@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="NRqim5/tSrBQxN99"
Content-Disposition: inline
In-Reply-To: <20200625225853.GA11137@alvherre.pgsql>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--NRqim5/tSrBQxN99
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Alvaro Herrera (alvherre@2ndquadrant.com) wrote:
> On 2020-Jun-25, Andres Freund wrote:
>=20
> > >My point here is that maybe we don't need to offer a GUC to explicitly
> > >turn spilling off; it seems sufficient to let users change work_mem so
> > >that spilling will naturally not occur.  Why do we need more?
> >=20
> > That's not really a useful escape hatch, because I'll often lead to
> > other nodes using more memory.
>=20
> Ah -- other nodes in the same query -- you're right, that's not good.

It's exactly how the system has been operating for, basically, forever,
for everything.  Yes, it'd be good to have a way to manage the
overall amount of memory that a query is allowed to use but that's a
huge change and inventing some new 'hash_mem' or some such GUC doesn't
strike me as a move in the right direction- are we going to have
sort_mem next?  What if having one large hash table for aggregation
would be good, but having the other aggregate use a lot of memory would
run the system out of memory?  Yes, we need to do better, but inventing
new node_mem GUCs isn't the direction to go in.

That HashAgg previously didn't care that it was going wayyyyy over
work_mem was, if anything, a bug.  Inventing new GUCs late in the
cycle like this under duress seems like a *really* bad idea.  Yes,
people are going to have to adjust work_mem if they want these queries
to continue using a ton of memory to run when the planner didn't think
it'd actually take that much memory- but then, in lots of the kinds of
cases that I think you're worrying about, the stats aren't actually that
far off and people did increase work_mem to get the HashAgg plan in the
first place.

I'm also in support of having enable_hashagg_disk set to true as the
default, just like all of the other enable_*'s.

Thanks,

Stephen

--NRqim5/tSrBQxN99
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJfBdGFAAoJEO1sijiDR2RVO9QP/imb1GfkM0clmroNwBSs2xnt
HRAZwuk88kEAZvJB8qmdQO1NHQi/z0KMpEa4O2U52BAkktfdNKcL1uIf2Pq9VJrb
hCCLSR+miY6dK+/pckcg4TXqv1ldLdlSnnHgzdQ4CIm1VI8AlPPQou3SpNTy6BNp
FhtYJxPEwH3Xcv3GJG7JprPLRRq85Y/OpMfScFDOL2/bcnpk1pCzYPz6CSfO2/Ix
Tp2cqOuyPrddsK0hJ/DmNj/NeCSmFL8qsYqxBh/FIraBvzbs1zM4y47YZQezglYe
GnBija9tHW6JXu7m2+B5vb40nm1sZpmfI/xyT38a1iZ2cinBOMKMo6zkKBku9K6c
VGU5sXAzMXcw1RCEWoVFJUsovOcNiXcPWlMmgHq2Gaq8drpfJFqGJiiXYqZbNOpa
krKK2qanFoqJPDN0gJh1pZXtM7u0MvB8T3QgUcRPlvYHprSfVQoS61xk4Jebbv/S
7Dw/Or8XAZKaXLSmCvas9vtE9KvQs1kG9a/mRoRjFGunpvfiI8qEQhEcObwXoleR
g5roqQ6f8R4DIW9HcVwLum7tPhQydxiuOfF4yj0TGcXdR/p6PsUPZn50VEMQv/pw
RBEIaBk0OYcwyXhX12fvgCgvXHOS6rVR4SSp2C/3+jKFKKw+WD+yd2gOARWdS/F1
UiNzxmbdMcX0au53Nnvs
=1wLE
-----END PGP SIGNATURE-----

--NRqim5/tSrBQxN99--