pg.ddx.io  pgsql-docs@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Jeff Davis <pgsql@j-davis.com>
To: Andres Freund <andres@anarazel.de>
To: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Justin Pryzby <pryzby@telsasoft.com>
Cc: Bruce Momjian <bruce@momjian.us>
Cc: David Rowley <dgrowleyml@gmail.com>
Cc: Robert Haas <robertmhaas@gmail.com>
Cc: Melanie Plageman <melanieplageman@gmail.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: Default setting for enable_hashagg_disk
Date: Thu, 25 Jun 2020 09:42:33 -0700
Message-ID: <3e07d41d5104466cb46e3fd076c47989fe65f085.camel@j-davis.com> (raw)
In-Reply-To: <20200624193122.cqolrvl44fh4w4s4@alap3.anarazel.de>
References: <CA+TgmoagMLfruY4y_40oKLmFFdBRVSGraZsTa5ss9wH546EYhQ@mail.gmail.com>
	<ee1a7be071349b93c01ffb9ef4cf3a6830d9e780.camel@j-davis.com>
	<CA+TgmoY4-zSMFf8XbpO6uRsMX4vNPPLXvtajAZpG3f9eoEyjdA@mail.gmail.com>
	<129c8f1739957589584d9525e1d8782eae3f412f.camel@j-davis.com>
	<CAApHDvqFZikXhAGW=UKZKq1_FzHy+XzmUzAJiNj6RWyTHH4UfA@mail.gmail.com>
	<20200624090628.GA19640@momjian.us>
	<20200624123843.GX4107@telsasoft.com>
	<2214502.1593019796@sss.pgh.pa.us>
	<20200624183203.cnqcol27ruastit7@development>
	<2217145.1593024050@sss.pgh.pa.us>
	<20200624193122.cqolrvl44fh4w4s4@alap3.anarazel.de>

On Wed, 2020-06-24 at 12:31 -0700, Andres Freund wrote:
> nodeAgg.c already treats those separately:
> 
> void
> hash_agg_set_limits(double hashentrysize, uint64 input_groups, int
> used_bits,
> 					Size *mem_limit, uint64
> *ngroups_limit,
> 					int *num_partitions)
> {
> 	int			npartitions;
> 	Size		partition_mem;
> 
> 	/* if not expected to spill, use all of work_mem */
> 	if (input_groups * hashentrysize < work_mem * 1024L)
> 	{
> 		if (num_partitions != NULL)
> 			*num_partitions = 0;
> 		*mem_limit = work_mem * 1024L;
> 		*ngroups_limit = *mem_limit / hashentrysize;
> 		return;
> 	}

The reason this code exists is to decide how much of work_mem to set
aside for spilling (each spill partition needs an IO buffer).

The alternative would be to fix the number of partitions before
processing a batch, which didn't seem ideal. Or, we could just ignore
the memory required for IO buffers, like HashJoin.

Granted, this is an example where an underestimate can give an
advantage, but I don't think we want to extend the concept into other
areas.

Regards,
	Jeff Davis







view thread (91+ messages)  latest in thread

Message-ID: <3e07d41d5104466cb46e3fd076c47989fe65f085.camel@j-davis.com>
Permalink:  ../3e07d41d5104466cb46e3fd076c47989fe65f085.camel@j-davis.com/
Also on:    postgresql.org/message-id/3e07d41d5104466cb46e3fd076c47989fe65f085.camel@j-davis.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-docs@postgresql.org
  Cc: pgsql@j-davis.com, andres@anarazel.de, tgl@sss.pgh.pa.us, tomas.vondra@2ndquadrant.com, pryzby@telsasoft.com, bruce@momjian.us, dgrowleyml@gmail.com, robertmhaas@gmail.com, melanieplageman@gmail.com, pgsql-hackers@postgresql.org
  Subject: Re: Default setting for enable_hashagg_disk
  In-Reply-To: <3e07d41d5104466cb46e3fd076c47989fe65f085.camel@j-davis.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox