pg.ddx.io  pgsql-hackers@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
To: Tels <nospam-pg-abuse@bloodgate.com>
To: David Rowley <david.rowley@2ndquadrant.com>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Stephen Frost <sfrost@snowman.net>
Cc: Andres Freund <andres@anarazel.de>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: Parallel Aggregates for string_agg and array_agg
Date: Thu, 5 Apr 2018 22:46:26 +0200
Message-ID: <fdbf52dc-2e80-f5bc-5d43-b66a2deba021@2ndquadrant.com> (raw)
In-Reply-To: <f459ffbfad91ece161b8dc6095f3475e.squirrel@sm.webmail.pair.com>
References: <CAKJS1f8cB3cc5AK=wPvByhU_jKkgd81iRt2kE6cru8uxx2HCuA@mail.gmail.com>
	<CAKJS1f8JVpUUkYP2c+RyVRdpS1dnjB_DA5fbcDSL3qfMRqc81Q@mail.gmail.com>
	<6538.1522096067@sss.pgh.pa.us>
	<25812.1522108164@sss.pgh.pa.us>
	<CAKJS1f87rFqL0=ksfs+3mmQvg_R0RRGEXyayxHkwYnFqeKpngQ@mail.gmail.com>
	<CAKJS1f9oTWG4oc7B2WfDhT11KhxeV1miLQ+PzB6hY=iJeHM28A@mail.gmail.com>
	<19216.1522207699@sss.pgh.pa.us>
	<18741.1522245278@sss.pgh.pa.us>
	<1ffa699d-ff84-4c21-10cb-4d8b36017bab@2ndquadrant.com>
	<CAKJS1f95uF3h0PdMj-zqnTcJXnF68bT4nUKgiXtAdEz44W5HWg@mail.gmail.com>
	<13e3fd13-8152-72be-4e3c-8095e6ec0995@2ndquadrant.com>
	<CAKJS1f-S493AXpKAiri7cSFfhy2vy5kLHzkTDXC5QzoxMJ5Lkg@mail.gmail.com>
	<73ea70fa-9ada-deb5-549e-02fa2f0969c6@2ndquadrant.com>
	<CAKJS1f9jP_b=yoLa5bt9WPC5mtNgMKFgP6FH2ejZ_PqYyxbDAQ@mail.gmail.com>
	<ad2283b9-64bd-3016-da60-648ba9c13c5f@2ndquadrant.com>
	<CAKJS1f_yDsOrSffHBqAN3Gx27R1vW6OHaKy_EwnhnLZJmo8+Mw@mail.gmail.com>
	<f459ffbfad91ece161b8dc6095f3475e.squirrel@sm.webmail.pair.com>



On 04/05/2018 09:10 PM, Tels wrote:
> Moin,
> 
> On Wed, April 4, 2018 11:41 pm, David Rowley wrote:
>> Hi Tomas,
>>
>> Thanks for taking another look.
>>
>> On 5 April 2018 at 07:12, Tomas Vondra <tomas.vondra@2ndquadrant.com>
>> wrote:
>>> Other than that, the patch seems fine to me, and it's already marked as
>>> RFC so I'll leave it at that.
>>
>> Thanks.
> 
> I have one more comment - sorry for not writing sooner, the flu got to me ...
> 
> Somewhere in the code there is a new allocation of memory when the string
> grows beyond the current size - and that doubles the size. This can lead
> to a lot of wasted space (think: constructing a string that is a bit over
> 1 Gbyte, which would presumable allocate 2 GByte).
> 

I don't think we support memory chunks above 1GB, so that's likely going
to fail anyway. See

  #define MaxAllocSize       ((Size) 0x3fffffff) /* 1 gigabyte - 1 */
  #define AllocSizeIsValid(size)     ((Size) (size) <= MaxAllocSize)

But I get your point - we may be wasting space here. But that's hardly
something this patch should mess with - that's a more generic allocation
question.

> The same issue happens when each worker allocated 512 MByte for a 256
> Mbyte + 1 result.
> 
> IMHO a factor of like 1.4 or 1.2 would work better here - not sure what
> the current standard in situations like this in PG is.
> 

With a 2x scale factor, we only waste 25% of the space on average.
Consider that you're growing because you've reached the current size,
and you double the size - say, from 1MB to 2MB. But the 1MB wasted space
is the worst case - in reality we'll use something between 1MB and 2MB,
so 1.5MB on average. At which point we've wasted just 0.5MB, i.e. 25%.

That sounds perfectly reasonable to me. Lower factor would be more
expensive in terms of repalloc, for example.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services




view thread (77+ messages)  latest in thread

Message-ID: <fdbf52dc-2e80-f5bc-5d43-b66a2deba021@2ndquadrant.com>
Permalink:  ../fdbf52dc-2e80-f5bc-5d43-b66a2deba021@2ndquadrant.com/
Also on:    postgresql.org/message-id/fdbf52dc-2e80-f5bc-5d43-b66a2deba021@2ndquadrant.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-hackers@postgresql.org
  Cc: tomas.vondra@2ndquadrant.com, nospam-pg-abuse@bloodgate.com, david.rowley@2ndquadrant.com, tgl@sss.pgh.pa.us, sfrost@snowman.net, andres@anarazel.de
  Subject: Re: Parallel Aggregates for string_agg and array_agg
  In-Reply-To: <fdbf52dc-2e80-f5bc-5d43-b66a2deba021@2ndquadrant.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox