From: Adam Lee <ali@pivotal.io>
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Jeff Davis <pgsql@j-davis.com>
Cc: Melanie Plageman <melanieplageman@gmail.com>
Cc: Heikki Linnakangas <hlinnaka@iki.fi>
Cc: Taylor Vesely <tvesely@pivotal.io>
Cc: Melanie Plageman <mplageman@pivotal.io>
Cc: Pg Hackers <pgsql-hackers@postgresql.org>
Subject: Re: Memory-Bounded Hash Aggregation
Date: Thu, 20 Feb 2020 13:47:19 +0800
Message-ID: <20200220054719.GB4389@earth.local> (raw)
In-Reply-To: <20200219191636.gvdywx32kwbix6kd@development>
References: <20190802152123.ichmacgx3y3x7dc3@development>
<CAFaX_4Ls5UGQ2UDYkWhn2xFq=V6BKpUbKSQnWmy27VWhQ2=enA@mail.gmail.com>
<e52707d4665e32867fb6dd181825ef15f853b8bf.camel@j-davis.com>
<20191214173225.3kzsbncr6brpxdll@development>
<d72e494db7b075ef02c3163e6614e698701d2c3e.camel@j-davis.com>
<775d754c-9a61-5363-6998-f87d15f1b592@iki.fi>
<CAAKRu_aefEsv+UkQWqu+ioEnoiL2LJu9Diuh9BR8MbyXuZ0j4A@mail.gmail.com>
<20200218185749.srk733n4z57lzxf2@development>
<7612ea4d636b6341d0606f4e852d584798e04901.camel@j-davis.com>
<20200219191636.gvdywx32kwbix6kd@development>
On Wed, Feb 19, 2020 at 08:16:36PM +0100, Tomas Vondra wrote:
> 5) Assert(nbuckets > 0);
> ...
> This however quickly fails on this assert in BuildTupleHashTableExt (see
> backtrace1.txt):
>
> Assert(nbuckets > 0);
>
> The value is computed in hash_choose_num_buckets, and there seem to be
> no protections against returning bogus values like 0. So maybe we should
> return
>
> Min(nbuckets, 1024)
>
> or something like that, similarly to hash join. OTOH maybe it's simply
> due to agg_refill_hash_table() passing bogus values to the function?
>
>
> 6) Another thing that occurred to me was what happens to grouping sets,
> which we can't spill to disk. So I did this:
>
> create table t2 (a int, b int, c int);
>
> -- run repeatedly, until there are about 20M rows in t2 (1GB)
> with tx as (select array_agg(a) as a, array_agg(b) as b
> from (select a, b from t order by random()) foo),
> ty as (select array_agg(a) AS a
> from (select a from t order by random()) foo)
> insert into t2 select unnest(tx.a), unnest(ty.a), unnest(tx.b)
> from tx, ty;
>
> analyze t2;
> ...
>
> which fails with segfault at execution time:
>
> tuplehash_start_iterate (tb=0x18, iter=iter@entry=0x2349340)
> 870 for (i = 0; i < tb->size; i++)
> (gdb) bt
> #0 tuplehash_start_iterate (tb=0x18, iter=iter@entry=0x2349340)
> #1 0x0000000000654e49 in agg_retrieve_hash_table_in_memory ...
>
> That's not surprising, because 0x18 pointer is obviously bogus. I guess
> this is simply an offset 18B added to a NULL pointer?
I did some investigation, have you disabled the assert when this panic
happens? If so, it's the same issue as "5) nbucket == 0", which passes a
zero size to allocator when creates that endup-with-0x18 hashtable.
Sorry my testing env goes weird right now, haven't reproduced it yet.
--
Adam Lee
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: ali@pivotal.io, tomas.vondra@2ndquadrant.com, pgsql@j-davis.com, melanieplageman@gmail.com, hlinnaka@iki.fi, tvesely@pivotal.io, mplageman@pivotal.io
Subject: Re: Memory-Bounded Hash Aggregation
In-Reply-To: <20200220054719.GB4389@earth.local>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox