pg.ddx.io  pgsql-bugs@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Todd A. Cook <tcook@blackducksoftware.com>
To: Andres Freund <andres@anarazel.de>
Cc: Tom Lane <tgl@sss.pgh.pa.us>
Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Cc: Thomas Munro <thomas.munro@enterprisedb.com>
Cc: PostgreSQL Bugs <pgsql-bugs@postgresql.org>
Subject: Re: BUG #14932: SELECT DISTINCT val FROM table gets stuck in an infinite loop
Date: Tue, 30 Jan 2018 13:57:44 -0500
Message-ID: <039b8fa5-8272-8b14-3710-926de5844677@blackducksoftware.com> (raw)
In-Reply-To: <20180129192647.a6ni3jslsq5uzjkz@alap3.anarazel.de>
References: <1773c302-2fb1-749f-8ccc-62b23617c5ac@blackducksoftware.com>
	<25629.1512580464@sss.pgh.pa.us>
	<20171206171923.dg7phzfh7nb4lou3@alap3.anarazel.de>
	<7de4b2d6-b119-7d15-04d8-4c0fcb29cbc3@2ndquadrant.com>
	<20180126234537.dyu6jwnqz7snpq7n@alap3.anarazel.de>
	<14184.1517010515@sss.pgh.pa.us>
	<20180129010213.rt3nu73ilff756f6@alap3.anarazel.de>
	<15ae5ae2-bc74-ebef-f9d6-34b16423cc04@blackducksoftware.com>
	<20180129183118.rdtm34fshwxlajzm@alap3.anarazel.de>
	<633f3730-0cba-9aed-2724-3ae4295bb0b2@blackducksoftware.com>
	<20180129192647.a6ni3jslsq5uzjkz@alap3.anarazel.de>

On 01/29/18 14:26, Andres Freund wrote:
> On 2018-01-29 13:56:37 -0500, Todd A. Cook wrote:
>> With those patches applied, I'm able to process a 175 million row data set
>> without any problems.
> 
> Cool!
> 
>> I'll try the 900 million row set next.

We're also able to process those 900 million rows without problems. :)
It took about 28 minutes (average of 3 runs).

Out of curiosity, I then modified hashint8() as previously described.
With that change, run time dropped to 11 minutes (also average of 3
runs).

FWIW, the data values in these sets are sort-of random (where I can't
explain the "sort-of" in a public forum), but strongly biased towards
negative infinity.  Starting again from scratch, we could probably remove
the bias, but we have 28-30 billion of these things collected over the
last 14 years (starting in PostgreSQL 7.4) and it's kinda tough to change
directions at this point...

-- todd




view thread (70+ messages)  latest in thread

Message-ID: <039b8fa5-8272-8b14-3710-926de5844677@blackducksoftware.com>
Permalink:  ../039b8fa5-8272-8b14-3710-926de5844677@blackducksoftware.com/
Also on:    postgresql.org/message-id/039b8fa5-8272-8b14-3710-926de5844677@blackducksoftware.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-bugs@postgresql.org
  Cc: tcook@blackducksoftware.com, andres@anarazel.de, tgl@sss.pgh.pa.us, tomas.vondra@2ndquadrant.com, thomas.munro@enterprisedb.com
  Subject: Re: BUG #14932: SELECT DISTINCT val FROM table gets stuck in an infinite loop
  In-Reply-To: <039b8fa5-8272-8b14-3710-926de5844677@blackducksoftware.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox