Received-SPF: PermError (protection.outlook.com: domain of
 blackducksoftware.com used an invalid SPF mechanism)
Subject: Re: BUG #14932: SELECT DISTINCT val FROM table gets stuck in an
 infinite loop
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>, <pgsql-bugs@postgresql.org>
References: <20171127185700.1470.20362@wrigleys.postgresql.org>
 <fd924fd9-8f27-d945-04a9-57b503896107@2ndquadrant.com>
From: "Todd A. Cook" <tcook@blackducksoftware.com>
Message-ID: <5d764e7f-269f-7f6d-3885-88295dc9e03d@blackducksoftware.com>
Date: Thu, 25 Jan 2018 17:31:34 -0500
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.5.2
MIME-Version: 1.0
In-Reply-To: <fd924fd9-8f27-d945-04a9-57b503896107@2ndquadrant.com>
Content-Type: text/plain; charset="utf-8"; format=flowed
Content-Language: en-US
Content-Transfer-Encoding: 7bit
SpamDiagnosticOutput: 1:99
SpamDiagnosticMetadata: NSPM
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jan 2018 22:31:35.5321
 (UTC)
X-MS-Exchange-CrossTenant-Network-Message-Id: 
 eae53c5d-3648-43a6-ae67-08d5644364ab
X-MS-Exchange-CrossTenant-Id: a8cff409-0d9d-48cf-b483-b7442cbeb39d
X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: 
 TenantId=a8cff409-0d9d-48cf-b483-b7442cbeb39d;Ip=[192.231.134.1];Helo=[ops-mail01.dc2.lan]
X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem
X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6PR10MB1873
List-Id: <pgsql-bugs.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:pgsql-bugs@lists.postgresql.org>
List-Owner: <mailto:pgsql-bugs-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/pgsql-bugs>
Precedence: bulk

On 11/27/17 14:17, Tomas Vondra wrote:
> Hi,
> 
> On 11/27/2017 07:57 PM, tcook@blackducksoftware.com wrote:
>> The following bug has been logged on the website:
>>
>> Bug reference:      14932
>> Logged by:          Todd Cook
>> Email address:      tcook@blackducksoftware.com
>> PostgreSQL version: 10.1
>> Operating system:   CentOS Linux release 7.4.1708 (Core)
>> Description:
>>
>> It hangs on a table with 167834 rows, though it works fine with only 167833
>> rows.  When it hangs, CTRL-C does not interrupt it, and the backend has to
>> be killed to stop it.
>>
> 
> Can you share the query and data, so that we can reproduce the issue?
> 
> Based on the stack traces this smells like a bug in the simplehash,
> introduced in PostgreSQL 10. Perhaps somewhere in tuplehash_grow(),
> which gets triggered for 167834 rows (but not for 167833).

FWIW, changing the guts of hashint8() to

+       if (val >= INT32_MIN && val <= INT32_MAX)
+               return hash_uint32((uint32) val);
+       else
+               return hash_any((unsigned char *) &val, sizeof(val));

allows us to process a full-sized data set of around 900 million rows.  However,
memory usage seemed to be rather excessive (we can only run 7 of these jobs in parallel
on a 128GB system before the OOM killer kicked in, rather than the usual 24); if there's
any interest, I can try to measure exactly how excessive.

-- todd