Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1isjfD-0002KZ-GK for pgsql-bugs@arkaria.postgresql.org; Sat, 18 Jan 2020 08:41:04 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1isjfB-0001WB-80 for pgsql-bugs@arkaria.postgresql.org; Sat, 18 Jan 2020 08:41:01 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1isjfA-0001RB-PJ for pgsql-bugs@lists.postgresql.org; Sat, 18 Jan 2020 08:41:01 +0000 Received: from cyclops.postgrespro.ru ([93.174.131.138] helo=mail.postgrespro.ru) by makus.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1isjf2-0000mV-Ux for pgsql-bugs@lists.postgresql.org; Sat, 18 Jan 2020 08:40:59 +0000 Received: from localhost (localhost [127.0.0.1]) by mail.postgrespro.ru (Postfix) with ESMTP id 3883621C5839 for ; Sat, 18 Jan 2020 11:40:50 +0300 (MSK) X-Virus-Scanned: Debian amavisd-new at postgrespro.ru X-Spam-Flag: NO X-Spam-Score: 0 X-Spam-Level: X-Spam-Status: No, score=x tagged_above=-99 required=4 WHITELISTED tests=[] autolearn=unavailable Received: from [192.168.129.217] (unknown [94.25.177.121]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mail.postgrespro.ru (Postfix) with ESMTPSA id D5BD721C5837 for ; Sat, 18 Jan 2020 11:40:49 +0300 (MSK) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=postgrespro.ru; s=mail; t=1579336850; bh=2aPQ/0cJYr8Zgs+F5GA/h1u/hu+ay3HhlZeviJtsEeE=; h=To:From:Subject:Date; b=gl7ehKPRTb01I94VqEhwbzQC7ENVRLVpO4yXn0vXxmTRiS320nnKEbIV2c37YVsZb SGs4MT06Y353NEvMAB0kljXPi6eNvJKlK/QN82d66OExQwDy6foFLfdBd5So4h4jIu 3x4iG7z78aopUbqz3EYBgG2bc0CIRisfzLYS56uY= To: PostgreSQL mailing lists From: Andrey Lepikhov Subject: Wrong hash table size calculation in Parallel Hash Join Organization: Postgres Professional Message-ID: <5fc5ac28-8ed3-c2d7-8f51-761b2761808c@postgrespro.ru> Date: Sat, 18 Jan 2020 13:40:48 +0500 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.2.2 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------06C5466F64F6E7D129EA9992" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------06C5466F64F6E7D129EA9992 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit While running the Join Order Benchmark [1] test on my laptop with 8GB RAM I saw strange situation where linux (Ubuntu 18, PostgreSQL master branch) killed a backend (or a parallel worker in some cases) with signal 9. Dmesg showed an error: [176313.266426] Out of memory: Kill process 16011 (postgres) score 135 or sacrifice child [176313.266432] Killed process 16011 (postgres) total-vm:1602320kB, anon-rss:1325908kB, file-rss:1136kB, shmem-rss:44612kB I spend time investigating and saw the problem. Calculation of hash table size based on GUC "work_mem". But if we have huge relation, we divides it into the batches, stored into a shared tuplestores. At the first write into the batch, tuplestore allocates memory buffer: accessor->write_chunk = (SharedTuplestoreChunk *) MemoryContextAllocZero(accessor->context, STS_CHUNK_PAGES * BLCKSZ); if we have many batches, we have many additional memory allocations. In my case work_mem=64MB, nbatches=65000 and we need about 2GB of additional memory. I don't specialized in the Parallel Hash Join code before, but considering the code, more accurate calculation of the size of hash table will be based on the quadratic equation, something like this: M_h = M+sqrt(M^2 -M_b*RelSize)/2 here M - work_mem GUC; M_h - estimation of the hash table size; RelSize - size of relation in bytes; M_b=4*STS_CHUNK_PAGES * BLCKSZ. In the case of M^2 < 4*STS_CHUNK_PAGES * BLCKSZ we need to increase work_mem locally. As some sketch of the solution i prepared the patch (see in attachment). If this is a significant problem, I'm ready to continue the solution. [1] https://github.com/gregrahn/join-order-benchmark -- Andrey Lepikhov Postgres Professional https://postgrespro.com The Russian Postgres Company --------------06C5466F64F6E7D129EA9992 Content-Type: text/x-patch; charset=UTF-8; name="hash_table_size_calculation.patch" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="hash_table_size_calculation.patch" diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor/nodeHash.c index 568938667f..a5f6adfbb1 100644 --- a/src/backend/executor/nodeHash.c +++ b/src/backend/executor/nodeHash.c @@ -700,8 +700,13 @@ ExecChooseHashTableSize(double ntuples, int tupwidth, bool useskew, /* * Target in-memory hashtable size is work_mem kilobytes. */ - hash_table_bytes = work_mem * 1024L; - + if (work_mem <= sqrt(4. * 4 * BLCKSZ * inner_rel_bytes) / 1024) + hash_table_bytes = (long) sqrt(4. * 4 * BLCKSZ * inner_rel_bytes) / 2; + else + { + long wm = work_mem * 1024L; + hash_table_bytes = (wm + sqrt(wm*wm - 4. * 4 * BLCKSZ * inner_rel_bytes)) / 2; + } /* * Parallel Hash tries to use the combined work_mem of all workers to * avoid the need to batch. If that won't work, it falls back to work_mem --------------06C5466F64F6E7D129EA9992--