Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kfXQ9-00072r-GK for pgsql-bugs@arkaria.postgresql.org; Thu, 19 Nov 2020 00:03:29 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kfXPb-0004Of-TI for pgsql-bugs@arkaria.postgresql.org; Thu, 19 Nov 2020 00:02:55 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kfXPb-0004OY-NA for pgsql-bugs@lists.postgresql.org; Thu, 19 Nov 2020 00:02:55 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kfXPU-0002qq-V3 for pgsql-bugs@lists.postgresql.org; Thu, 19 Nov 2020 00:02:55 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 0AJ02YZR1010422; Wed, 18 Nov 2020 19:02:34 -0500 From: Tom Lane To: Andres Freund cc: Magnus Hagander , Fujii Masao , skoposov@ed.ac.uk, PostgreSQL mailing lists Subject: Re: BUG #16722: PG hanging on COPY when table has close to 2^32 toasts in the table. In-reply-to: <20201118230529.6u7fyl6trnepekgc@alap3.anarazel.de> References: <16722-93043fb459a41073@postgresql.org> <95bfb4b5-2e0e-8ab7-12cf-03fae3367c52@oss.nttdata.com> <20201118192743.kr43onoxci5gexjx@alap3.anarazel.de> <20201118230529.6u7fyl6trnepekgc@alap3.anarazel.de> Comments: In-reply-to Andres Freund message dated "Wed, 18 Nov 2020 15:05:29 -0800" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <1010420.1605744154.1@sss.pgh.pa.us> Date: Wed, 18 Nov 2020 19:02:34 -0500 Message-ID: <1010421.1605744154@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Andres Freund writes: > On 2020-11-18 23:12:10 +0100, Magnus Hagander wrote: >> That said, I'd really prefer to see *some* kind of counter that could >> get people noticing this a bit earlier, rather than just bugging them >> in the logfile once it's gone over a threshold. A statistics counter >> maybe, but I guess that'd have to be tracked at a per-table level to >> be really useful, which would make it potentially fairly expensive to >> keep around... > I'd rather spend effort to solve the underlying issue. It's a bit more > work, but not crazily so, I think. Even the pg_upgrade pieces aren't > that hard to solve. Yeah. If we're going to put work into this, widening the IDs used to identify toast values seems like the right work to be doing. Having said that, I had imagined that we might never have to fix it, because if your table is big enough that it has a problem of this ilk then likely you want to partition it anyway. And partitioning solves the problem since each partition has its own toast table. regards, tom lane