Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mCQ5F-0006UU-BJ for pgsql-committers@arkaria.postgresql.org; Sat, 07 Aug 2021 17:26:05 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mCQ5D-0004H5-3n for pgsql-committers@arkaria.postgresql.org; Sat, 07 Aug 2021 17:26:03 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mCQ5C-0004Gw-UE for pgsql-committers@lists.postgresql.org; Sat, 07 Aug 2021 17:26:02 +0000 Received: from wout3-smtp.messagingengine.com ([64.147.123.19]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mCQ5A-0000SC-RO for pgsql-committers@lists.postgresql.org; Sat, 07 Aug 2021 17:26:02 +0000 Received: from compute2.internal (compute2.nyi.internal [10.202.2.42]) by mailout.west.internal (Postfix) with ESMTP id 2ED1C320024A; Sat, 7 Aug 2021 13:25:59 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute2.internal (MEProxy); Sat, 07 Aug 2021 13:25:59 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=rvJjUNOVJNP9m80JZ1iIkr+Gi4r fTXxmA+oyHERq55I=; b=UVLbuiZUwGun61TMyaIlEZTRg6nmTOW+T57EndVZbK/ OLEkofw66qiw2ZZfzeO+eSedwWyp/bcoBTkgqC2OUk2hNR77hwOj5NHUlqctvnrR mVzvmiqLSk32cj++wqhLUOoTHjj4EXOoR557X6KBTSZkS9SU6om7EtlvVSFPjWWh gDgaUNDWY67H4QL7wtBanZ/BIjjeXDSu5h+DvBuUVCq7MGH0xURm4G6YUKDzQGRz hPXZ2NZpFKfnQFmjT3JUJP3bts5YIGMgf9zocP2nc+94T5OEiJSrwxnwU8D1bcgn 0BeN4iyXf5OFBgQR2mQHanqpEvkIYO006soBkY+5spQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm3; bh=rvJjUN OVJNP9m80JZ1iIkr+Gi4rfTXxmA+oyHERq55I=; b=BqROVs154zxnqzhpHaQKiS pQwu1o+bOFP3qwXieMlDB1EyUfLJINdyYJtnxmAcL2YW6TD/QiLYHJ475mr4aBws uuH355JR9CZkR7N7o5Gm5BBwUmM1OP6oiiYDxNxTG4KOP5pAchTSaJhSUrhaZkXq zeQHr7uop1S3KPYCJsFwTjkZVuJsGw2gc7D5Pzx5PM+I42Arm8wbnhfrou96BWwn pmDKxBTK6ap7D0pSNfaNkIpX0Ry9lWafyQWnIcRD2nxY2rpY3buVQXljZBLl5naf I+f5GSdDyls0ODFo2j2u/LLTrJT4QhPmn7H8pKmoN27W2Mi42MCduY73ywYvvkNw == X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvtddrjeefgdduuddvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucenucfjughrpeffhffvuffkfhggtggujgesthdtre dttddtvdenucfhrhhomheptehnughrvghsucfhrhgvuhhnugcuoegrnhgurhgvshesrghn rghrrgiivghlrdguvgeqnecuggftrfgrthhtvghrnhepudekhfekleeugeevteehleffff ejgeelueduleeffeeutdelffeujeffhfeuffdunecuvehluhhsthgvrhfuihiivgeptden ucfrrghrrghmpehmrghilhhfrhhomheprghnughrvghssegrnhgrrhgriigvlhdruggv X-ME-Proxy: Received: by mail.messagingengine.com (Postfix) with ESMTPA; Sat, 7 Aug 2021 13:25:57 -0400 (EDT) Date: Sat, 7 Aug 2021 10:25:56 -0700 From: Andres Freund To: Tom Lane Cc: pgsql-committers@lists.postgresql.org Subject: Re: pgsql: pgstat: Bring up pgstat in BaseInit() to fix uninitialized use o Message-ID: <20210807172556.rda5m7jjhk7ktukl@alap3.anarazel.de> References: <2638683.1628304247@sss.pgh.pa.us> <20210807030652.qf5wsbhxk7dpsmtn@alap3.anarazel.de> <2640064.1628306008@sss.pgh.pa.us> <2640490.1628306556@sss.pgh.pa.us> <20210807044952.orwuf7hua2wytx4g@alap3.anarazel.de> <20210807051149.x2j3n64ejcu23hm4@alap3.anarazel.de> <2675112.1628350987@sss.pgh.pa.us> <20210807164850.52atvo2qkmtfnzgt@alap3.anarazel.de> <2771326.1628356007@sss.pgh.pa.us> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <2771326.1628356007@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Hi, On 2021-08-07 13:06:47 -0400, Tom Lane wrote: > The bigger picture here is that anyplace anybody ever wants to add stats > collection in suddenly becomes a "must run before shmem disconnection" > module. I think that way madness lies --- we can't have the entire > backend shut down before shmem disconnection. So I feel like there's > a serious design problem here, and it's not confined to temp files, > or at least it won't stay confined to temp files. (There may indeed > be other problems already, that we just haven't had the good luck to > have buildfarm timing vagaries expose to us.) I think more often it should not end up as "must run before shmem disconnection" as a whole, but should be split into a portion running at that point. > Maybe the solution is to acknowledge that we might lose some events > during backend shutdown, and redefine the behavior as "we ignore > event reports after pgstat shutdown", not "we assert that there never > can be any such reports". I think it's fine to make such calls, but that it ought to reside in the stats emitting modules. Only it can decide whether needing to emit stats during shutdown is a rare edge case or a commonly expected path. E.g. the case of parallel worker shutdown sending stats too late is something common enough to be problematic, so I don't want to make it too hard to detect such cases. > > I'm somewhat inclined to split InitFileAccess() into two by separating out > > InitTemporaryFileAccess() or such. InitFileAccess() would continue to happen > > early and register a proc exit hook that errors out when there's a temp file > > (as a backstop for non cassert builds). The new InitTemporaryFileAccess() > > would happen a bit later and schedule CleanupTempFiles() to happen via > > before_shmem_access(). And we add a Assert(!proc_exit_inprogress) to the > > routines for opening a temp file. > > Maybe that would work, but after you multiply it by a bunch of different > scenarios in different modules, it's going to get less and less attractive. I'm not quite convinced. Even if we had a nicer ordering / layering of subsystems, we'd still have to deal with subsystems that don't fit nicely because they have conflicting needs. And we'd still need detection of use of subsystems that already have not been initialized / are already shut down. One example of that is imo fd.c being a very low level module that might be needed during shutdown of other modules but which also currently depends on the stats subsystem for temp file management. Which doesn't really make sense, because stats very well could depend on fd.c routines. I think needing to split the current fd.c subsystem into a lower-level (file access) and a higher level (temporary file management) module is precisely what a better designed module layering system will *force* us to do. > > I've bitten by all this often enough to be motivated to propose > > something. However I want to get the basics of the shared memory stats stuff > > in first - it's a pain to keep it upated, and we'll need to find and solve all > > of the issues it has anyway, even if we go for a redesign of module / startup > > / shutdown layering. > > Fair. But I suggest that the first cut should look more like what > I suggest above, ie just be willing to lose events during shutdown. > The downsides of that are not so enormous that we should be willing > to undertake major klugery to avoid it before we've even got a > semi-working system. I think that's more likely to hide bugs unfortunately. Consider fa91d4c91f2 - I might not have found that if we had just ignored "too late" pgstats activity in pgstats.c or fd.c, and that's not an edge case. Greetings, Andres Freund