Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kirOK-0003Hm-VM for pgsql-bugs@arkaria.postgresql.org; Sat, 28 Nov 2020 03:59:21 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1kirOH-0006x7-Gm for pgsql-bugs@arkaria.postgresql.org; Sat, 28 Nov 2020 03:59:17 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1kirOH-0006x0-2W for pgsql-bugs@lists.postgresql.org; Sat, 28 Nov 2020 03:59:17 +0000 Received: from mail-pl1-x633.google.com ([2607:f8b0:4864:20::633]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1kirOE-0007lW-2L for pgsql-bugs@lists.postgresql.org; Sat, 28 Nov 2020 03:59:16 +0000 Received: by mail-pl1-x633.google.com with SMTP id p6so3565212plr.7 for ; Fri, 27 Nov 2020 19:59:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=leadboat.com; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=O39nm/678t4GHx5tifPzyy7DFEj3TgY+N9nJeVTTNxA=; b=OHkl9R+VedhVKknvGfHCP9c8mWnJfeq0mLRuLmUgVvTzEstdt401/MpUNmPXRw/ShF K6q2HoBx1vw3BRXzCkXSNKlv2A1teHLTyGHOr0U5uqrMF7uhzq1JlKjYQHyC+ZuCMlsO kjozt8rBtQ7Fpbhc+1mCF2aqWc3hLEBVBMeos= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=O39nm/678t4GHx5tifPzyy7DFEj3TgY+N9nJeVTTNxA=; b=AJxMUKxlxHCekQ8SATjwa+IEjR/JdZKrC4mPwGMNSUyN21lAv7i/FcKi/PFU0aelpD Mcgy3WHz2bYCTbbXnlPaa7EGAib4xRBeJ1w4c7JzOPC0T1g6uEvBnoPX5YnV3KsjoeIP rCrdMKzwla8/La6Jm1U/0KoA9kBNZg4EzfQmnkqiREklxbc9I/m6hNamyMVOsYoUYhtq G77I8vE/zw+qM00K09WdMkqj0ChFVirYpIU00UevodkawzavH/qGTBbjOw5oqTugoU+v tKGDqpbfKnuSI/co/PTvr4gpE++AvEa4NbTSole0SgfMDEOGllxaumTHk6U77z5eHcx5 hsag== X-Gm-Message-State: AOAM530F+jXIYFbj74Nlukg6jTRbPLbV8LoSOJhFDXfB5OG5gyGNrwvj mbXL7GhoyZaUELoEGPGQ4g9xFg== X-Google-Smtp-Source: ABdhPJzkVTEy9WI25nXmCudmm8E3vE4FpNjR+WiVSrhsmzoJB+9cC+defOPeZeHRN9u56uXrBIpT1w== X-Received: by 2002:a17:902:b905:b029:d8:ad03:8c93 with SMTP id bf5-20020a170902b905b02900d8ad038c93mr9707041plb.15.1606535950849; Fri, 27 Nov 2020 19:59:10 -0800 (PST) Received: from rfd.leadboat.com ([2600:1702:a20:5750::43]) by smtp.gmail.com with ESMTPSA id d15sm13222537pjr.27.2020.11.27.19.59.08 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 27 Nov 2020 19:59:09 -0800 (PST) Date: Fri, 27 Nov 2020 19:59:06 -0800 From: Noah Misch To: Tom Lane Cc: Gustavsson Mikael , pgsql-bugs@lists.postgresql.org, Svensson Peter , Almen Anders Subject: Re: SV: Problem with pg_notify / listen Message-ID: <20201128035906.GA556656@rfd.leadboat.com> References: <1b8561412e8a4f038d7a491c8b922788@smhi.se> <704331.1606490493@sss.pgh.pa.us> <6c10dcdd2eb64bbab4814e4f2abd5ab4@smhi.se> <7a0df07fb3f94dfe80839942ab5993c7@smhi.se> <716909.1606498665@sss.pgh.pa.us> <730247.1606515208@sss.pgh.pa.us> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <730247.1606515208@sss.pgh.pa.us> User-Agent: Mutt/1.5.24 (2015-08-30) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On Fri, Nov 27, 2020 at 05:13:28PM -0500, Tom Lane wrote: > I wrote: > > * Change the code back to being atomic, ie go ahead and update > > QUEUE_TAIL immediately, and truncate the SLRU only afterward. > > Why is it necessary, or even safe, to perform the SLRU truncation > > before changing QUEUE_TAIL? (IOW, I wonder if there isn't a bug > > there in HEAD too.) Commit d4031d7 added this comment about necessity: + * ... Mutual exclusion must end after any limit + * update that would permit other backends to write fresh data into the + * segment immediately preceding the one containing cutoffPage. Otherwise, + * when the SLRU is quite full, SimpleLruTruncate() might delete that segment + * after it has accrued freshly-written data. ... but ... > After thinking more about that, I'm pretty sure there is a bug there: > a newly-arriving backend could attempt to scan the queue starting at > QUEUE_TAIL, concurrently with SimpleLruTruncate removing the page(s) > it wants to scan. In typical cases no failure will occur because > Exec_ListenPreCommit could advance its queue pointer to a safe place > by observing the pointers of other backends. However, if the new > listener were the only one in its database, we'd have trouble. ... agreed. In general, recycling SLRU space entails three steps that shall not overlap: 1. Stop reading data in the space, regulated by some "logical tail". 2. Unlink files wholly within the bounds of the space. 3. Start writing data into the space, regulated by some "physical tail" (most often called a "stop limit"). Commit d4031d7 fixed overlap of (2) and (3). For pg_notify, though, it introduced overlap of (1) and (2). I've now checked the other SLRUs for similar problems, but I found nothing urgent: - pg_xact and pg_subtrans have oldestClogXid as their logical tail and xidStopLimit as their physical tail. All good. - pg_multixact/offsets and pg_multixact/members store no logical tails. They have multiStopLimit and offsetStopLimit as physical tails. Under the right race condition, pg_get_multixact_members() could get a low-level failure. That's an undocumented debug function, so letting it fail that way is fine. - pg_serial has tailXid as its logical tail. It stores no physical tail. This causes some nearly-impossible bugs, discussed in the comments that https://postgr.es/m/20201109045319.GA459206@rfd.leadboat.com adds to predicate.c. > @@ -286,6 +288,7 @@ static AsyncQueueControl *asyncQueueControl; > > #define QUEUE_HEAD (asyncQueueControl->head) > #define QUEUE_TAIL (asyncQueueControl->tail) > +#define QUEUE_TAIL_PAGE (asyncQueueControl->tailPage) I think we don't yet have the right name here, seeing QUEUE_TAIL_PAGE != QUEUE_POS_PAGE(QUEUE_TAIL) sounds paradoxical, yet happens regularly. How about naming it QUEUE_STOP_PAGE? Otherwise, this looks good. Thanks for diagnosing and fixing this defect.