Date: Fri, 20 Aug 2021 10:50:18 -0400
From: "alvherre@alvh.no-ip.org" <alvherre@alvh.no-ip.org>
To: "Bossart, Nathan" <bossartn@amazon.com>
Cc: Kyotaro Horiguchi <horikyota.ntt@gmail.com>,
	"x4mmm@yandex-team.ru" <x4mmm@yandex-team.ru>,
	"a.lubennikova@postgrespro.ru" <a.lubennikova@postgrespro.ru>,
	"hlinnaka@iki.fi" <hlinnaka@iki.fi>,
	"matsumura.ryo@fujitsu.com" <matsumura.ryo@fujitsu.com>,
	"masao.fujii@gmail.com" <masao.fujii@gmail.com>,
	"pgsql-hackers@postgresql.org" <pgsql-hackers@postgresql.org>
Subject: Re: archive status ".ready" files may be created too early
Message-ID: <202108201450.6aao43wd3ju3@alvherre.pgsql>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
In-Reply-To: <202108201433.dkqbqi7c6uw5@alvherre.pgsql>
Archived-At: 
 <https://www.postgresql.org/message-id/202108201450.6aao43wd3ju3%40alvherre.pgsql>
Precedence: bulk

Two things.

1. We use a hash table in shared memory.  That's great.  The part that's
   not so great is that in both places where we read items from it, we
   have to iterate in some way.  This seems a bit silly.  An array would
   serve us better, if only we could expand it as needed.  However, in
   shared memory we can't do that.  (I think the list of elements we
   need to memoize is arbitrary long, if enough processes can be writing
   WAL at the same time.)

   Now that I think about this again, maybe it's limited by
   NUM_XLOGINSERT_LOCKS, since there can only be that many records being
   written down at a time ...

2. There is a new LWLock acquisition that may be a new contention point.
   We acquire the lock in these cases:
   - WAL record insert, when a segment boundary is crosses (rare
     enough).
   - XLogWrite, when a segment needs to be notified.

   Looking again, I think the last point might be a problem actually,
   because XLogWrite is called with WALWriteLock held.  Maybe we should
   take the NotifySegmentsReadyForArchive() call outside the section
   locked by WALWriteLock (so put it in XLogWrite callers instead of
   XLogWrite itself).

-- 
Álvaro Herrera         PostgreSQL Developer  —  https://www.EnterpriseDB.com/
"Cómo ponemos nuestros dedos en la arcilla del otro. Eso es la amistad; jugar
al alfarero y ver qué formas se pueden sacar del otro" (C. Halloway en
La Feria de las Tinieblas, R. Bradbury)