Date: Tue, 18 Sep 2018 15:34:57 +0900
From: Michael Paquier <michael@paquier.xyz>
To: Amit Kapila <amit.kapila16@gmail.com>
Cc: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>,
	Robert Haas <robertmhaas@gmail.com>, hlinnaka <hlinnaka@iki.fi>,
	Dilip Kumar <dilipbalaut@gmail.com>,
	pgsql-hackers <pgsql-hackers@postgresql.org>
Subject: Re: Problem while setting the fpw with SIGHUP
Message-ID: <20180918063457.GL31460@paquier.xyz>
References: <20180728231911.GB1471@paquier.xyz>
 <20180801.162545.234930371.horiguchi.kyotaro@lab.ntt.co.jp>
 <CAA4eK1K7dVgKU4zrNTSCW=EoqALG38XmNT0HK9Wdkr935iwTQg@mail.gmail.com>
 <20180828.193436.253621888.horiguchi.kyotaro@lab.ntt.co.jp>
 <20180906233728.GR2726@paquier.xyz>
 <20180914072655.GI29332@paquier.xyz>
 <CAA4eK1+Xfx5jD2CHmLPNqXeOzqRLKG9TNr8wfs3-cPf9Ln9RVg@mail.gmail.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="HxQt9wRz9wbL5Edg"
Content-Disposition: inline
In-Reply-To: 
 <CAA4eK1+Xfx5jD2CHmLPNqXeOzqRLKG9TNr8wfs3-cPf9Ln9RVg@mail.gmail.com>
Precedence: bulk


--HxQt9wRz9wbL5Edg
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Sep 14, 2018 at 04:30:37PM +0530, Amit Kapila wrote:
> On Fri, Sep 14, 2018 at 12:57 PM Michael Paquier <michael@paquier.xyz> wr=
ote:
>> So, I have been working on this problem again and I have reviewed the
>> thread, and there have been many things discussed in the last couple of
>> months:
>> 1) We do not want to initialize XLogInsert stuff unconditionally for all
>> processes at the moment recovery begins, but we just want to initialize
>> it once WAL write is open for business.
>> 2) Both the checkpointer and the startup process can call
>> UpdateFullPageWrites() which can cause Insert->fullPageWrites to get
>> incorrect values.
>=20
> Can you share the steps to reproduce this problem?

This refers to the first problem reported on this thread:
https://www.postgresql.org/message-id/CAFiTN-u4BA8KXcQUWDPNgaKAjDXC%3DC2whn=
zBM8TAcv%3DstckYUw%40mail.gmail.com

In order to reproduce the problem, you can for example stop the server
in immediate mode.  Then attach a debugger to it and add a breakpoint to
UpdateFullPageWrites.  You can check that XLOG insert has not been
initialized yet by looking at xloginsert_cxt ot ThisTimeLineID.  On a
second session, switch full_page_writes to on or off, reload the
parameters and then trigger a checkpoint.  The important point is to
trigger an inconsistency between XLogCtl->Insert->fullPageWrites and
the value of fullPageWrites within the checkpointer context.  With the
checkpoint triggered, the debugger will stop at UpdateFullPageWrites
immediately.  At this point, you can simply check if fullPageWrites
Insert->fullPageWrites have the same value or a different one.  If the
values match, simply switch full_page_writes and reload again, with the
checkpointer still waiting at the beginning of UpdateFullPageWrites.
SIGHUP will make the checkpointer process hang a bit, and then it will
move on.  At this point you will be able to see the failure:
TRAP: FailedAssertion("!(CritSectionCount =3D=3D 0)", File: "mcxt.c", Line:=
 731)
2018-09-18 15:06:39 JST [7396]: [11-1] db=3D,user=3D,app=3D,client=3D LOG:
checkpointer process (PID 7399) was terminated by signal 6: Aborted

> On a regular startup when there is no recovery, it won't allow us to
> log the WAL record (XLOG_FPW_CHANGE) which can happen without above
> change.  You can check that by setting full_page_writes=3Doff and start
> the system.

Oh, good point, InRecovery is set to false in this case so that would be
skipped.  We can simply fix that by adding a flag, say "force" to
UpdateFullPageWrites to allow a process to enforce the update of FPW
even if RecoveryInProgress returns true, which would be the case for the
startup process.
--
Michael

--HxQt9wRz9wbL5Edg
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----

iQIzBAABCgAdFiEEG72nH6vTowiyblFKnvQgOdbyQH0FAlugnJEACgkQnvQgOdby
QH3mYQ/9HKz/Yn1fL8HaNhBZkUud6HzcwYprvE3BV8OhlTaobffAbz6ENl3qaWBq
PQI34cJiFxDX27Zjhq8WyUp6Jn4Ah+yAIbNNb4KNJ7K94rzUL354ygNCMdn66Wd4
aW/Fasn3gYom+rfW5Rrn/LvAytX3GZEW7QsIzOIpV6vhvw6aGldZYRzA2G0TCOqJ
EsvP//r8kcpQZjlhdd5nvJ1FTsndgRbIxJTeqknlDfqKxG0QKWh0my8JqwjR/5/2
DQGfseRMKuBZH4HBo744dKU0qMVV0mULy1eOy2YClc3pBAK1d103phdgqB+m7dao
bgsKG9bDzKS/JXfS0ripbGgW3Cyjyu0wJGEbYSE/d4QP6xIikP4gP1XaVJjXFZQ6
dz8w/tGzAJV1jkrEuhkr90RIgZPoWpFQOXnkClIEQybK+p3tki5yFC8EJlX2xJMs
rhO0XuUO/e35+IOe2nSTDfn9x6jDUmOqVULnwZQ7oan1+YyYCCEaAsjDd/ThBuVT
PgVuTfv5j3U71KsP46GDmMQkExErvipeiL4x8N18Vhu3XaugqoJFavSR4cjd8/4Z
lezR4FPRlPinA02uNyeCWE5igSrLMlLJPUeYwMYbh/fskI5sE3Uslz7VhM9w4wNQ
u3xljFG0c+vEiYVv49BRhUXO/rPRR8ZGB9oisQ8/p1o21LjmqsU=
=sTcv
-----END PGP SIGNATURE-----

--HxQt9wRz9wbL5Edg--