From: Tom Lane <tgl@sss.pgh.pa.us>
To: Jeremy Schneider <schnjere@amazon.com>
cc: neeraj kumar <neeru.cse@gmail.com>, pgsql-admin@postgresql.org,
        pgsql-general@postgresql.org
Subject: Re: Query on pg_stat_activity table got stuck
In-reply-to: <e99f759e-4d18-d297-9e1a-639c7379faf2@amazon.com>
References: 
 <CAPR3Wj5Z17=+eeyrn_ZDG3NQGYgMEOY6JV6Y-WRRhGgwc16U3Q@mail.gmail.com>
 <15877.1557178386@sss.pgh.pa.us>
 <CAPR3Wj5CUVNqF2r6fNnrueDgn2k+qyADCvYZ2878SCytTrPPng@mail.gmail.com>
 <21502.1557339176@sss.pgh.pa.us>
 <CAPR3Wj4fT1hOumweROsAifTAMLT7oM+YejxBAcpPsXuMAecBZw@mail.gmail.com>
 <29660.1557352732@sss.pgh.pa.us>
 <e99f759e-4d18-d297-9e1a-639c7379faf2@amazon.com>
Comments: In-reply-to Jeremy Schneider <schnjere@amazon.com>
	message dated "Thu, 09 May 2019 12:46:32 -0700"
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
Content-ID: <30686.1557432052.1@sss.pgh.pa.us>
Date: Thu, 09 May 2019 16:00:52 -0400
Message-ID: <30687.1557432052@sss.pgh.pa.us>
Precedence: bulk

Jeremy Schneider <schnjere@amazon.com> writes:
> Seems to me that at a minimum, this loop shouldn't go on forever. Even
> having an arbitrary, crazy high, hard-coded number of attempts before
> failure (like a million) would be better than spinning on the CPU
> forever - which is what we are seeing.

I don't think it's the readers' fault.  The problem is that the
writer is violating the protocol.  If we put an upper limit on
the number of spin cycles on the reader side, we'll just be creating
a new failure mode when a writer gets swapped out at the wrong moment.

IMO we need to (a) get the failure-prone code out of the critical
section, and then (b) fix the pgstat_increment_changecount macros
so that the critical sections around these shmem changes really are
critical sections (ie bump CritSectionCount).  That way, if somebody
makes the same mistake again, at least there'll be a pretty obvious
failure rather than a lot of stuck readers.

			regards, tom lane