Date: Sat, 28 Apr 2018 11:10:54 -0400
From: Stephen Frost <sfrost@snowman.net>
To: Andres Freund <andres@anarazel.de>
Cc: Bruce Momjian <bruce@momjian.us>, Craig Ringer <craig@2ndquadrant.com>,
	PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Subject: Re: Postgres, fsync, and OSs (specifically linux)
Message-ID: <20180428151054.GR27724@tamriel.snowman.net>
References: <20180427222842.in2e4mibx45zdth5@alap3.anarazel.de>
 <20180427230447.GA32605@momjian.us>
 <20180427231043.of7vhjmhx4qzexm7@alap3.anarazel.de>
 <20180427233830.GB32605@momjian.us>
 <20180427234332.wtkjvtkbpdl6mu6g@alap3.anarazel.de>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha512;
	protocol="application/pgp-signature"; boundary="UIcbgApIkCRwFatL"
Content-Disposition: inline
In-Reply-To: <20180427234332.wtkjvtkbpdl6mu6g@alap3.anarazel.de>
User-Agent: Mutt/1.5.24 (2015-08-30)
Precedence: bulk


--UIcbgApIkCRwFatL
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings,

* Andres Freund (andres@anarazel.de) wrote:
> On 2018-04-27 19:38:30 -0400, Bruce Momjian wrote:
> > On Fri, Apr 27, 2018 at 04:10:43PM -0700, Andres Freund wrote:
> > > On 2018-04-27 19:04:47 -0400, Bruce Momjian wrote:
> > > > On Fri, Apr 27, 2018 at 03:28:42PM -0700, Andres Freund wrote:
> > > > > - We need more aggressive error checking on close(), for ENOSPC a=
nd
> > > > >   EIO. In both cases afaics we'll have to trigger a crash recovery
> > > > >   cycle. It's entirely possible to end up in a loop on NFS etc, b=
ut I
> > > > >   don't think there's a way around that.
> > > >=20
> > > > If the no-space or write failures are persistent, as you mentioned
> > > > above, what is the point of going into crash recovery --- why not j=
ust
> > > > shut down?
> > >=20
> > > Well, I mentioned that as an alternative in my email. But for one we
> > > don't really have cases where we do that right now, for another we ca=
n't
> > > really differentiate between a transient and non-transient state. It's
> > > entirely possible that the admin on the system that ran out of space
> > > fixes things, clearing up the problem.
> >=20
> > True, but if we get a no-space error, odds are it will not be fixed at
> > the time we are failing.  Wouldn't the administrator check that the
> > server is still running after they free the space?
>=20
> I'd assume it's pretty common that those are separate teams. Given that
> we currently don't behave that way for other cases where we *already*
> can enter crash-recovery loops I don't think we need to introduce that
> here. It's far more common to enter this kind of problem with pg_xlog
> filling up the ordinary way. And that can lead to such loops.

When we crash-restart, we also go through and clean things up some, no?
Seems like that gives us the potential to end up fixing things ourselves
and allowing the crash-restart to succeed.

Consider unlogged tables, temporary tables, on-disk sorts, etc.  It's
entirely common for a bad query to run the system out of disk space (but
have a write of a regular table be what discovers the out-of-space
problem...) and if we crash-restart properly then we'd hopefully clean
things out, freeing up space, and allowing us to come back up.

Now, of course, ideally admins would set up temp tablespaces and
segregate WAL onto its own filesystem, etc, but...

Thanks!

Stephen

--UIcbgApIkCRwFatL
Content-Type: application/pgp-signature; name="signature.asc"

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJa5I7+AAoJEO1sijiDR2RVkfcQAJMwcfJ7/bk8YofT9aNSnf8m
VgiYzbwTztsOF+OKhTA1y6G55o7m0dXlLh5WUY7h/P3XCET4TXJ2Wo14DbaSZgyW
YzBy+iCmpiOT3TC8bY9buegeTm1uTo0bXTST/bPIVk4hYW9li4ecOW0vNFgum1TU
oCz/x244+wbZvI80BCQCXLsqyXzfTorz1TXyvo/pcbot+ndYJFsN8+gDDxIuBRAA
pkOJdkPub91N0xShYdZGyd9TCODUmxJZ/j9+LwwfwbGlNLJkz+93vib86wJvHIve
68q3C+J/k7J2pVjS3RZGEZuCU676f/BDZc/djYzd27wYYdZYg7YjDGcXgJXCLuaY
WOg5o6HrA5/5L3NHndz2R71lP9jjWIQcw5XzDxEoMjgGrsnUhkfmwl5tWOHdSodZ
9vHPubP8L0J+BzITa8b+JCRr2HPv/xECusxOEwQXt9iQWDiOHSlFwVf0Z91BObDx
Kde+JvoKxjKCwIEf5VD4FAcbm+wmHMxiAv4Q4uCLBHTk8on3xd4L+/KDqL8QNWoE
t0q7vrOcuajspva9T/EvHqMAUP+lLi/5T9Q0+1hg0jSHXQAebwASYTHWRpnk7WzX
ujaKj+9zb5SjTf8yH39voICmluJ3QAovVrc5z6mj2EEuzGrjyd2tktIXy/L7GvQj
C63u0SRowTLq87nizTdr
=7uit
-----END PGP SIGNATURE-----

--UIcbgApIkCRwFatL--