Message-ID: <4E0CABC3.7000101@agliodbs.com>
Date: Thu, 30 Jun 2011 10:00:51 -0700
From: Josh Berkus <josh@agliodbs.com>
Organization: PostgreSQL Experts Inc.
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US;
	rv:1.9.2.15) Gecko/20110303 Lightning/1.0b2 Thunderbird/3.1.9
MIME-Version: 1.0
To: pgsql-hackers@postgresql.org
Subject: Re: time-delayed standbys
References: <BANLkTi==TTzHDqWzwJDjmOf__8YuA7L1jw@mail.gmail.com>
	<BANLkTi=QScpDTkUw3iw6JaO15uUvx_P4qg@mail.gmail.com>
	<BANLkTimarB9eLhHpW73ja2oGvCaWNQuD6g@mail.gmail.com>
	<BANLkTi=hR3vGUDX-SQf=nFcL9ycmrQ6Fcg@mail.gmail.com>
	<BANLkTikPdjn98yjSR-VYywyjMe64wdrdDQ@mail.gmail.com>
	<26007.1303309175@sss.pgh.pa.us>
	<BANLkTikqGottQvyBzrXsARkaWMB2OEMGEA@mail.gmail.com>
	<27149.1303312506@sss.pgh.pa.us>
	<BANLkTin=22tyhK5zaNm7Nt-Q+MHcsNHdLg@mail.gmail.com>
	<BANLkTikFhVOhmf_XKERe=MkU3Ht_K7posQ@mail.gmail.com>
	<BANLkTi=COgk+8cieWxhM7YN9guYXOrDb_g@mail.gmail.com>
	<BANLkTingeD+XrxN-gDv0pzUMHrPcHWSF+A@mail.gmail.com>
	<BANLkTim8rKuUbDP=TxWmtfvZk8C1vmgmsw@mail.gmail.com>
	<BANLkTi=ufQk9i2Ria1U-c0VDUMUNxKjPgg@mail.gmail.com>
	<BANLkTi=33oiYV36cM_nOW7_Pi3t4E=P32g@mail.gmail.com>
	<4E0BD74B.7020105@agliodbs.com>
	<BANLkTimntX0B4ao_YKZemo7VtF8C3vy3-g@mail.gmail.com>
	<BANLkTi=nvWefCymUz8_mT_UK16trw-si6Q@mail.gmail.com>
In-Reply-To: <BANLkTi=nvWefCymUz8_mT_UK16trw-si6Q@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 7bit

On 6/30/11 2:00 AM, Simon Riggs wrote:
>>> Manual (or scripted) intervention is always necessary if you reach disk
>>> >> 100% full.
>> >
>> > Wow, that's a pretty crappy failure mode... but I don't think we need
>> > to fix it just on account of this patch.  It would be nice to fix, of
>> > course.
> How is that different to running out of space in the main database?
> 
> If I try to pour a pint of milk into a small cup, I don't blame the cup.

I have to agree with Simon here.  ;-)

We can do some things to make this easier for administrators, but
there's no way to "solve" the problem.  And the things we could do would
have to be advanced optional modes which aren't on by default, so they
wouldn't really help the DBA with poor planning skills.  Here's my
suggestions:

1) Have a utility (pg_archivecleanup?) which checks if we have more than
a specific settings's worth of archive_logs, and breaks replication and
deletes the archive logs if we hit that number.  This would also require
some way for the standby to stop replicating *without* becoming a
standalone server, which I don't think we currently have.

2) Have a setting where, regardless of standby_delay settings, the
standby will interrupt any running queries and start applying logs as
fast as possible if it hits a certain number of unapplied archive logs.
 Of course, given the issues we had with standby_delay, I'm not sure I
want to complicate it further.

I think we've already fixed the biggest issue in 9.1, since we now have
a limit on the number of WALs the master will keep if archiving is
failing ... yes?  That's the only big *avoidable* failure mode we have,
where a failing standby effectively shuts down the master.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com