pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  

help / color / mirror / Atom feed
From: Shaun Thomas <sthomas@peak6.com>
To: Marti Raudsepp <marti@juffo.org>
Cc: Andy Colson <andy@squeakycode.net>
Cc: Igor Chudov <ichudov@gmail.com>
Cc: pgsql-performance@postgresql.org
Subject: Re: Postgres for a "data warehouse", 5-10 TB
Date: Mon, 12 Sep 2011 11:22:55 -0500
Message-ID: <4E6E31DF.6020407@peak6.com> (raw)
In-Reply-To: <CABRT9RB-etP-K-0u0y-Wv+BysSd9ToZm71BqcY7hFQZC2tmjYQ@mail.gmail.com>
References: <CAMhtkAbK7saRO_Tayb1nS3YLWN=Wns=Ci47qs-3YbX0T2+hkpA@mail.gmail.com>
	<CAOR=d=0zmw-EGRJyYmwCTxR5g6osbFLS5i-3kzjG05j7S=1JaA@mail.gmail.com>
	<CAMhtkAah2c4XfSec=OtgL1V51wpD=jygbZnBVBRYCV02MscebQ@mail.gmail.com>
	<4E6CC458.4010206@squeakycode.net>
	<CABRT9RB-etP-K-0u0y-Wv+BysSd9ToZm71BqcY7hFQZC2tmjYQ@mail.gmail.com>

On 09/11/2011 12:02 PM, Marti Raudsepp wrote:

> Which brings me to another important point: don't do lots of small
> write transactions, SAVEPOINTs or PL/pgSQL subtransactions. Besides
> being inefficient, they introduce a big maintenance burden.

I'd like to second this. Before a notable application overhaul, we were 
handling about 300-million transactions per day (250M of that was over a 
6-hour period). To avoid the risk of mid-day vacuum-freeze, we disabled 
autovacuum and run a nightly vacuum over the entire database. And that 
was *after* bumping  autovacuum_freeze_max_age to 600-million.

You do *not* want to screw with that if you don't have to, and a setting 
of 600M is about 1/3 of the reasonable boundary there. If not for the 
forced autovacuums, a database with this much traffic would be corrupt 
in less than a week. We've managed to cut that transaction traffic by 
60%, and it greatly improved the database's overall health.

-- 
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 800 | Chicago IL, 60604
312-676-8870
sthomas@peak6.com

______________________________________________

See http://www.peak6.com/email-disclaimer/ for terms and conditions related to this email



view thread (48+ messages)  latest in thread

Message-ID: <4E6E31DF.6020407@peak6.com>
Permalink:  ../4E6E31DF.6020407@peak6.com/
Also on:    postgresql.org/message-id/4E6E31DF.6020407@peak6.com

 · 

reply
Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: pgsql-performance@postgresql.org
  Cc: sthomas@peak6.com, marti@juffo.org, andy@squeakycode.net, ichudov@gmail.com
  Subject: Re: Postgres for a "data warehouse", 5-10 TB
  In-Reply-To: <4E6E31DF.6020407@peak6.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox