Received: from maia.hub.org (unknown [200.46.204.183]) by mail.postgresql.org (Postfix) with ESMTP id 81A17635192 for ; Tue, 10 Nov 2009 11:00:43 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 71514-05 for ; Tue, 10 Nov 2009 15:00:34 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-ew0-f218.google.com (mail-ew0-f218.google.com [209.85.219.218]) by mail.postgresql.org (Postfix) with ESMTP id 68C8F633484 for ; Tue, 10 Nov 2009 11:00:09 -0400 (AST) Received: by ewy18 with SMTP id 18so106751ewy.19 for ; Tue, 10 Nov 2009 07:00:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type; bh=wZ4RjgOi+hBWg7jrQZFU4IU1urwPzGeFVWQ891ypGuY=; b=QzSLu48gm3EitPChZD69e3lleH1oNuDUhxPYhn3oduBB9gmeFGfrxYmmRgwUmdzmfX Tt9fettLrETHWV+13EInwqOvZ85dDe2zaHpUjWYk6jhecE1EuByl64IzHL0BZ6+mU72s dE2Gffar+cqKTzs5NI8mqi5YnzRnR/yb80SsU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; b=AJGz1QLuZNf5LVS5bcCh2+2Ui7HvL8jHBKKjkEnPbCV8sQv7qIMW0ErpfI1ZUwJiy/ 1FviLYP3cpJdKFLycoXo7/XOPFNaAGToOPgU2vl29bqNvCvdpM1D2W93gnlia9WuhiQD 8kaX8ocEKjlG23lA6ORdvswl7JA3XIx5Tnz/8= MIME-Version: 1.0 Received: by 10.216.90.139 with SMTP id e11mr55025wef.111.1257865206550; Tue, 10 Nov 2009 07:00:06 -0800 (PST) In-Reply-To: References: <8a1bfe660911100355l34a557ffj8244079c1fb689b5@mail.gmail.com> Date: Tue, 10 Nov 2009 16:00:06 +0100 Message-ID: <8a1bfe660911100700l2a44f638m97ea6136b65deb21@mail.gmail.com> Subject: Re: limiting performance impact of wal archiving. From: Laurent Laborde To: Ivan Voras Cc: pgsql-performance@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.284 tagged_above=-10 required=5 tests=BAYES_00=-2.599, SARE_MILLIONSOF=0.315 X-Spam-Level: X-Archive-Number: 200911/99 X-Sequence-Number: 36267 On Tue, Nov 10, 2009 at 3:05 PM, Ivan Voras wrote: > Laurent Laborde wrote: >> >> Hi ! >> We recently had a problem with wal archiving badly impacting the >> performance of our postgresql master. > > Hmmm, do you want to say that copying 16 MB files over the network (and > presumably you are not doing it absolutely continually - there are pauses > between log shipping - or you wouldn't be able to use bandwidth limiting) in > an age when desktop drives easily read 60 MB/s (and besides most of the file > should be cached by the OS anyway) is a problem for you? Slow hardware? > > (or I've misunderstood the problem...) Desktop drive can easily do 60MB/s in *sequential* read/write. We use high performance array of 15.000rpm SAS disk on an octocore 32GB and IO is always a problem. I explain the problem : This server (doing wal archiving) is the master node of the over-blog's server farm. hundreds of GB of data, tens of millions of articles and comments, millions of user, ... ~250 read/write sql requests per seconds for the master ~500 read sql request per slave. Awefully random access overload our array at 10MB/s at best. Of course, when doing sequential read it goes to +250MB/s :) Waiting for "cheap" memory to be cheap enough to have 512Go of ram per server ;) We tought about SSD. But interleaved read/write kill any SSD performance and is not better than SSD. Just more expensive with an unknown behaviour over age. -- ker2x sysadmin & DBA @ http://www.over-blog.com/