Received: from maia.hub.org (unknown [200.46.204.183]) by mail.postgresql.org (Postfix) with ESMTP id AFEBA63278C for ; Wed, 4 Nov 2009 09:59:35 -0400 (AST) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 57312-02 for ; Wed, 4 Nov 2009 13:59:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pw0-f44.google.com (mail-pw0-f44.google.com [209.85.160.44]) by mail.postgresql.org (Postfix) with ESMTP id 6F02263275D for ; Wed, 4 Nov 2009 09:59:27 -0400 (AST) Received: by pwj15 with SMTP id 15so157544pwj.3 for ; Wed, 04 Nov 2009 05:59:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:in-reply-to:references :date:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=0zg3T+hIv3XoSjESDLgEwewErTVtekZQjhi4Q2SnRq4=; b=PES1ndFIO5WPxMZJuf7t6W1vT08UO9qHcPQz8CjHHYNB827lBGwbEmUXnsSVeInZZ8 HboxljsSOqaD4ZtZ2LNNDbHE3MwyMIYv4C0M4Np9KFKtc+ciWE/vWFDyKvLryjLCsyrH FySY65XSV9uq0pkxxDI52EoaUuqYYdEc3lMoU= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type:content-transfer-encoding; b=o+HNXcpddcPrFBCNhtJKDs5bCaozSjyLxcFwNoA8WDVwy+RysaFXIli7Oe71uBRw7l 24scIzKU2wV24xuswJWYxXwTy2A2MS0l0J7A1ZgutUe8dg/frAEVEV3gwnxHIsmZaO3e 99Pr1fNRqjS6IVtDuqPYXrrryiKHI0tFAEU3c= MIME-Version: 1.0 Received: by 10.114.69.13 with SMTP id r13mr2084760waa.16.1257343165589; Wed, 04 Nov 2009 05:59:25 -0800 (PST) In-Reply-To: <60B0F2124D07B942988329B5B7CA393D01E5B93FF7@mail2.FireEye.com> References: <95652FF3B9EF3045AC4AE8FE5AF8671EED65CF6B@mail2.FireEye.com> <60B0F2124D07B942988329B5B7CA393D01E5B93FF7@mail2.FireEye.com> Date: Wed, 4 Nov 2009 08:59:25 -0500 Message-ID: Subject: Re: High Frequency Inserts to Postgres Database vs Writing to a File From: Merlin Moncure To: Jay Manni Cc: "pgsql-performance@postgresql.org" Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-2.599 X-Spam-Level: X-Archive-Number: 200911/42 X-Sequence-Number: 36210 n Tue, Nov 3, 2009 at 10:12 PM, Jay Manni wrote: > Hi: > > I have an application wherein a process needs to read data from a stream = and > store the records for further analysis and reporting. The data in the str= eam > is in the form of variable length records with clearly defined fields =96= so > it can be stored in a database or in a file. The only caveat is that the > rate of records coming in the stream could be several 1000 records a seco= nd. Postgres doing this is going to depend on primarily two things: *) Your hardware *) The mechanism you use to insert the data into the database Postgres can handle multiple 1000 insert/sec but your hardware most likely can't handle multiple 1000 transaction/sec if fsync is on. You definitely want to batch the insert into the database somehow, so that something accumulates the data (could be a simple file), and flushes it in to the database. The 'flush' ideally should use copy but multiple row insert is ok too. Try to avoid inserting one row at a time even if in a transaction. If you are bulk inserting 1000+ records/sec all day long, make sure you have provisioned enough storage for this (that's 86M records/day), and you should immediately start thinking about partitioning and rotating the log table (if you log to the database, partition/rotate is basically already baked in anyways). The effects on other users of the database are really hard to predict -- it's going to depend on how much resources you have (cpu and especially disk) to direct towards the loading and how the database is being used. I expect it shouldn't be too bad unless your dataase is already i/o loaded. The good news is testing this is relatively easy you can simulate a load test and just run it during typical use and see how it affects other users. Standard o/s tools (iostat, top), and database log with min_duration_statement are going to be a big help here. If you start seeing big leaps in iowait corresponding with unexpectedly lagging queries in your app , you probably should think about scrapping the idea. merlin