Received: from localhost (unknown [200.46.204.183]) by postgresql.org (Postfix) with ESMTP id 5D63464FCEB for ; Sun, 17 Aug 2008 21:05:17 -0300 (ADT) Received: from postgresql.org ([200.46.204.86]) by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024) with ESMTP id 95336-07 for ; Sun, 17 Aug 2008 21:04:52 -0300 (ADT) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from fg-out-1718.google.com (fg-out-1718.google.com [72.14.220.153]) by postgresql.org (Postfix) with ESMTP id 27AE264FCC7 for ; Sun, 17 Aug 2008 21:04:55 -0300 (ADT) Received: by fg-out-1718.google.com with SMTP id 16so1852483fgg.23 for ; Sun, 17 Aug 2008 17:04:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from:to :subject:cc:in-reply-to:mime-version:content-type :content-transfer-encoding:content-disposition:references; bh=eyPtj2996AkRKu3dnDjfzT3RSnyJrkZXzg2wARikU18=; b=U4mq6vPiWdVtlJ9n2qZ7z+dP5AGbtq2d9VHi12O4V149wTj96VLFBvpyTtETYxSVIT O/gpc/xR+4MTanHZW+dKCyMBgXh/xWacS8kJvrxFDRuS5b8Eubu+REvvEyJR5L85trZg yoh/80HCYNE24V8ZFNO4CSTOwexOQ+gtr0tcM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:to:subject:cc:in-reply-to:mime-version :content-type:content-transfer-encoding:content-disposition :references; b=uFNZRHZ0E6//rl+XGCJk1MhAUIt2shUGqb6dHFecqcKqvoHNTzNtyZ3k9Nw2c4yB43 k8ehvfaNKqmF79n6eeD6/VrzH3ZMvtnuG0kni5aO8tzMAJ4r84FCSah+k5r6kivXsGKn LAFSlVjNVCNC2IGNWuq1W4RRK+6etIpTbv3t8= Received: by 10.103.131.18 with SMTP id i18mr3532257mun.126.1219017893630; Sun, 17 Aug 2008 17:04:53 -0700 (PDT) Received: by 10.103.12.18 with HTTP; Sun, 17 Aug 2008 17:04:53 -0700 (PDT) Message-ID: Date: Sun, 17 Aug 2008 20:04:53 -0400 From: "Merlin Moncure" To: "Sabin Coanda" Subject: Re: long transaction Cc: pgsql-performance@postgresql.org In-Reply-To: MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Content-Disposition: inline References: <120621.32315.qm@web59503.mail.ac4.yahoo.com> X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0 tagged_above=0 required=5 tests=none X-Spam-Level: X-Archive-Number: 200808/187 X-Sequence-Number: 31068 On Wed, Aug 13, 2008 at 2:07 AM, Sabin Coanda wrote: >> >> have you considered importing to a temporary 'holding' table with >> copy, then doing 'big' sql statements on it to check constraints, etc? >> > > Yes I considered it, but the problem is the data is very tight related > between different tables and is important to keep the import order of each > entity into the database. With other words, the entity imprt serialization > is mandatory. In fact the import script doesn't keep just insert but also > delete and update for different entities. So copy is not enough. Also using > 'big' sql statements cannot guarantee the import order. More than likely, to solve your problem (outside of buying bigger box or hacking fsync) is to rethink your import along the lines of what I'm suggesting. You're welcome to give more specific details of what/how your imports are running, in order to get more specific advice. merlin