Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id C42321337B6C for ; Sun, 31 Oct 2010 02:45:10 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 21867-02-2 for ; Sun, 31 Oct 2010 05:45:03 +0000 (UTC) Received: from mx3.hub.org (mx3.hub.org [206.223.169.73]) by mail.postgresql.org (Postfix) with ESMTP id B99121337B81 for ; Sun, 31 Oct 2010 02:45:02 -0300 (ADT) X-Greylist: delayed 00:08:10.93389 by SQLgrey-1.7.6 Received: from mo-p00-ob.rzone.de (mo-p00-ob.rzone.de [81.169.146.160]) by mx3.hub.org (Postfix) with ESMTP id 4ECFB37C337 for ; Sun, 31 Oct 2010 02:28:38 -0300 (ADT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed/relaxed; t=1288502463; l=753; s=domk; d=4c-ag.de; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References: In-Reply-To:Subject:Cc:To:From:Date:X-RZG-CLASS-ID:X-RZG-AUTH; bh=8Eg7XqZcSRRk6ITF6HCnL3PDSTI=; b=iTo6ZL4qHQOPuf4B3upugOm4jsDKeVBKGJwgnF4YP5UjpL/jX+0+V3M0a1N6U7xxp8L QWsEaJ+cerFzZOGBlV07eEIzYqvANcGp8Z6ED7KuS4O2EsGBleZrBUU5Jh1/n9BrwRLNw I4H8A8P5CcibP1CVT2a2FqM7eqk3y7QmHEw= X-RZG-AUTH: :ICIXYUWhdf07X6m/b2XtAlrlJW5vzQNIp+WDp5lRVhY+t6aUn/IJXTddFpQVLo2N X-RZG-CLASS-ID: mo00 Received: from localhost (p54AA5E22.dip.t-dialin.net [84.170.94.34]) by post.strato.de (mrclete mo25) (RZmta 24.2) with ESMTP id 2034a5m9V40mDt ; Sun, 31 Oct 2010 06:08:57 +0100 (MET) Date: Sun, 31 Oct 2010 07:08:57 +0100 From: Lutz Steinborn To: Viktor =?UTF-8?B?Qm9qb3ZpxIc=?= Cc: pgsql-sql@postgresql.org Subject: Re: large xml database Message-Id: <20101031070857.a9298792.l.steinborn@4c-ag.de> In-Reply-To: References: Organization: 4c AG X-Mailer: Sylpheed 2.6.0 (GTK+ 2.20.1; x86_64-pc-linux-gnu) Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-5 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201010/78 X-Sequence-Number: 34729 On Sat, 30 Oct 2010 23:49:29 +0200 Viktor Bojović wrote: > > many tries have failed because 8GB of ram and 10gb of swap were not enough. > also sometimes i get that more than 2^32 operations were performed, and > functions stopped to work. > we have a similar problem and we use the Amara xml Toolkit for python. To avoid the big memory consumption use pushbind. A 30G bme catalog file takes a maximum up to 20min to import. It might be faster because we are preparing complex objects with an orm. So the time consumption depends how complex the catalog is. If you use amara only to perform a conversion from xml to csv the final import can be done much faster. regards -- Lutz http://www.4c-gmbh.de