From: Lutz Steinborn <l.steinborn@4c-ag.de>
To: Viktor Bojović <viktor.bojovic@gmail.com>
Cc: pgsql-sql@postgresql.org
Subject: Re: large xml database
Date: Sun, 31 Oct 2010 07:08:57 +0100
Message-ID: <20101031070857.a9298792.l.steinborn@4c-ag.de> (raw)
In-Reply-To: <AANLkTi=ZEO1W=FOxdH+S3twj_A1VCpyyzes7HFghJ7T-@mail.gmail.com>
References: <AANLkTi=ZEO1W=FOxdH+S3twj_A1VCpyyzes7HFghJ7T-@mail.gmail.com>
On Sat, 30 Oct 2010 23:49:29 +0200
Viktor Bojović <viktor.bojovic@gmail.com> wrote:
>
> many tries have failed because 8GB of ram and 10gb of swap were not enough.
> also sometimes i get that more than 2^32 operations were performed, and
> functions stopped to work.
>
we have a similar problem and we use the Amara xml Toolkit for python. To avoid
the big memory consumption use pushbind. A 30G bme catalog file takes a maximum
up to 20min to import. It might be faster because we are preparing complex
objects with an orm. So the time consumption depends how complex the catalog is.
If you use amara only to perform a conversion from xml to csv the final import
can be done much faster.
regards
--
Lutz
http://www.4c-gmbh.de
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-sql@postgresql.org
Cc: l.steinborn@4c-ag.de, viktor.bojovic@gmail.com
Subject: Re: large xml database
In-Reply-To: <20101031070857.a9298792.l.steinborn@4c-ag.de>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox