pg.ddx.io pgsql-sql@postgresql.org mailing list archivehelp / color / mirror / Atom feed
Loading lots of data in a SQL command 2+ messages / 2 participants [nested] [flat]
* Loading lots of data in a SQL command @ 2006-01-04 01:32 frank church <pgsql@adontendev.net> 0 siblings, 1 reply; 2+ messages in thread From: frank church @ 2006-01-04 01:32 UTC (permalink / raw) To: pgsql-sql I am load lots of data via SQL into a database and wrapping it into transactions speeds it up. However this fails a number of times. The queries results are logged so it is easy for me to find problem records. However a single failure causes the whole transaction to fail. Is there a setting or feature that allows which allows the same performance as transactions, without causing the whole process to fail, like a delayed updates or write mechanism of some sort. It is something I would like to set in that particular data looad. Frank ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program. ^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Loading lots of data in a SQL command @ 2006-01-04 09:21 Richard Huxton <dev@archonet.com> parent: frank church <pgsql@adontendev.net> 0 siblings, 0 replies; 2+ messages in thread From: Richard Huxton @ 2006-01-04 09:21 UTC (permalink / raw) To: frank church <pgsql@adontendev.net>; +Cc: pgsql-sql frank church wrote: > I am load lots of data via SQL into a database and wrapping it into transactions > speeds it up. > > However this fails a number of times. The queries results are logged so it is > easy for me to find problem records. > > However a single failure causes the whole transaction to fail. > > Is there a setting or feature that allows which allows the same performance as > transactions, without causing the whole process to fail, like a delayed updates > or write mechanism of some sort. Not as it stands. I tend to use a small perl wrapper myself that loads in batches of e.g. 10000 rows and if there is an error deal with it separately. I seem to recall it being discussed as a built-in feature recently though, so there might be someone working on it for a future version. > It is something I would like to set in that particular data looad. You might find the "pgloader" project meets your needs exactly: http://pgfoundry.org/projects/pgloader/ -- Richard Huxton Archonet Ltd ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2006-01-04 09:21 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2006-01-04 01:32 Loading lots of data in a SQL command frank church <pgsql@adontendev.net> 2006-01-04 09:21 ` Richard Huxton <dev@archonet.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox