pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedWhat is best way to stream terabytes of data into postgresql?
3+ messages / 3 participants
[nested] [flat]
* What is best way to stream terabytes of data into postgresql?
@ 2005-07-21 15:58 Jeffrey Tenny <jeffrey.tenny@comcast.net>
0 siblings, 1 reply; 3+ messages in thread
From: Jeffrey Tenny @ 2005-07-21 15:58 UTC (permalink / raw)
To: pgsql-performance
Preferably via JDBC, but by C/C++ if necessary.
Streaming being the operative word.
Tips appreciated.
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: What is best way to stream terabytes of data into
@ 2005-07-21 18:26 Frank Wosczyna <frankw@greenplum.com>
0 siblings, 0 replies; 3+ messages in thread
From: Frank Wosczyna @ 2005-07-21 18:26 UTC (permalink / raw)
To: Jeffrey Tenny <jeffrey.tenny@comcast.net>; pgsql-performance
> Subject: [PERFORM] What is best way to stream terabytes of
> data into postgresql?
>
> Preferably via JDBC, but by C/C++ if necessary.
>
> Streaming being the operative word.
>
> Tips appreciated.
>
Hi,
We granted our Java Loader to the Bizgres Open Source,
http://www.bizgres.org/assets/BZ_userguide.htm#50413574_pgfId-110126
You can load from STDIN instead of a file, as long as you prepend the
stream with the Loader Control file, for example:
for name in customer orders lineitem partsupp supplier part;do;cat
TPCH_load_100gb_${name}.ctl /mnt/<remote-host>/TPCH-Data/${name}.tbl.* |
loader.sh -h localhost -p 10001 -d tpch -t -u mpp; done
You can also run the loader from a remote host as well, with the "-h"
<host> being the target system with the Postgres database.
If you have terabytes of data, you might want to set a batch size (-b
switch) to commit occasionally.
Feel free to contact me directly if you have questions.
Thanks,
Frank
Frank Wosczyna
Systems Engineer
Greenplum / Bizgres MPP
www.greenplum.com
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: What is best way to stream terabytes of data into postgresql?
@ 2005-07-22 00:26 Josh Berkus <josh@agliodbs.com>
parent: Jeffrey Tenny <jeffrey.tenny@comcast.net>
0 siblings, 0 replies; 3+ messages in thread
From: Josh Berkus @ 2005-07-22 00:26 UTC (permalink / raw)
To: pgsql-performance; +Cc: Jeffrey Tenny <jeffrey.tenny@comcast.net>
Jeff,
> Streaming being the operative word.
Not sure how much hacking you want to do, but the TelegraphCQ project is
based on PostgreSQL:
http://telegraph.cs.berkeley.edu/telegraphcq/v0.2/
--
--Josh
Josh Berkus
Aglio Database Solutions
San Francisco
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2005-07-22 00:26 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2005-07-21 15:58 What is best way to stream terabytes of data into postgresql? Jeffrey Tenny <jeffrey.tenny@comcast.net>
2005-07-22 00:26 ` Josh Berkus <josh@agliodbs.com>
2005-07-21 18:26 Re: What is best way to stream terabytes of data into Frank Wosczyna <frankw@greenplum.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox