pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
looking for large dataset
4+ messages / 4 participants
[nested] [flat]

* looking for large dataset
@ 2003-05-03 05:52  Antoine <asolomon15@nyc.rr.com>
  0 siblings, 2 replies; 4+ messages in thread

From: Antoine @ 2003-05-03 05:52 UTC (permalink / raw)
  To: pgsql-performance

I was woundering where could I find a nice large dataset.  Perhaps 50
thousand records or more 
-- 
Antoine <asolomon15@nyc.rr.com>




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: looking for large dataset
@ 2003-05-03 10:04  Christopher Kings-Lynne <chriskl@familyhealth.com.au>
  parent: Antoine <asolomon15@nyc.rr.com>
  1 sibling, 1 reply; 4+ messages in thread

From: Christopher Kings-Lynne @ 2003-05-03 10:04 UTC (permalink / raw)
  To: Antoine <asolomon15@nyc.rr.com>; +Cc: pgsql-performance

That's a very small dataset :)

Chris

On 3 May 2003, Antoine wrote:

> I was woundering where could I find a nice large dataset.  Perhaps 50
> thousand records or more
> --
> Antoine <asolomon15@nyc.rr.com>
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
>




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: looking for large dataset
@ 2003-05-03 15:13  Becky Neville <rebecca.neville@yale.edu>
  parent: Christopher Kings-Lynne <chriskl@familyhealth.com.au>
  0 siblings, 0 replies; 4+ messages in thread

From: Becky Neville @ 2003-05-03 15:13 UTC (permalink / raw)
  To: Christopher Kings-Lynne <chriskl@familyhealth.com.au>; +Cc: Antoine <asolomon15@nyc.rr.com>; pgsql-performance

If you can create a flat file with some rows, it's pretty easy to 
duplicate them as many times as you need to get up to 50k (which, as 
previously mentioned, is relatively small)

This might not work if you need "real" data - but I started with 67k rows 
of real data in my table, then copied them to a temp table, 
updated the 3 key fields with previous value + max value, 
and inserted back into the original table.  (Just to ensure my new rows 
had new values for those 3 fields.)

On Sat, 3 May 2003, Christopher Kings-Lynne wrote:

> That's a very small dataset :)
> 
> Chris
> 
> On 3 May 2003, Antoine wrote:
> 
> > I was woundering where could I find a nice large dataset.  Perhaps 50
> > thousand records or more
> > --
> > Antoine <asolomon15@nyc.rr.com>
> >
> >
> > ---------------------------(end of broadcast)---------------------------
> > TIP 2: you can get off all lists at once with the unregister command
> >     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)
> >
> 
> 
> ---------------------------(end of broadcast)---------------------------
> TIP 5: Have you checked our extensive FAQ?
> 
> http://www.postgresql.org/docs/faqs/FAQ.html
> 




^ permalink  raw  reply  [nested|flat] 4+ messages in thread

* Re: looking for large dataset
@ 2003-05-05 16:25  scott.marlowe <scott.marlowe@ihs.com>
  parent: Antoine <asolomon15@nyc.rr.com>
  1 sibling, 0 replies; 4+ messages in thread

From: scott.marlowe @ 2003-05-05 16:25 UTC (permalink / raw)
  To: Antoine <asolomon15@nyc.rr.com>; +Cc: pgsql-performance

On 3 May 2003, Antoine wrote:

> I was woundering where could I find a nice large dataset.  Perhaps 50
> thousand records or more 

I've attached a PHP script called mktestdb that reads in the dictionary at 
/usr/share/dict/words,  and inserts a user defined number of rows into a 
user defined number of columns.

It's ugly and simple.  Just pipe the output to a text file or psql and off 
you go.

usage:

mktestdb tablename [rows [cols]]

default of 1 column and 1000 rows.

It would be easy enough to rewrite this in something more portable if 
someone wanted to.

Attachments:

mktestdb (text/plain, 1.1K) download #!/usr/local/bin/php -q <?php set_time_limit(1200); if ($argc<2||$argc>4) die("Usage:\n\n mktestdb tablename [rows [cols]]\n\n"); $tablename = $argv[1]; if ($argc>=3) $rows = $argv[2]; else $rows = 1000; if ($argc==4) $cols = $argv[3]; else $cols = 1; function make_seed() { list($usec, $sec) = explode(' ', microtime()); return (float) $sec + ((float) $usec * 100000); } srand(make_seed()); $filename = "/usr/share/dict/words"; $fp = fopen($filename,"r"); $body = fread($fp,filesize($filename)); $lines = explode("\n",$body); $count = count($lines); $query = "create table $tablename (\n"; for ($i=0;$i<$cols;$i++){ $query.="\tcol_".$i; $query.=" text"; if ($i<$cols-1) $query.=", \n"; else $query.="\n"; } $query.=");"; print $query."\n\n"; print "-- Data for $tablename\n\n"; print "COPY $tablename ("; for ($i=0;$i<$cols;$i++){ print "col_".$i; if ($i<$cols-1) print ","; } print ") FROM STDIN;\n"; for ($i=0;$i<$rows;$i++){ for ($j=0;$j<$cols;$j++){ $k=rand(0,$count-1); print $lines[$k]; if ($j<$cols-1) print "\t"; } print "\n"; } print '\.'; print "\n"; ?>
^ permalink raw reply [nested|flat] 4+ messages in thread


end of thread, other threads:[~2003-05-05 16:25 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2003-05-03 05:52 looking for large dataset Antoine <asolomon15@nyc.rr.com>
2003-05-03 10:04 ` Christopher Kings-Lynne <chriskl@familyhealth.com.au>
2003-05-03 15:13   ` Becky Neville <rebecca.neville@yale.edu>
2003-05-05 16:25 ` scott.marlowe <scott.marlowe@ihs.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox