pg.ddx.io  pgsql-performance@postgresql.org mailing list archive  
help / color / mirror / Atom feed
OSS database needed for testing
19+ messages / 9 participants
[nested] [flat]

* OSS database needed for testing
@ 2003-04-03 17:55 Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  0 siblings, 1 reply; 19+ messages in thread

From: Josh Berkus @ 2003-04-03 17:55 UTC (permalink / raw)
  To: pgsql-general@postgresql.org; pgsql-performance; pgsql-hackers@postgresql.org

Folks,

Please pardon the cross-posting.

A small group of us on the Performance list were discussing the first steps 
toward constructing a comprehensive Postgresql installation benchmarking 
tool, mostly to compare different operating systems and file systemsm but 
later to be used as a foundation for a tuning wizard.   

To do this, we need one or more real (not randomly generated*) medium-large 
database which is or can be BSD-licensed (data AND schema).   This database 
must have:

1) At least one "main" table with 12+ columns and 100,000+ rows (each).
2) At least 10-12 additional tables of assorted sizes, at least half of which 
should have Foriegn Key relationships to the main table(s) or each other.
3) At least one large text or varchar field among the various tables.

In addition, the following items would be helpful, but are not required:
4) Views, triggers, and functions built on the database
5) A query log of database activity to give us sample queries to work with.
6) Some complex data types, such as geometric, network, and/or custom data 
types.

Thanks for any leads you can give me!

(* To forestall knee-jerk responses:  Randomly generated data does not look or 
perform the same as real data in my professional opinion, and I'm the one 
writing the test scripts.)

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
@ 2003-04-03 18:26 ` pgsql@mohawksoft.com
  2003-04-03 21:01   ` Re: [PERFORM] [HACKERS] OSS database needed for testing Bruno Wolff III <bruno@wolff.to>
  2003-04-04 02:43   ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
  0 siblings, 2 replies; 19+ messages in thread

From: pgsql@mohawksoft.com @ 2003-04-03 18:26 UTC (permalink / raw)
  To: josh@agliodbs.com; +Cc: pgsql-general@postgresql.org; pgsql-performance; pgsql-hackers@postgresql.org

I don't know that it meets your criteria, but.....

I have a set of scripts and a program that will load the US Census TigerUA
database into PostgreSQL. The thing is absolutely freak'n huge. I forget
which, but it is either 30g or 60g of data excluding indexes.

Also, if that is too much, I have a similar setup to load the FreeDB music
database, from www.freedb.org. It has roughly 670,000 entries in "cdtitles"
and 8 million entries in "cdsongs."

Either one of which, I would be willing to send you the actual DB on cd(s)
if you pay for postage and media. 
 

> Folks,
> 
> Please pardon the cross-posting.
> 
> A small group of us on the Performance list were discussing the first
> steps  toward constructing a comprehensive Postgresql installation
> benchmarking  tool, mostly to compare different operating systems and
> file systemsm but  later to be used as a foundation for a tuning
> wizard.   
> 
> To do this, we need one or more real (not randomly generated*)
> medium-large  database which is or can be BSD-licensed (data AND
> schema).   This database  must have:
> 
> 1) At least one "main" table with 12+ columns and 100,000+ rows (each).
> 2) At least 10-12 additional tables of assorted sizes, at least half of
> which  should have Foriegn Key relationships to the main table(s) or
> each other. 3) At least one large text or varchar field among the
> various tables.
> 
> In addition, the following items would be helpful, but are not
> required: 4) Views, triggers, and functions built on the database
> 5) A query log of database activity to give us sample queries to work
> with. 6) Some complex data types, such as geometric, network, and/or
> custom data  types.
> 
> Thanks for any leads you can give me!
> 
> (* To forestall knee-jerk responses:  Randomly generated data does not
> look or  perform the same as real data in my professional opinion, and
> I'm the one  writing the test scripts.)
> 
> -- 
> -Josh Berkus
>  Aglio Database Solutions
>  San Francisco
> 
> 
> ---------------------------(end of
> broadcast)--------------------------- TIP 1: subscribe and unsubscribe
> commands go to majordomo@postgresql.org




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
@ 2003-04-03 21:01   ` Bruno Wolff III <bruno@wolff.to>
  2003-04-03 22:19     ` Re: [PERFORM] [HACKERS] OSS database needed for testing mlw <pgsql@mohawksoft.com>
  1 sibling, 1 reply; 19+ messages in thread

From: Bruno Wolff III @ 2003-04-03 21:01 UTC (permalink / raw)
  To: pgsql@mohawksoft.com; +Cc: josh@agliodbs.com; pgsql-general@postgresql.org; pgsql-performance; pgsql-hackers@postgresql.org

On Thu, Apr 03, 2003 at 13:26:01 -0500,
  pgsql@mohawksoft.com wrote:
> I don't know that it meets your criteria, but.....
> 
> I have a set of scripts and a program that will load the US Census TigerUA
> database into PostgreSQL. The thing is absolutely freak'n huge. I forget
> which, but it is either 30g or 60g of data excluding indexes.

Are the data model or the loading scripts available publicly?
I have the tiger data and a program that uses it to convert addresses
to latitude and longitude, but I don't really like the program and
was thinking about trying to load the data into a database and do
queries against the database to find location.




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-03 21:01   ` Re: [PERFORM] [HACKERS] OSS database needed for testing Bruno Wolff III <bruno@wolff.to>
@ 2003-04-03 22:19     ` mlw <pgsql@mohawksoft.com>
  2003-04-03 22:20       ` Re: [PERFORM] [HACKERS] OSS database needed for testing Bruno Wolff III <bruno@wolff.to>
  0 siblings, 1 reply; 19+ messages in thread

From: mlw @ 2003-04-03 22:19 UTC (permalink / raw)
  To: Bruno Wolff III <bruno@wolff.to>; +Cc: josh@agliodbs.com; pgsql-general@postgresql.org; pgsql-performance; pgsql-hackers@postgresql.org



Bruno Wolff III wrote:

>On Thu, Apr 03, 2003 at 13:26:01 -0500,
>  pgsql@mohawksoft.com wrote:
>  
>
>>I don't know that it meets your criteria, but.....
>>
>>I have a set of scripts and a program that will load the US Census TigerUA
>>database into PostgreSQL. The thing is absolutely freak'n huge. I forget
>>which, but it is either 30g or 60g of data excluding indexes.
>>    
>>
>
>Are the data model or the loading scripts available publicly?
>I have the tiger data and a program that uses it to convert addresses
>to latitude and longitude, but I don't really like the program and
>was thinking about trying to load the data into a database and do
>queries against the database to find location.
>
>  
>
I have a set of scripts, SQL table defs, a small C program, along with a 
set of field with files that loads it into PGSQL using the "copy from 
stdin" It works fairly well, but takes a good long time to load it all.

Should I put it in the download section of my website?




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-03 21:01   ` Re: [PERFORM] [HACKERS] OSS database needed for testing Bruno Wolff III <bruno@wolff.to>
  2003-04-03 22:19     ` Re: [PERFORM] [HACKERS] OSS database needed for testing mlw <pgsql@mohawksoft.com>
@ 2003-04-03 22:20       ` Bruno Wolff III <bruno@wolff.to>
  0 siblings, 0 replies; 19+ messages in thread

From: Bruno Wolff III @ 2003-04-03 22:20 UTC (permalink / raw)
  To: mlw <pgsql@mohawksoft.com>; +Cc: josh@agliodbs.com; pgsql-general@postgresql.org; pgsql-performance; pgsql-hackers@postgresql.org

On Thu, Apr 03, 2003 at 17:19:13 -0500,
  mlw <pgsql@mohawksoft.com> wrote:
> 
> I have a set of scripts, SQL table defs, a small C program, along with a 
> set of field with files that loads it into PGSQL using the "copy from 
> stdin" It works fairly well, but takes a good long time to load it all.
> 
> Should I put it in the download section of my website?

Yes. I would be interested in looking at it even if I don't use exactly
the same way to do stuff. Taking a logn time to load the data into the
database isn't a big deal for me. reading through the tiger (and FIPS) data
documentation it seemed like there might be some gotchas in unusual cases
and I am not sure the google contest program really handled things right
so I would like to see another implementation. I am also interested in the
data model as that will save me some time.




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
@ 2003-04-04 02:43   ` Jeffrey D. Brower <jeff@pointhere.net>
  2003-04-04 04:29     ` Re: [HACKERS] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  1 sibling, 1 reply; 19+ messages in thread

From: Jeffrey D. Brower @ 2003-04-04 02:43 UTC (permalink / raw)
  To: pgsql@mohawksoft.com; josh@agliodbs.com; +Cc: pgsql-performance

Hi Josh,

Let me vote on the Tiger data.  I used to use this database.  It is public,
updated by the government, VERY useful in own right, it works well with the
earthdistance contribution, a real world database a lot of us use and I
think you can put together some killer scripts on it.

Can I vote twice?  <g>

    Jeff

----- Original Message -----
From: <pgsql@mohawksoft.com>
To: <josh@agliodbs.com>
Cc: <pgsql-general@postgresql.org>; <pgsql-performance@postgresql.org>;
<pgsql-hackers@postgresql.org>
Sent: Thursday, April 03, 2003 1:26 PM
Subject: Re: [PERFORM] [HACKERS] OSS database needed for testing


> I don't know that it meets your criteria, but.....
>
> I have a set of scripts and a program that will load the US Census TigerUA
> database into PostgreSQL. The thing is absolutely freak'n huge. I forget
> which, but it is either 30g or 60g of data excluding indexes.
>
> Also, if that is too much, I have a similar setup to load the FreeDB music
> database, from www.freedb.org. It has roughly 670,000 entries in
"cdtitles"
> and 8 million entries in "cdsongs."
>
> Either one of which, I would be willing to send you the actual DB on cd(s)
> if you pay for postage and media.
>
>
> > Folks,
> >
> > Please pardon the cross-posting.
> >
> > A small group of us on the Performance list were discussing the first
> > steps  toward constructing a comprehensive Postgresql installation
> > benchmarking  tool, mostly to compare different operating systems and
> > file systemsm but  later to be used as a foundation for a tuning
> > wizard.
> >
> > To do this, we need one or more real (not randomly generated*)
> > medium-large  database which is or can be BSD-licensed (data AND
> > schema).   This database  must have:
> >
> > 1) At least one "main" table with 12+ columns and 100,000+ rows (each).
> > 2) At least 10-12 additional tables of assorted sizes, at least half of
> > which  should have Foriegn Key relationships to the main table(s) or
> > each other. 3) At least one large text or varchar field among the
> > various tables.
> >
> > In addition, the following items would be helpful, but are not
> > required: 4) Views, triggers, and functions built on the database
> > 5) A query log of database activity to give us sample queries to work
> > with. 6) Some complex data types, such as geometric, network, and/or
> > custom data  types.
> >
> > Thanks for any leads you can give me!
> >
> > (* To forestall knee-jerk responses:  Randomly generated data does not
> > look or  perform the same as real data in my professional opinion, and
> > I'm the one  writing the test scripts.)
> >
> > --
> > -Josh Berkus
> >  Aglio Database Solutions
> >  San Francisco
> >
> >
> > ---------------------------(end of
> > broadcast)--------------------------- TIP 1: subscribe and unsubscribe
> > commands go to majordomo@postgresql.org
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 2: you can get off all lists at once with the unregister command
>     (send "unregister YourEmailAddressHere" to majordomo@postgresql.org)




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-04 02:43   ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
@ 2003-04-04 04:29     ` Josh Berkus <josh@agliodbs.com>
  2003-04-04 04:54       ` Re: [HACKERS] OSS database needed for testing Tom Lane <tgl@sss.pgh.pa.us>
  2003-04-04 12:21       ` Re: [HACKERS] OSS database needed for testing mlw <pgsql@mohawksoft.com>
  2003-04-04 12:45       ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@Green-Visor.US>
  0 siblings, 3 replies; 19+ messages in thread

From: Josh Berkus @ 2003-04-04 04:29 UTC (permalink / raw)
  To: Jeffrey D. Brower <jeff@pointhere.net>; pgsql@mohawksoft.com; +Cc: pgsql-performance

Jeff,

> Let me vote on the Tiger data.  I used to use this database.  It is public,
> updated by the government, VERY useful in own right, it works well with the
> earthdistance contribution, a real world database a lot of us use and I
> think you can put together some killer scripts on it.

We'd have to use a subset of it.   30G is a little larger than anything we 
want people to download as  a test package.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-04 02:43   ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
  2003-04-04 04:29     ` Re: [HACKERS] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
@ 2003-04-04 04:54       ` Tom Lane <tgl@sss.pgh.pa.us>
  2 siblings, 0 replies; 19+ messages in thread

From: Tom Lane @ 2003-04-04 04:54 UTC (permalink / raw)
  To: Josh Berkus <josh@agliodbs.com>; +Cc: Jeffrey D. Brower <jeff@pointhere.net>; pgsql@mohawksoft.com; pgsql-performance

Josh Berkus <josh@agliodbs.com> writes:
> We'd have to use a subset of it.   30G is a little larger than anything we 
> want people to download as  a test package.

Yeah, it seems a bit over the top ...

The FCC database sounded like an interesting alternative to me.

			regards, tom lane




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-04 02:43   ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
  2003-04-04 04:29     ` Re: [HACKERS] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
@ 2003-04-04 12:21       ` mlw <pgsql@mohawksoft.com>
  2 siblings, 0 replies; 19+ messages in thread

From: mlw @ 2003-04-04 12:21 UTC (permalink / raw)
  To: Josh Berkus <josh@agliodbs.com>; +Cc: Jeffrey D. Brower <jeff@pointhere.net>; pgsql-performance



Josh Berkus wrote:

>Jeff,
>
>  
>
>>Let me vote on the Tiger data.  I used to use this database.  It is public,
>>updated by the government, VERY useful in own right, it works well with the
>>earthdistance contribution, a real world database a lot of us use and I
>>think you can put together some killer scripts on it.
>>    
>>
>
>We'd have to use a subset of it.   30G is a little larger than anything we 
>want people to download as  a test package.
>
>  
>
Actually, come to think of it, the TigerUA DB is in chunks. You can use 
as much or as little as you want. I'll put the loader scripts on my 
download page tonight.

Here is the home page for the data:
http://www.census.gov/geo/www/tiger/tigerua/ua_tgr2k.html
 




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [HACKERS] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-04 02:43   ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
  2003-04-04 04:29     ` Re: [HACKERS] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
@ 2003-04-04 12:45       ` Jeffrey D. Brower <jeff@Green-Visor.US>
  2003-04-04 16:09         ` Re: [PERFORM] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2 siblings, 1 reply; 19+ messages in thread

From: Jeffrey D. Brower @ 2003-04-04 12:45 UTC (permalink / raw)
  To: Josh Berkus <josh@agliodbs.com>; pgsql@mohawksoft.com; +Cc: pgsql-performance

Absolutely.  We could just use one large state or several small ones and let
folks download the whole thing if they wanted.  Using that technique you
could control the size of the test quite closely and still make something
potentially quite valuable as a contribution beyond the bench.

----- Original Message -----
From: "Josh Berkus" <josh@agliodbs.com>
To: "Jeffrey D. Brower" <jeff@pointhere.net>; <pgsql@mohawksoft.com>
Cc: <pgsql-performance@postgresql.org>
Sent: Thursday, April 03, 2003 11:29 PM
Subject: Re: [PERFORM] [HACKERS] OSS database needed for testing


> Jeff,
>
> > Let me vote on the Tiger data.  I used to use this database.  It is
public,
> > updated by the government, VERY useful in own right, it works well with
the
> > earthdistance contribution, a real world database a lot of us use and I
> > think you can put together some killer scripts on it.
>
> We'd have to use a subset of it.   30G is a little larger than anything we
> want people to download as  a test package.
>
> --
> Josh Berkus
> Aglio Database Solutions
> San Francisco




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-04 02:43   ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
  2003-04-04 04:29     ` Re: [HACKERS] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-04 12:45       ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@Green-Visor.US>
@ 2003-04-04 16:09         ` Josh Berkus <josh@agliodbs.com>
  2003-04-04 16:42           ` Re: OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
  0 siblings, 1 reply; 19+ messages in thread

From: Josh Berkus @ 2003-04-04 16:09 UTC (permalink / raw)
  To: Jeffrey D. Brower <jeff@Green-Visor.US>; pgsql@mohawksoft.com; +Cc: pgsql-performance; pgsql-hackers@postgresql.org

Jeff, Mlw, 

> Absolutely.  We could just use one large state or several small ones and
> let folks download the whole thing if they wanted.  Using that technique
> you could control the size of the test quite closely and still make
> something potentially quite valuable as a contribution beyond the bench.

Hold on a second.  The FCC database is still a better choice because it is 
more complex with a carefully defined schema.   The Tiger database would be 
good for doing tests of type 1 and 3, but not for tests of types 2 and 4.

It would certainly be interesting to use the Tiger database as the basis for 
an additional type of test:

6) Very Large Data Set: querying, then updating, 300+ selected rows from a 
2,000,000 + row table.

... but I still see the FCC database as our best candidate for the battery of 
tests 1-5.

-- 
Josh Berkus
Aglio Database Solutions
San Francisco




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: OSS database needed for testing
  2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-03 18:26 ` Re: [HACKERS] OSS database needed for testing pgsql@mohawksoft.com
  2003-04-04 02:43   ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@pointhere.net>
  2003-04-04 04:29     ` Re: [HACKERS] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-04 12:45       ` Re: [HACKERS] OSS database needed for testing Jeffrey D. Brower <jeff@Green-Visor.US>
  2003-04-04 16:09         ` Re: [PERFORM] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
@ 2003-04-04 16:42           ` Jeffrey D. Brower <jeff@pointhere.net>
  0 siblings, 0 replies; 19+ messages in thread

From: Jeffrey D. Brower @ 2003-04-04 16:42 UTC (permalink / raw)
  To: pgsql@mohawksoft.com; Josh Berkus <josh@agliodbs.com>; +Cc: pgsql-performance

I think you got me there.  I have to agree with both points.

(Besides, you are the one coding this thing and I think you understand it
better than I do.)

Let me know if I can help.

    Jeff

----- Original Message -----
From: "Josh Berkus" <josh@agliodbs.com>
To: "Jeffrey D. Brower" <jeff@Green-Visor.US>; <pgsql@mohawksoft.com>
Cc: <pgsql-performance@postgresql.org>; <pgsql-hackers@postgresql.org>
Sent: Friday, April 04, 2003 11:09 AM
Subject: Re: [PERFORM] [HACKERS] OSS database needed for testing


> Jeff, Mlw,
>
> > Absolutely.  We could just use one large state or several small ones and
> > let folks download the whole thing if they wanted.  Using that technique
> > you could control the size of the test quite closely and still make
> > something potentially quite valuable as a contribution beyond the bench.
>
> Hold on a second.  The FCC database is still a better choice because it is
> more complex with a carefully defined schema.   The Tiger database would
be
> good for doing tests of type 1 and 3, but not for tests of types 2 and 4.
>
> It would certainly be interesting to use the Tiger database as the basis
for
> an additional type of test:
>
> 6) Very Large Data Set: querying, then updating, 300+ selected rows from a
> 2,000,000 + row table.
>
> ... but I still see the FCC database as our best candidate for the battery
of
> tests 1-5.
>
> --
> Josh Berkus
> Aglio Database Solutions
> San Francisco
>
>
> ---------------------------(end of broadcast)---------------------------
> TIP 4: Don't 'kill -9' the postmaster




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [HACKERS] OSS database needed for testing
@ 2003-04-03 18:12 Merlin Moncure <merlin.moncure@rcsonline.com>
  0 siblings, 0 replies; 19+ messages in thread

From: Merlin Moncure @ 2003-04-03 18:12 UTC (permalink / raw)
  To: josh@agliodbs.com; +Cc: pgsql-general@postgresql.org; pgsql-performance; pgsql-hackers@postgresql.org

Josh Berkus wrote:
> 1) At least one "main" table with 12+ columns and 100,000+ rows
(each).
> 2) At least 10-12 additional tables of assorted sizes, at least half
of
> which
> should have Foriegn Key relationships to the main table(s) or each
other.
> 3) At least one large text or varchar field among the various tables.
> 
> In addition, the following items would be helpful, but are not
required:
> 4) Views, triggers, and functions built on the database
> 5) A query log of database activity to give us sample queries to work
> with.
> 6) Some complex data types, such as geometric, network, and/or custom
data
> types.
> 
Might I recommend the FCC database of transmitters.  Its publicly
available via anonymous FTP, medium largish with tables running 100k ->
1m+ records, and demonstrates many interesting test cases.  For example,
lat/lon spatial queries (RTree vs. GIST) can be tested with a decent
volume.  Also it demonstrates a good example of the use of schemas.
Email me if you want info.

Format is pipe delimited (non quoted), and data turnover is < 1% a week.

Merlin




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] OSS database needed for testing
@ 2003-04-04 16:47 Merlin Moncure <merlin.moncure@rcsonline.com>
  2003-04-04 17:37 ` Re: [PERFORM] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  2003-04-04 19:08 ` Re: [PERFORM] OSS database needed for testing Lamar Owen <lamar.owen@wgcr.org>
  0 siblings, 2 replies; 19+ messages in thread

From: Merlin Moncure @ 2003-04-04 16:47 UTC (permalink / raw)
  To: Josh Berkus <josh@agliodbs.com>; +Cc: pgsql-hackers@postgresql.org

The fcc FTP site is ftp.fcc.gov

The location of the data of interest is at
/pub/Bureaus/Wireless/Databases/uls/.

There are zip files (pipe delimited) in complete and the daily changed
files in daily.  Theres lots of info in documentation which includes
excel spreadsheets of the schema.  These will have to be converted to
sql statemtents.

The ULS is the database system that holds the data for Fixed and Mobile
wireless services.  This includes most two way systems and point to
multipoint (microwave) but not broadcast (AM, FM, TV) and not advanced
radio.

The database is really a database of applications.  It contains
application data submitted by wireless applicants.  

There are two families of tables, prefixed with 'a' and 'l'.  The 'a'
tables stand for application records that are pending being granted by
the fcc.  The 'l' tables have received licenses and may or may not be
operating.

Combined, the 'a' and 'l' zipfiles represent a specific service.  For
example, 'a_micro' and 'l_micro' contain the applications and licensed
data for microwave systems.  The different services have slightly
different layouts because they have different requirements.

I strongly suggest looking at LMcomm and LMpriv first.  These are the
fixed land mobile systems, and 90% of the entire database.  They also
have identical layouts.

There are a great deal of files in each zipfile, but here are the most
interesting:

hd: header data
ad: application detail
an: antenna data
lo: location data
fr: frequency data
em: emission data

There are others.  I can help you write meaningful queries that are
quite complex and will require optimization techniques.

Merlin




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] OSS database needed for testing
  2003-04-04 16:47 Re: [PERFORM] OSS database needed for testing Merlin Moncure <merlin.moncure@rcsonline.com>
@ 2003-04-04 17:37 ` Josh Berkus <josh@agliodbs.com>
  1 sibling, 0 replies; 19+ messages in thread

From: Josh Berkus @ 2003-04-04 17:37 UTC (permalink / raw)
  To: Merlin Moncure <merlin.moncure@rcsonline.com>; +Cc: pgsql-hackers@postgresql.org; pgsql-performance

Merlin,

> The fcc FTP site is ftp.fcc.gov
> 
> The location of the data of interest is at
> /pub/Bureaus/Wireless/Databases/uls/.

Cool.   I'll tackle this in a week or two.  Right now, I'm being paid to 
convert a client's data and that'll keep me busy through the weekend ...

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] OSS database needed for testing
  2003-04-04 16:47 Re: [PERFORM] OSS database needed for testing Merlin Moncure <merlin.moncure@rcsonline.com>
@ 2003-04-04 19:08 ` Lamar Owen <lamar.owen@wgcr.org>
  2003-04-04 19:27   ` Re: [PERFORM] OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  1 sibling, 1 reply; 19+ messages in thread

From: Lamar Owen @ 2003-04-04 19:08 UTC (permalink / raw)
  To: Merlin Moncure <merlin.moncure@rcsonline.com>; Josh Berkus <josh@agliodbs.com>; +Cc: pgsql-hackers@postgresql.org

On Friday 04 April 2003 11:47, Merlin Moncure wrote:
> The location of the data of interest is at
> /pub/Bureaus/Wireless/Databases/uls/.

> wireless services.  This includes most two way systems and point to
> multipoint (microwave) but not broadcast (AM, FM, TV) and not advanced
> radio.

Also check out the cdbs files (which contain the broadcast stuff as well as 
more) at /pub/Bureaus/Mass_Media/Databases/cdbs/ (which I would be more 
interested in doing, since I am a broadcast engineer by profession....)
-- 
Lamar Owen
WGCR Internet Radio
1 Peter 4:11




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] OSS database needed for testing
  2003-04-04 16:47 Re: [PERFORM] OSS database needed for testing Merlin Moncure <merlin.moncure@rcsonline.com>
  2003-04-04 19:08 ` Re: [PERFORM] OSS database needed for testing Lamar Owen <lamar.owen@wgcr.org>
@ 2003-04-04 19:27   ` Josh Berkus <josh@agliodbs.com>
  0 siblings, 0 replies; 19+ messages in thread

From: Josh Berkus @ 2003-04-04 19:27 UTC (permalink / raw)
  To: Lamar Owen <lamar.owen@wgcr.org>; Merlin Moncure <merlin.moncure@rcsonline.com>; +Cc: pgsql-hackers@postgresql.org

Lamar,

> Also check out the cdbs files (which contain the broadcast stuff as well as 
> more) at /pub/Bureaus/Mass_Media/Databases/cdbs/ (which I would be more 
> interested in doing, since I am a broadcast engineer by profession....)

Hey, if you're willing to do the text --> postgres conversions, I'll use 
whichever tables you want ...

-- 
-Josh Berkus
 Aglio Database Solutions
 San Francisco




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: [PERFORM] OSS database needed for testing
@ 2003-04-04 18:00 Merlin Moncure <merlin.moncure@rcsonline.com>
  2003-04-04 18:07 ` Re: OSS database needed for testing Josh Berkus <josh@agliodbs.com>
  0 siblings, 1 reply; 19+ messages in thread

From: Merlin Moncure @ 2003-04-04 18:00 UTC (permalink / raw)
  To: josh@agliodbs.com; +Cc: pgsql-hackers@postgresql.org; pgsql-performance

Josh Berkus wrote:
> Cool.   I'll tackle this in a week or two.  Right now, I'm being paid
to
> convert a client's data and that'll keep me busy through the weekend
...

I would suggest downloading the data now.  I can help get you started
with the create table statements and the import scripts.  There are not
very many ways to get the data in a reasonable timeframe: the spi
functions or the copy command are a good place to start.  Do not bother
with running stuff through insert queries: take my word for it, it just
won't work.  Of course, if you use copy, you have to pre-format.  Be
aware that you will have many gigabytes (like more than 20) of data
before you are done.

Whatever you decide to do, document the process: the difficulty of
getting large amounts of data into postgres quickly and easily has been
a historical complaint of mine.  Using mysql, it was a snap to get the
data in but using *that* database I really felt it couldn't handle this
much data.
  
I can also get you started with some example queries that should be
quite a challenge to set up to run quickly.  After that, it's your
ballgame.

Merlin




^ permalink  raw  reply  [nested|flat] 19+ messages in thread

* Re: OSS database needed for testing
  2003-04-04 18:00 Re: [PERFORM] OSS database needed for testing Merlin Moncure <merlin.moncure@rcsonline.com>
@ 2003-04-04 18:07 ` Josh Berkus <josh@agliodbs.com>
  0 siblings, 0 replies; 19+ messages in thread

From: Josh Berkus @ 2003-04-04 18:07 UTC (permalink / raw)
  To: Merlin Moncure <merlin.moncure@rcsonline.com>; +Cc: pgsql-performance

Merlin, 

> I would suggest downloading the data now.  I can help get you started

OK, downloading now.

> with the create table statements and the import scripts.  There are not
> very many ways to get the data in a reasonable timeframe: the spi
> functions or the copy command are a good place to start.  Do not bother
> with running stuff through insert queries: take my word for it, it just
> won't work.  Of course, if you use copy, you have to pre-format.  Be
> aware that you will have many gigabytes (like more than 20) of data
> before you are done.



^ permalink  raw  reply  [nested|flat] 19+ messages in thread


end of thread, other threads:[~2003-04-04 19:27 UTC | newest]

Thread overview: 19+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2003-04-03 17:55 OSS database needed for testing Josh Berkus <josh@agliodbs.com>
2003-04-03 18:26 ` pgsql@mohawksoft.com
2003-04-03 21:01   ` Bruno Wolff III <bruno@wolff.to>
2003-04-03 22:19     ` mlw <pgsql@mohawksoft.com>
2003-04-03 22:20       ` Bruno Wolff III <bruno@wolff.to>
2003-04-04 02:43   ` Jeffrey D. Brower <jeff@pointhere.net>
2003-04-04 04:29     ` Josh Berkus <josh@agliodbs.com>
2003-04-04 04:54       ` Tom Lane <tgl@sss.pgh.pa.us>
2003-04-04 12:21       ` mlw <pgsql@mohawksoft.com>
2003-04-04 12:45       ` Jeffrey D. Brower <jeff@Green-Visor.US>
2003-04-04 16:09         ` Josh Berkus <josh@agliodbs.com>
2003-04-04 16:42           ` Jeffrey D. Brower <jeff@pointhere.net>
2003-04-03 18:12 Re: [HACKERS] OSS database needed for testing Merlin Moncure <merlin.moncure@rcsonline.com>
2003-04-04 16:47 Re: [PERFORM] OSS database needed for testing Merlin Moncure <merlin.moncure@rcsonline.com>
2003-04-04 17:37 ` Josh Berkus <josh@agliodbs.com>
2003-04-04 19:08 ` Lamar Owen <lamar.owen@wgcr.org>
2003-04-04 19:27   ` Josh Berkus <josh@agliodbs.com>
2003-04-04 18:00 Re: [PERFORM] OSS database needed for testing Merlin Moncure <merlin.moncure@rcsonline.com>
2003-04-04 18:07 ` Josh Berkus <josh@agliodbs.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox