Subject: Location of PostgreSQL Performance Test Lab

From: Josh Berkus <[email-masked]>
Date: Tue, 31 Jul 2007 09:19:10 -0700

Folks,

In addition to the machine Gavin Roy is loaning, I'm putting together an 
online performance test lab for the commmunity using donated equipment.  
Fairly soon, however, I have to decide where it will be hosted.  Since the 
plurality of the equipment is coming from California, I'd prefer something on 
the West Coast, but the critical issue is somewhere where we have ready 
access to sysadmin help to connect things, replace drives etc.

I was thinking of OSL in Corvalis Oregon, for a few reasons:
-- Easy shipping
-- Full-time sysadmin staff
-- One machine is already there
-- Close to Mark Wong
-- Well-known for OSS hosting
However, OSL has the drawback that we would need to raise money to cover power 
costs for the new machines, probably a couple $thousand.  

Other locations which we have here in the US include:
-- ArcServe in San Jose (Sean Chittenden)
-- EHPG in Fremont (Gavin Roy)
-- Someone in Seattle?

Note that I'm talking about between 1/2 and a full rack of equipment here ... 
not just a couple servers.  I'd prefer not to have the servers scattered 
hither and yon, either, if I can avoid it.

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: Peter Eisentraut <[email-masked]>
Date: Tue, 31 Jul 2007 18:30:51 +0200

Am Dienstag, 31. Juli 2007 18:19 schrieb Josh Berkus:
> Fairly soon, however, I have to decide where it will be hosted.

Why is it important that all the machines be hosted at one site?

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

---

From: Josh Berkus <[email-masked]>
Date: Tue, 31 Jul 2007 09:43:18 -0700

Peter,

> Why is it important that all the machines be hosted at one site?

Administration and the ability to to do multi-machine configurations.  Also 
the ability to swap storage to newer machines.

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Tue, 31 Jul 2007 18:48:22 +0200

> costs for the new machines, probably a couple $thousand.  

per year I assume ?

Stefan

---

From: Peter Eisentraut <[email-masked]>
Date: Tue, 31 Jul 2007 19:14:40 +0200

Am Dienstag, 31. Juli 2007 18:43 schrieb Josh Berkus:
> Peter,
>
> > > In addition to the machine Gavin Roy is loaning, I'm putting together
> > > an online performance test lab for the commmunity using donated
> > > equipment. Fairly soon, however, I have to decide where it will be
> > > hosted.
> >
> Administration

Why does administration require that all the machines be hosted at one site?  
Most administration is by remote login, and for the physical maintenance you 
could have someone on site do it.

> and the ability to to do multi-machine configurations.

Well, that is interesting, but not a requirement.  If the machines are not at 
the same site, just don't use them for multi-machine configurations; just 
pick different ones.  (Hey, maybe someone wants to test WAN configurations.)

> Also the ability to swap storage to newer machines.

I don't know what you mean by this.

---

From: Josh Berkus <[email-masked]>
Date: Tue, 31 Jul 2007 10:22:30 -0700

Peter,

> Why does administration require that all the machines be hosted at one
> site? Most administration is by remote login, and for the physical
> maintenance you could have someone on site do it.

Right, and that's made more complex if the main users aren't clear on where 
the machine is.  

> Well, that is interesting, but not a requirement.  If the machines are not
> at the same site, just don't use them for multi-machine configurations;
> just pick different ones.  (Hey, maybe someone wants to test WAN
> configurations.)

Actually, it *is* a requirement.  Doing a real DBT5 or EAStress test run 
requires at least 2 machines, plus storage, and 3-4 machines is more 
interesting.  Having the machines scattered hither and yon will both 
complicate the maintenance of the test lab and limit its usefulness.

I also think you're arguing for the sake of arguing; you don't do performance 
testing and AFAIK are not planning to host any of the machines.  So why are 
you picking nits?

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: Peter Eisentraut <[email-masked]>
Date: Tue, 31 Jul 2007 20:00:38 +0200

Josh Berkus wrote:
> Right, and that's made more complex if the main users aren't clear on
> where the machine is.

You have certainly heard of the world wide web, where you can easily 
publish fabulous lists of information in easily accessible form.

> Actually, it *is* a requirement.  Doing a real DBT5 or EAStress test
> run requires at least 2 machines, plus storage, and 3-4 machines is
> more interesting.  Having the machines scattered hither and yon will
> both complicate the maintenance of the test lab and limit its
> usefulness.

A requirement to have 4 machines at one site to perform a particular 
test is not a logical reason to reject the possibility of having, say, 
20 machines at 5 sites to perform all kinds of tests, when the 
alternative is to have no machines at no sites.

If you can find someone to host everything, great, but you have in fact 
already rejected two machines because they are not at the right 
location, when I can see great use for those machines no matter where 
they are.

> I also think you're arguing for the sake of arguing; you don't do
> performance testing and AFAIK are not planning to host any of the
> machines.

I think you are wrong.

---

From: Josh Berkus <[email-masked]>
Date: Tue, 31 Jul 2007 11:34:52 -0700

Peter,

> alternative is to have no machines at no sites.

Ah, I see the misunderstanding.  Would that we were talking about 20 
machines.  Right now I'm twisting arms to get 5 servers and 4 JBODs, which 
we need in the same location to make use of them.  So we need these 
initial machines in one place, as much as possible, to make use of them.

> location, when I can see great use for those machines no matter where

The peformance testing we can do without storage (drives) is very limited.  
Currently, the only storage I have is located in the US.  In my 
experience, storage is the hardest thing to get donated so I don't really 
expect to find some in Europe soon.

Gavin's machine is the exception becuase it comes with a 28-disk array.  
Still, we could do even more with it if we had another machine or two in 
that location.

-- 
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: "Gavin M. Roy" <[email-masked]>
Date: Tue, 31 Jul 2007 14:36:02 -0400

That could possibly be arranged, let me make some calls... 1/2 cab?

On 7/31/07, Josh Berkus <[email-masked]> wrote:
> ---------------------------(end of broadcast)---------------------------
> TIP 3: Have you checked our extensive FAQ?
>
>                http://www.postgresql.org/docs/faq
>

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 31 Jul 2007 15:44:39 -0300

> publish fabulous lists of information in easily accessible form.

Have to agree with Peter on this one ... not necessarily with as much sarcasm, 
of course ... that one is a weak argument ...

> location, when I can see great use for those machines no matter where

Have to agree here as well ... sounds like you are looking for utopia where we 
generally live in a world of limbo ...

>> performance testing and AFAIK are not planning to host any of the

And on this one, I definitely agree, but for different reasons ... there is 
nobody on this list that a) can provide hosting for this or b) you would host 
with ... and those that will/would do the performance testing don't care in the 
least if the servers are in California or Timbuktoo ... so if you didn't want 
discussion on this, why bother posting it here ... ?

- ----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . [email-masked]                              MSN . [email-masked]
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 31 Jul 2007 17:54:32 -0700

> I don't know what you mean by this.

Often IO has not improved, while the cage has (more ram, more 
processors, better cache etc)... So we can move the 22 spindle array to 
a new machine easily.

Joshua D. Drake

-- 

       === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
Providing the most comprehensive  PostgreSQL solutions since 1997
              http://www.commandprompt.com/

Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 31 Jul 2007 17:56:44 -0700

> That could possibly be arranged, let me make some calls... 1/2 cab?

It should be noted that:

A. Command Prompt has physical access to its location
B. Command Prompt has tons of bandwidth
C. Command Prompt has power
D. Command Prompt has proven a reliable provider
E. Command Prompt loves the PostgreSQL community ;)
F. We would probably have to charge a bit for power cause we are talking 
about some serious machines.
G. We are close to Mark Wong as well

Sincerely,

Joshua D. Drake

> TIP 6: explain analyze is your friend
> 

--

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 31 Jul 2007 17:58:03 -0700

> hither and yon, either, if I can avoid it.

See my post please.

Joshua D. Drake

--

---

From: Josh Berkus <[email-masked]>
Date: Tue, 31 Jul 2007 20:17:48 -0700

JD,

> G. We are close to Mark Wong as well

OK, I'm sold, barring objections from the donors.  Anyone?

-- 
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: Dave Page <[email-masked]>
Date: Wed, 01 Aug 2007 08:45:56 +0100

> OK, I'm sold, barring objections from the donors.  Anyone?

How much of Command Prompt have access to the location? We've been
waiting for a number of weeks to get the new pgFoundry machine plugged
in and switched on because only JD had access to the DC and he was busy
preparing for OSCON.

I don't object in principle, I'd just like some reassurance that we
won't see the perf machines lying idle or awaiting maintenance for the
same or similar reasons.

Regards, Dave

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 1 Aug 2007 10:55:09 +0200

> won't see the perf machines lying idle or awaiting maintenance for the

Yeah, that's especially important given the talk about being able to easily
move disk array between servers and such. While we don't need
30-minute-on-site committment, we do need some level of commitment for
something like that.

//Magnus

---

From: Josh Berkus <[email-masked]>
Date: Wed, 01 Aug 2007 15:31:00 -0700

> won't see the perf machines lying idle or awaiting maintenance for the

That is the advantage of OSL, which has on-site staff and are pretty 
much immediately available during US working hours.

--Josh Berkus

---

From: "Joshua D. Drake" <[email-masked]>
Date: Wed, 01 Aug 2007 21:44:21 -0700

> won't see the perf machines lying idle or awaiting maintenance for the

Sure, but I have mentioned that this would be a bad time of year for the 
pgfoundry machine. Basically I had prep for oscon, oscon, and then I am 
teaching a class ;). This was the *worst* time of year to try and prep 
something new.

But to answer the question, I have to provision the power and rack space 
but other than that we have immediate access to the location that does 
not require me.

Joshua D. Drake

--

---

From: "Joshua D. Drake" <[email-masked]>
Date: Wed, 01 Aug 2007 21:45:09 -0700

> 30-minute-on-site committment, we do need some level of commitment for

As said, it is only for provisioning that I am required.

Joshua D. Drake

> TIP 2: Don't 'kill -9' the postmaster
> 

--

---

From: Dave Page <[email-masked]>
Date: Thu, 02 Aug 2007 09:02:57 +0100

> but other than that we have immediate access to the location that does

So just to be clear, if a couple of disks die and we need to reinstall
the OS on new ones, there are people other than you who can replace the
disks and perform a base install to get us back up and running again
within a day or two? You personally are *only* required for the initial
physical install into the rack?

Regards, Dave

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 2 Aug 2007 10:05:41 +0200

> but other than that we have immediate access to the location that does 

Good - that makes it a much nicer option ;)

But to bug you a bit further - how many other people ware we talking about,
within reasonable distance? Or does the datacenter have on-call people
locally?

(I mean, it's not like we can call on Devrim to go there and plug something
in :-P)

//Magnus

---

From: "Joshua D. Drake" <[email-masked]>
Date: Thu, 02 Aug 2007 16:58:34 -0700

> within reasonable distance? Or does the datacenter have on-call people

We have two people that are CMD people that are local to the machines. 
However the data center has people in house (not just on call but 20 
feet away) and any thing that needs to be done 8-6PST could be done via 
phone.

It is actually how CMD does *everything* now with the exception of 
provisioning which I reserve for myself to make sure everything stays sane.

> (I mean, it's not like we can call on Devrim to go there and plug something

Nope ;) but we can request a plug in or swap or whatever. When the 
machines show up we will label and document everything so it will be 
basically legos for the data center folks.

Oh and just so everyone understands the relationship with the data 
center, I started it... years before CMD existed. I drink beer and golf 
with the owners, I am invited to the weddings etc...

In other words, I trust them, they are good folks.

Regardless, we would also want to get a kvm for the machines.

Joshua D. Drake

> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match
>

---

From: David Fetter <[email-masked]>
Date: Thu, 2 Aug 2007 19:17:45 -0700

On Thu, Aug 02, 2007 at 04:58:34PM -0700, Joshua D. Drake wrote:
> center, I started it... years before CMD existed.  I drink beer and
> golf with the owners, I am invited to the weddings etc...

That's quite adventurous.  What does golf taste like? ;)

> Regardless, we would also want to get a kvm for the machines.

KVM/IP is awesome.  I'm told that some of the really good ones let you
use local media (CD, DVD, etc.) to boot the remote machine.

Cheers,
D
-- 
David Fetter <[email-masked]> http://fetter.org/
phone: +1 415 235 3778        AIM: dfetter666
                              Skype: davidfetter

Remember to vote!
Consider donating to PostgreSQL: http://www.postgresql.org/about/donate

---

From: "Joshua D. Drake" <[email-masked]>
Date: Thu, 02 Aug 2007 20:41:28 -0700

> That's quite adventurous.  What does golf taste like? ;)

Bloody Mary ;)

> use local media (CD, DVD, etc.) to boot the remote machine.

Yes they do :)

Joshua D. Drake

--

---

From: Dave Page <[email-masked]>
Date: Fri, 03 Aug 2007 08:26:23 +0100

> That's quite adventurous.  What does golf taste like? ;)

Oh thats a gem - I nearly fell off my chair!

/D

---

From: Dave Page <[email-masked]>
Date: Fri, 03 Aug 2007 08:42:20 +0100

> provisioning which I reserve for myself to make sure everything stays sane.

OK, then I have no objection to you guys hosting these machines.

Regards, Dave.

---

From: Magnus Hagander <[email-masked]>
Date: Fri, 3 Aug 2007 09:49:12 +0200

> OK, then I have no objection to you guys hosting these machines.

Agreed from here.

//Magnus

---

From: Josh Berkus <[email-masked]>
Date: Fri, 3 Aug 2007 08:41:02 -0700

Folks,

So, we're all good that barring objections from equipment/funding donors, 
we'll rack our equipment at CommandPrompt and list them as a contributor to 
the lab?

Josh, can you figure out what I need to raise in power expenses per year?

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: "Joshua D. Drake" <[email-masked]>
Date: Fri, 03 Aug 2007 08:56:41 -0700

> Josh, can you figure out what I need to raise in power expenses per year?

It won't be much and actually will depend on the "type" of machines 
because we may have to pull in power. E.g; right now we are wired for 
110, but some bigger machines will require 220.

It isn't a big deal either way as we have the power available (I 
actually spoke with them last night when I brought up the new pgfoundry).

As a note, the power is all conditioned and backed by natural gas generator.

Joshua D. Drake

--

---

From: Josh Berkus <[email-masked]>
Date: Fri, 3 Aug 2007 09:54:07 -0700

Josh,

> 110, but some bigger machines will require 220.

Yeah, I think some of the Sun machines I've asked for are 220.  

What's the sort of cost we're looking at for pulling in power?  We don't want 
to do this with CMD if it's going to be *more* expensive than OSL.

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: Josh Berkus <[email-masked]>
Date: Mon, 20 Aug 2007 14:18:58 -0700

Folks,

We never made a final decision on the CMD vs. OSL decision on where to 
house these.  I'll need to ship some boxes this week, so it would be great 
if we could settle this.

Josh, what's our power cost likely to be with you?

-- 
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: "Joshua D. Drake" <[email-masked]>
Date: Mon, 20 Aug 2007 14:26:20 -0700

> Josh, what's our power cost likely to be with you?

I don't have an exact number and wasn't going to worry about it until it
became an issue. I believe the comparative was about 2k a year which we
can beat as I currently have quite the surplus on power.

Sincerely,

Joshua D. Drake

- --

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
			UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

---

From: Dave Page <[email-masked]>
Date: Mon, 20 Aug 2007 22:37:08 +0100

> can beat as I currently have quite the surplus on power.

2K per year!!! What the heck is Josh shipping - a Cray XMP?

/D

---

From: "Marc G. Fournier" <[email-masked]>
Date: Mon, 20 Aug 2007 18:43:30 -0300

> 2K per year!!! What the heck is Josh shipping - a Cray XMP?

- From how Joshua worded it about, my take was that $2k was what OSL quoted
JoshB 
... but, with OSL, was that 'just power', or was that 'power and bandwidth'?

- ----

---

From: Josh Berkus <[email-masked]>
Date: Mon, 20 Aug 2007 14:48:06 -0700

Marc, Josh,

> From how Joshua worded it about, my take was that $2k was what OSL
> quoted JoshB

That was "we'd have to donate a couple $thousand to cover power & help 
getting our rack set up."

-- 
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco