pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feeda heavy duty operation on an "unused" table kills my server
25+ messages / 9 participants
[nested] [flat]
* a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 04:59 Eduardo Piombino <drakorg@gmail.com>
0 siblings, 3 replies; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-13 04:59 UTC (permalink / raw)
To: pgsql-performance
Hi list, I'm having a problem when dealing with operations that asks too
much CPU from the server.
The scenario is this:
I have a multithreaded server, each thread with its own connection to the
database. Everything is working fine, actually great, actually
outstandingly, in normal operation.
I've a table named "a" with 1.8 million records, and growing, but I'm ok
with it, at least for the moment. Maybe in the near future we will cut it
down, backup old data, and free it up. But this is not the issue, as I said,
everything is working great. I have a cpl of indexes to help some queries,
and that's it.
Now my problem started when I tried to do some model refactoring on this
production table.
First I tried a dumb approach.
I connected from pgadmin, opened a new session.
I tried an ALTER TABLE on this table just to turn a char(255) field into
char(250), and it locked up my system.
No surprise, since I had many threads waiting for this alter table to
finish. What I did not foresee was that this alter table would take up so
much time. Ok, my fault, for not having calculated the time that it would
take the ALTER TABLE to complete.
Now, with this experience, I tried a simple workaround.
Created an empty version of "a" named "a_empty", identical in every sense.
renamed "a" to "a_full", and "a_empty" to "a". This procedure costed me like
0 seconds of downtime, and everything kept working smoothly. Maybe a cpl of
operations could have failed if they tried to write in the very second that
there was actually no table named "a", but since the operation was
transactional, the worst scenario was that if the operation should have
failed, the client application would just inform of the error and ask the
user for a retry. No big deal.
Now, this table, that is totally unattached to the system in every way (no
one references this table, its like a dumpster for old records), is not
begin accessed by no other thread in the system, so an ALTER table on it, to
turn a char(255) to char(250), should have no effect on the system.
So, with this in mind, I tried the ALTER TABLE this time on the "a_full"
(totally unrelated) table.
The system went non-responsive again, and this time it had nothing to do
with threads waiting for the alter table to complete. The pgAdmin GUI went
non-responsive, as well as the application's server GUI, whose threads kept
working on the background, but starting to take more and more time for every
clients request (up to 25 seconds, which are just ridiculous and completely
unacceptable in normal conditions).
This resulted in my client applications to start disconnecting after their
operations failed due to timeout, and the system basically went down again,
from a users point of view.
This time, since I saw no relation between my operation on a totally
unrelated table, and the server BIG slowdown, I blamed the servers memory.
After some tests, I came up to the conclusion that any heavy duty operation
on any thread (ALTER TABLE on 1.8 million records tables, updates on this
table, or an infinite loop, just to make my point), would affect the whole
server.
Bottom line is, I can't seem to do any heavy processing on the database (or
any operation that would require the server to enter into high CPU usage),
and still expect the server to behave normally. Whatever heavy duty
operation, DDL, DML, on whatever table (related, or unrelated), on whatever
thread, would tear down my servers integrity.
My question then is: is there a way to limit the CPU assigned to a specific
connection?
I mean, I don't care if my ALTER TABLE takes 4 days instead of 4 hours.
Something like:
pg_set_max_cpu_usage(2/100);
and rest assured that no matter what that thread is asking the database to
do, it just wont affect the other running threads. Obviosly, assuring that
the process itself does not involve any locking of the other threads.
Is something like that possible?
Thanks in advance,
Eduardo.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 05:14 Craig James <craig_james@emolecules.com>
parent: Eduardo Piombino <drakorg@gmail.com>
2 siblings, 1 reply; 25+ messages in thread
From: Craig James @ 2010-01-13 05:14 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: pgsql-performance
Eduardo Piombino wrote:
> Hi list, I'm having a problem when dealing with operations that asks too
> much CPU from the server.
> The scenario is this:
A nice description below, but ... you give no information about your system: number of CPUs, disk types and configuration, how much memory, what have you changed in your Postgres configuration? And what operating system, what version of Postgres, etc., etc. The more information you give, the better the answer.
If you're operating on a single disk with a tiny amount of memory, and old, misconfigured Postgres on a laptop computer, that's a whole different problem than if you're on a big sytem with 16 CPUs and a huge RAID 1+0 with battery-backed cache.
Craig
>
> I have a multithreaded server, each thread with its own connection to
> the database. Everything is working fine, actually great, actually
> outstandingly, in normal operation.
>
> I've a table named "a" with 1.8 million records, and growing, but I'm ok
> with it, at least for the moment. Maybe in the near future we will cut
> it down, backup old data, and free it up. But this is not the issue, as
> I said, everything is working great. I have a cpl of indexes to help
> some queries, and that's it.
>
> Now my problem started when I tried to do some model refactoring on this
> production table.
>
> First I tried a dumb approach.
> I connected from pgadmin, opened a new session.
> I tried an ALTER TABLE on this table just to turn a char(255) field into
> char(250), and it locked up my system.
>
> No surprise, since I had many threads waiting for this alter table to
> finish. What I did not foresee was that this alter table would take up
> so much time. Ok, my fault, for not having calculated the time that it
> would take the ALTER TABLE to complete.
>
> Now, with this experience, I tried a simple workaround.
> Created an empty version of "a" named "a_empty", identical in every sense.
> renamed "a" to "a_full", and "a_empty" to "a". This procedure costed me
> like 0 seconds of downtime, and everything kept working smoothly. Maybe
> a cpl of operations could have failed if they tried to write in the very
> second that there was actually no table named "a", but since the
> operation was transactional, the worst scenario was that if the
> operation should have failed, the client application would just inform
> of the error and ask the user for a retry. No big deal.
>
> Now, this table, that is totally unattached to the system in every way
> (no one references this table, its like a dumpster for old records), is
> not begin accessed by no other thread in the system, so an ALTER table
> on it, to turn a char(255) to char(250), should have no effect on the
> system.
>
> So, with this in mind, I tried the ALTER TABLE this time on the "a_full"
> (totally unrelated) table.
> The system went non-responsive again, and this time it had nothing to do
> with threads waiting for the alter table to complete. The pgAdmin GUI
> went non-responsive, as well as the application's server GUI, whose
> threads kept working on the background, but starting to take more and
> more time for every clients request (up to 25 seconds, which are just
> ridiculous and completely unacceptable in normal conditions).
>
> This resulted in my client applications to start disconnecting after
> their operations failed due to timeout, and the system basically went
> down again, from a users point of view.
>
> This time, since I saw no relation between my operation on a totally
> unrelated table, and the server BIG slowdown, I blamed the servers memory.
>
> After some tests, I came up to the conclusion that any heavy duty
> operation on any thread (ALTER TABLE on 1.8 million records tables,
> updates on this table, or an infinite loop, just to make my point),
> would affect the whole server.
>
> Bottom line is, I can't seem to do any heavy processing on the database
> (or any operation that would require the server to enter into high CPU
> usage), and still expect the server to behave normally. Whatever heavy
> duty operation, DDL, DML, on whatever table (related, or unrelated), on
> whatever thread, would tear down my servers integrity.
>
> My question then is: is there a way to limit the CPU assigned to a
> specific connection?
> I mean, I don't care if my ALTER TABLE takes 4 days instead of 4 hours.
>
> Something like:
> pg_set_max_cpu_usage(2/100);
>
> and rest assured that no matter what that thread is asking the database
> to do, it just wont affect the other running threads. Obviosly, assuring
> that the process itself does not involve any locking of the other threads.
>
> Is something like that possible?
>
> Thanks in advance,
> Eduardo.
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 05:41 Craig Ringer <craig@postnewspapers.com.au>
parent: Eduardo Piombino <drakorg@gmail.com>
2 siblings, 0 replies; 25+ messages in thread
From: Craig Ringer @ 2010-01-13 05:41 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: pgsql-performance
On 13/01/2010 12:59 PM, Eduardo Piombino wrote:
> My question then is: is there a way to limit the CPU assigned to a
> specific connection?
> I mean, I don't care if my ALTER TABLE takes 4 days instead of 4 hours.
>
> Something like:
> pg_set_max_cpu_usage(2/100);
You're assuming the issue is CPU. I think that unlikely. In general, a
single thread/process that wants as much CPU as it can get won't bring
any machine with a half-decent OS to its knees. Any UNIX system should
barely notice - everything else will slow down somewhat, depending on
its scheduler, but in any sane setup shouldn't slow down by more than
1/2. Modern Windows tends to be fairly well behaved here too.
What's much more likely is that you're working with a crappy disk setup
- such as a RAID 5 array without battery-backed cache, or a single slow
disk. You probably also have quite deep write queuing in the RAID
controller / disk / OS. This means that your disk-intensive ALTER TABLE
makes your disk subsystem so busy that it takes ages before any other
process gets a look-in. It's not unlikely that I/O requests are being
queued so deeply that it (often) takes several seconds for the
controller to get around to executing a newly submitted read or write
request. If your other queries need to do more than a few steps where
they read some data, think about it, and read other data depending on
the first read, then they're going to take forever, because they're
going to have to ensure a long delay before disk access each time.
Of course, that's just a guess, since you've provided no information on
your hardware. Try collecting up some of the information shown here:
http://wiki.postgresql.org/wiki/Guide_to_reporting_problems#Things_you_need_to_mention
In any case, if it *is* I/O related, what to do about it depends on
exactly what sort of I/O issue it is. Extremely deep queuing? Looks good
for throughput benchmarks, but is stupid if you care about latency and
have some I/O that's higher priority than others, so reduce your queue
depth. Very slow writes hammering reads? Don't use RAID 5. Etc.
--
Craig Ringer
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 05:47 Eduardo Piombino <drakorg@gmail.com>
parent: Craig James <craig_james@emolecules.com>
0 siblings, 3 replies; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-13 05:47 UTC (permalink / raw)
To: Craig James <craig_james@emolecules.com>; +Cc: pgsql-performance
I'm sorry.
The server is a production server HP Proliant, I don't remember the exact
model, but the key features were:
4 cores, over 2GHz each (I'm sorry I don't remember the actual specs), I
think it had 16G of RAM (if that is possible?)
It has two 320G disks in RAID (mirrored).
I don't even have the emails with the specs here, but I can give you the
exact configuration by tomorrow.
Operating system: Windows 2003 server, with latest patches.
Postgres version: 8.2.4, with all defaults, except DateStyle and TimeZone.
At any given time, the server is on 0% CPU load, with peaks of 1%, 2%, max.
In normal operation.
I've been digging a little in the archives, and one thing that it helped me
come up with, is that I don't really remember seeing high CPU usage (fact
that surprised me, but i do remember seeing high IO activity). I'm sorry,
its pretty late here.
I know this single statement is enough to almost change everything I've just
asked.
Please try interpreting again my original mail, considering that when I said
"high CPU usage" It might very well be "high IO usage".
The final effect was that the server went non-responsive, for all matters,
not even the TaskManager would come up when i hit CTRL-ALT-DEL, and of
course, every client would suffer horrific (+20 secs) for the simplest
operations like SELECT NOW();
I've just made a little modification to my original questions, to extend to
the possibility of a IO usage issue, instead of just CPU.
*
*
>
> *Bottom line is, I can't seem to do any heavy processing on the database
> (or any operation that would require the server to enter into high CPU usage
> **or IO USAGE), and still expect the server to behave normally. Whatever
> heavy duty operation, DDL, DML, on whatever table (related, or unrelated),
> on whatever thread, would tear down my servers integrity.*
>
> * My question then is: is there a way to limit the CPU or IO USAGEassigned to a specific connection?
> *
> * I mean, I don't care if my ALTER TABLE takes 4 days instead of 4 hours.*
>
> * Something like:*
> * pg_set_max_cpu _or_io_usage(2/100);*
>
On Wed, Jan 13, 2010 at 2:14 AM, Craig James <craig_james@emolecules.com>wrote:
> Eduardo Piombino wrote:
>
>> Hi list, I'm having a problem when dealing with operations that asks too
>> much CPU from the server.
>> The scenario is this:
>>
>
> A nice description below, but ... you give no information about your
> system: number of CPUs, disk types and configuration, how much memory, what
> have you changed in your Postgres configuration? And what operating system,
> what version of Postgres, etc., etc. The more information you give, the
> better the answer.
>
> If you're operating on a single disk with a tiny amount of memory, and old,
> misconfigured Postgres on a laptop computer, that's a whole different
> problem than if you're on a big sytem with 16 CPUs and a huge RAID 1+0 with
> battery-backed cache.
>
Craig
>
>
>
>> I have a multithreaded server, each thread with its own connection to the
>> database. Everything is working fine, actually great, actually
>> outstandingly, in normal operation.
>>
>> I've a table named "a" with 1.8 million records, and growing, but I'm ok
>> with it, at least for the moment. Maybe in the near future we will cut it
>> down, backup old data, and free it up. But this is not the issue, as I said,
>> everything is working great. I have a cpl of indexes to help some queries,
>> and that's it.
>>
>> Now my problem started when I tried to do some model refactoring on this
>> production table.
>>
>> First I tried a dumb approach.
>> I connected from pgadmin, opened a new session.
>> I tried an ALTER TABLE on this table just to turn a char(255) field into
>> char(250), and it locked up my system.
>>
>> No surprise, since I had many threads waiting for this alter table to
>> finish. What I did not foresee was that this alter table would take up so
>> much time. Ok, my fault, for not having calculated the time that it would
>> take the ALTER TABLE to complete.
>>
>> Now, with this experience, I tried a simple workaround.
>> Created an empty version of "a" named "a_empty", identical in every sense.
>> renamed "a" to "a_full", and "a_empty" to "a". This procedure costed me
>> like 0 seconds of downtime, and everything kept working smoothly. Maybe a
>> cpl of operations could have failed if they tried to write in the very
>> second that there was actually no table named "a", but since the operation
>> was transactional, the worst scenario was that if the operation should have
>> failed, the client application would just inform of the error and ask the
>> user for a retry. No big deal.
>>
>> Now, this table, that is totally unattached to the system in every way (no
>> one references this table, its like a dumpster for old records), is not
>> begin accessed by no other thread in the system, so an ALTER table on it, to
>> turn a char(255) to char(250), should have no effect on the system.
>>
>> So, with this in mind, I tried the ALTER TABLE this time on the "a_full"
>> (totally unrelated) table.
>> The system went non-responsive again, and this time it had nothing to do
>> with threads waiting for the alter table to complete. The pgAdmin GUI went
>> non-responsive, as well as the application's server GUI, whose threads kept
>> working on the background, but starting to take more and more time for every
>> clients request (up to 25 seconds, which are just ridiculous and completely
>> unacceptable in normal conditions).
>>
>> This resulted in my client applications to start disconnecting after their
>> operations failed due to timeout, and the system basically went down again,
>> from a users point of view.
>>
>> This time, since I saw no relation between my operation on a totally
>> unrelated table, and the server BIG slowdown, I blamed the servers memory.
>>
>> After some tests, I came up to the conclusion that any heavy duty
>> operation on any thread (ALTER TABLE on 1.8 million records tables, updates
>> on this table, or an infinite loop, just to make my point), would affect the
>> whole server.
>>
>> Bottom line is, I can't seem to do any heavy processing on the database
>> (or any operation that would require the server to enter into high CPU usage
>> *or IO USAGE*), and still expect the server to behave normally. Whatever
>> heavy duty operation, DDL, DML, on whatever table (related, or unrelated),
>> on whatever thread, would tear down my servers integrity.
>>
>> My question then is: is there a way to limit the CPU* or **IO USAGE* assigned
>> to a specific connection?
>> I mean, I don't care if my ALTER TABLE takes 4 days instead of 4 hours.
>>
>> Something like:
>> pg_set_max_cpu*_or_io_*usage(2/100);
>>
>> and rest assured that no matter what that thread is asking the database to
>> do, it just wont affect the other running threads. Obviosly, assuring that
>> the process itself does not involve any locking of the other threads.
>>
>> Is something like that possible?
>>
>> Thanks in advance,
>> Eduardo.
>>
>>
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 06:02 Craig Ringer <craig@postnewspapers.com.au>
parent: Eduardo Piombino <drakorg@gmail.com>
2 siblings, 1 reply; 25+ messages in thread
From: Craig Ringer @ 2010-01-13 06:02 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: Craig James <craig_james@emolecules.com>; pgsql-performance
On 13/01/2010 1:47 PM, Eduardo Piombino wrote:
> I'm sorry.
>
> The server is a production server HP Proliant, I don't remember the
> exact model, but the key features were:
> 4 cores, over 2GHz each (I'm sorry I don't remember the actual specs), I
> think it had 16G of RAM (if that is possible?)
> It has two 320G disks in RAID (mirrored).
Plain 'ol SATA disks in RAID-1?
Hardware RAID (and if so, controller model)? With battery backup? Write
cache on or off?
Or software RAID? If so, Windows build-in sw raid, or some vendor's
fakeraid (Highpoint, Promise, Adaptec, etc) ?
Anyway, with two disks in RAID-1 I'm not surprised you're seeing some
performance issues with heavy writes, especially since it seems unlikely
that you have a BBU hardware RAID controller. In RAID-1 a write must hit
both disks, so a 1Mb write effectively costs twice as much as a 1Mb
read. Since many controllers try for high throughput (because it looks
good in benchmarks) at the expense of latency they also tend to try to
batch writes into long blocks, which keeps the disks busy in extended
bursts. That slaughters read latencies.
I had this sort of issue with a 3Ware 8500-8, and landed up modifying
and recompiling the driver to reduce its built-in queue depth. I also
increased readahead. It was still pretty awful as I was working with
RAID 5 on SATA disks, but it made a big difference and more importantly
meant that my Linux server was able to honour `ionice' priorities and
feed more important requests to the controller first.
On windows, I really don't know what to do about it beyond getting a
better I/O subsystem. Google may help - look into I/O priorities, queue
depths, reducing read latencies, etc.
> I don't even have the emails with the specs here, but I can give you the
> exact configuration by tomorrow.
>
> Operating system: Windows 2003 server, with latest patches.
> Postgres version: 8.2.4, with all defaults, except DateStyle and TimeZone.
Urk. 8.2 ?
Pg on Windows improves a lot with each release, and that's an old buggy
version of 8.2 at that. Looking into an upgrade would be a really,
REALLY good idea.
> Please try interpreting again my original mail, considering that when I
> said "high CPU usage" It might very well be "high IO usage".
>
> The final effect was that the server went non-responsive, for all
> matters, not even the TaskManager would come up when i hit CTRL-ALT-DEL,
> and of course, every client would suffer horrific (+20 secs) for the
> simplest operations like SELECT NOW();
That sounds a LOT like horrible read latencies caused by total I/O
overload. It could also be running out of memory and swapping heavily,
so do keep an eye out for that, but I wouldn't expect to see that with
an ALTER TABLE - especially on a 16GB server.
> / My question then is: is there a way to limit the CPU* or **IO
> USAGE* assigned to a specific connection?/
In win32 you can set CPU priorities manually in Task Manager, but only
once you already know the process ID of the Pg backend that's going to
be hammering the machine. Not helpful.
I don't know of any way to do per-process I/O priorities in Win32, but I
only use win32 reluctantly and don't use it for anything I care about
(like a production Pg server) so I'm far from a definitive source.
--
Craig Ringer
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 07:03 Eduardo Piombino <drakorg@gmail.com>
parent: Craig Ringer <craig@postnewspapers.com.au>
0 siblings, 2 replies; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-13 07:03 UTC (permalink / raw)
To: pgsql-performance
Excellent, lots of useful information in your message.
I will follow your advices, and keep you posted on any progress. I have yet
to confirm you with some technical details of my setup, but I'm pretty sure
you hit the nail in any case.
One last question, this IO issue I'm facing, do you think it is just a
matter of RAID configuration speed, or a matter of queue gluttony (and not
leaving time for other processes to get into the IO queue in a reasonable
time)?
Because if it was just a matter of speed, ok, with my actual RAID
configuration lets say it takes 10 minutes to process the ALTER TABLE
(leaving no space to other IOs until the ALTER TABLE is done), lets say then
i put the fastest possible RAID setup, or even remove RAID for the sake of
speed, and it completes in lets say again, 10 seconds (an unreal
assumption). But if my table now grows 60 times, I would be facing the very
same problem again, even with the best RAID configuration.
The problem would seem to be in the way the OS (or hardware, or someone
else, or all of them) is/are inserting the IO requests into the queue.
What can I do to control the order in which these IO requests are finally
entered into the queue?
I mean .. what i would like to obtain is:
Considering the ALTER TABLE as a sequence of 100.000 READ/WRITE OPERATIONS
Considering the SELECT * FROM xxx as a sequence of 100 READ OPERATIONS
(totally unrelated in disk)
First i run the ALTER TABLE on a thread...
Lets say by the time it generates 1.000 READ/WRITE OPERATIONS, the other
thread starts with the SELECT * FROM xxx ...
I would expect the IO system to give chance to the those 100 READ OPERATIONS
to execute immediately (with no need to wait for the remaining 990.000
READ/WRITE OPERATIONS finish), that is, to enter the queue at *almost* the
very same moment the IO request were issued.
If I can not guarantee that, I'm kinda doomed, because the largest the
amount of IO operations requested by a "heavy duty operation", the longest
it will take any other thread to start doing anything.
What cards do I have to manipulate the order the IO requests are entered
into the "queue"?
Can I disable this queue?
Should I turn disk's IO operation caches off?
Not use some specific disk/RAID vendor, for instance?
I think I have some serious reading to do on this matter, google will help
of course, but as always, every advice for small it may seem, will be very
much appreciated.
Nonetheless, thanks a lot for all the light you already brought me on this
matter.
I really appreciate it.
Eduardo.
On Wed, Jan 13, 2010 at 3:02 AM, Craig Ringer
<craig@postnewspapers.com.au>wrote:
> On 13/01/2010 1:47 PM, Eduardo Piombino wrote:
>
>> I'm sorry.
>>
>> The server is a production server HP Proliant, I don't remember the
>> exact model, but the key features were:
>> 4 cores, over 2GHz each (I'm sorry I don't remember the actual specs), I
>> think it had 16G of RAM (if that is possible?)
>> It has two 320G disks in RAID (mirrored).
>>
>
> Plain 'ol SATA disks in RAID-1?
>
> Hardware RAID (and if so, controller model)? With battery backup? Write
> cache on or off?
>
> Or software RAID? If so, Windows build-in sw raid, or some vendor's
> fakeraid (Highpoint, Promise, Adaptec, etc) ?
>
> Anyway, with two disks in RAID-1 I'm not surprised you're seeing some
> performance issues with heavy writes, especially since it seems unlikely
> that you have a BBU hardware RAID controller. In RAID-1 a write must hit
> both disks, so a 1Mb write effectively costs twice as much as a 1Mb read.
> Since many controllers try for high throughput (because it looks good in
> benchmarks) at the expense of latency they also tend to try to batch writes
> into long blocks, which keeps the disks busy in extended bursts. That
> slaughters read latencies.
>
> I had this sort of issue with a 3Ware 8500-8, and landed up modifying and
> recompiling the driver to reduce its built-in queue depth. I also increased
> readahead. It was still pretty awful as I was working with RAID 5 on SATA
> disks, but it made a big difference and more importantly meant that my Linux
> server was able to honour `ionice' priorities and feed more important
> requests to the controller first.
>
> On windows, I really don't know what to do about it beyond getting a better
> I/O subsystem. Google may help - look into I/O priorities, queue depths,
> reducing read latencies, etc.
>
>
> I don't even have the emails with the specs here, but I can give you the
>> exact configuration by tomorrow.
>>
>> Operating system: Windows 2003 server, with latest patches.
>> Postgres version: 8.2.4, with all defaults, except DateStyle and TimeZone.
>>
>
> Urk. 8.2 ?
>
> Pg on Windows improves a lot with each release, and that's an old buggy
> version of 8.2 at that. Looking into an upgrade would be a really, REALLY
> good idea.
>
>
> Please try interpreting again my original mail, considering that when I
>> said "high CPU usage" It might very well be "high IO usage".
>>
>> The final effect was that the server went non-responsive, for all
>> matters, not even the TaskManager would come up when i hit CTRL-ALT-DEL,
>> and of course, every client would suffer horrific (+20 secs) for the
>> simplest operations like SELECT NOW();
>>
>
> That sounds a LOT like horrible read latencies caused by total I/O
> overload. It could also be running out of memory and swapping heavily, so do
> keep an eye out for that, but I wouldn't expect to see that with an ALTER
> TABLE - especially on a 16GB server.
>
> / My question then is: is there a way to limit the CPU* or **IO
>> USAGE* assigned to a specific connection?/
>>
>
> In win32 you can set CPU priorities manually in Task Manager, but only once
> you already know the process ID of the Pg backend that's going to be
> hammering the machine. Not helpful.
>
> I don't know of any way to do per-process I/O priorities in Win32, but I
> only use win32 reluctantly and don't use it for anything I care about (like
> a production Pg server) so I'm far from a definitive source.
>
> --
> Craig Ringer
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 07:39 Greg Smith <greg@2ndquadrant.com>
parent: Eduardo Piombino <drakorg@gmail.com>
2 siblings, 1 reply; 25+ messages in thread
From: Greg Smith @ 2010-01-13 07:39 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: Craig James <craig_james@emolecules.com>; pgsql-performance
Eduardo Piombino wrote:
> Postgres version: 8.2.4, with all defaults, except DateStyle and TimeZone.
Ugh...there are several features in PostgreSQL 8.3 and later
specifically to address the sort of issue you're running into. If you
want to get good write performance out of this system, you may need to
upgrade to at least that version. It's impossible to resolve several of
the common problems in write operations being too intense using any 8.2
version.
> The final effect was that the server went non-responsive, for all
> matters, not even the TaskManager would come up when i hit
> CTRL-ALT-DEL, and of course, every client would suffer horrific (+20
> secs) for the simplest operations like SELECT NOW();
The thing that you have to realize is that altering a table is basically
making a new copy of that table, which is a really heavy amount of
writing. It's quite easy for an I/O heavy operation like that to fill
up a lot of RAM with data to be written out, and when the database
periodically needs to force all that data out to disk the whole system
grinds to a halt when it happens. There's no way I'm aware of to
throttle that writing down to a reasonable amount under Windows either,
to achieve your goal of just making the ALTER run using less resources.
Some reading:
http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server goes over
basic tuning of the database server. If you haven't already increased
the checkpoint_segments parameters of your system, that's the first
thing to try--increase it *a lot* (32 or more, default is 3) because it
can really help with this problem. A moderate increase to
shared_buffers is in order too; since you're on Windows, increasing it
to 256MB is a reasonable change. The rest of the changes in there
aren't likely to help out with this specific problem.
http://www.westnet.com/~gsmith/content/postgresql/chkp-bgw-83.htm :
covers the most likely cause of the issue you're running into.
Unfortunately, most of the solutions you'll see there are things changed
in 8.3.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 08:27 Eduardo Piombino <drakorg@gmail.com>
parent: Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-13 08:27 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Craig James <craig_james@emolecules.com>; pgsql-performance
Yes, one of the things I will do asap is to migrate to the latest version.
On other occasion I went through the checkpoint parameters you mentioned,
but left them untouched since they seemed logical.
I'm a little reluctant of changing the checkpoint configuration just to let
me do a -once in a lifetime- ALTER.
The checkpoints would then remain too far away in time (or in traffic).
And thinking of touching it and retouching it every time I need to do sthing
different bugs me a little. But if there is no other option I will
definitely give it a try.
Are you sure, for instance, that the ALTER command (and the internal data it
may require to handle, lets say 1.8 million records * 1024 bytes/record
(aprox)) goes to RAM, then to disk, and gets logged in the WAL during the
whole process? Maybe it does not get logged at all until the ALTER is
completed? Since the original table can be left untouched until this copy of
the table gets updated ... Just guessing here.
On Wed, Jan 13, 2010 at 4:39 AM, Greg Smith <greg@2ndquadrant.com> wrote:
> Eduardo Piombino wrote:
>
>> Postgres version: 8.2.4, with all defaults, except DateStyle and TimeZone.
>>
>
> Ugh...there are several features in PostgreSQL 8.3 and later specifically
> to address the sort of issue you're running into. If you want to get good
> write performance out of this system, you may need to upgrade to at least
> that version. It's impossible to resolve several of the common problems in
> write operations being too intense using any 8.2 version.
>
>> The final effect was that the server went non-responsive, for all matters,
>> not even the TaskManager would come up when i hit CTRL-ALT-DEL, and of
>> course, every client would suffer horrific (+20 secs) for the simplest
>> operations like SELECT NOW();
>>
>
> The thing that you have to realize is that altering a table is basically
> making a new copy of that table, which is a really heavy amount of writing.
> It's quite easy for an I/O heavy operation like that to fill up a lot of
> RAM with data to be written out, and when the database periodically needs to
> force all that data out to disk the whole system grinds to a halt when it
> happens. There's no way I'm aware of to throttle that writing down to a
> reasonable amount under Windows either, to achieve your goal of just making
> the ALTER run using less resources.
>
> Some reading:
>
> http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server goes over
> basic tuning of the database server. If you haven't already increased the
> checkpoint_segments parameters of your system, that's the first thing to
> try--increase it *a lot* (32 or more, default is 3) because it can really
> help with this problem. A moderate increase to shared_buffers is in order
> too; since you're on Windows, increasing it to 256MB is a reasonable change.
> The rest of the changes in there aren't likely to help out with this
> specific problem.
>
> http://www.westnet.com/~gsmith/content/postgresql/chkp-bgw-83.htm<http://www.westnet.com/%7Egsmit...;: covers the most likely cause of the issue you're running into.
> Unfortunately, most of the solutions you'll see there are things changed in
> 8.3.
>
> --
> Greg Smith 2ndQuadrant Baltimore, MD
> PostgreSQL Training, Services and Support
> greg@2ndQuadrant.com www.2ndQuadrant.com
>
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 10:58 Craig Ringer <craig@postnewspapers.com.au>
parent: Eduardo Piombino <drakorg@gmail.com>
1 sibling, 0 replies; 25+ messages in thread
From: Craig Ringer @ 2010-01-13 10:58 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: pgsql-performance
On 13/01/2010 3:03 PM, Eduardo Piombino wrote:
> One last question, this IO issue I'm facing, do you think it is just a
> matter of RAID configuration speed, or a matter of queue gluttony (and
> not leaving time for other processes to get into the IO queue in a
> reasonable time)?
Hard to say with the data provided. It's not *just* a matter of a slow
array, but that might contribute.
Specifically, though, by "slow array" in this case I'm looking at
latency rather than throughput, particularly read latency under heavy
write load. Simple write throughput isn't really the issue, though bad
write throughput can make it fall apart under a lighter load than it
would otherwise.
High read latencies may not be caused by deep queuing, though that's one
possible cause. A controller that prioritizes batching sequential writes
efficiently over serving random reads would cause it too - though
reducing its queue depth so it can't see as many writes to batch would help.
Let me stress, again, that if you have a decent RAID controller with a
battery backed cache unit you can enable write caching and most of these
issues just go away. Using an array format with better read/write
concurrency, like RAID 10, may help as well.
Honestly, though, at this point you need to collect data on what the
system is actually doing, what's slowing it down and where. *then* look
into how to address it. I can't advise you much on that as you're using
Windows, but there must be lots of info on optimising windows I/O
latencies and throughput on the 'net...
> Because if it was just a matter of speed, ok, with my actual RAID
> configuration lets say it takes 10 minutes to process the ALTER TABLE
> (leaving no space to other IOs until the ALTER TABLE is done), lets say
> then i put the fastest possible RAID setup, or even remove RAID for the
> sake of speed, and it completes in lets say again, 10 seconds (an unreal
> assumption). But if my table now grows 60 times, I would be facing the
> very same problem again, even with the best RAID configuration.
Only if the issue is one of pure write throughput. I don't think it is.
You don't care how long the ALTER takes, only how much it impacts other
users. Reducing the impact on other users so your ALTER can complete in
its own time without stamping all over other work is the idea.
> The problem would seem to be in the way the OS (or hardware, or someone
> else, or all of them) is/are inserting the IO requests into the queue.
It *might* be. There's just not enough information to tell that yet.
You'll need to do quite a bit more monitoring. I don't have the
expertise to advise you on what to do and how to do it under Windows.
> What can I do to control the order in which these IO requests are
> finally entered into the queue?
No idea. You probably need to look into I/O priorities on Windows.
Ideally you shouldn't have to, though. If you can keep read latencies at
sane levels under high write load on your array, you don't *need* to
mess with this.
Note that I'm still guessing about the issue being high read latencies
under write load. It fits what you describe, but there isn't enough data
to be sure, and I don't know how to collect it on Windows.
> What cards do I have to manipulate the order the IO requests are entered
> into the "queue"?
> Can I disable this queue?
> Should I turn disk's IO operation caches off?
> Not use some specific disk/RAID vendor, for instance?
Don't know. Contact your RAID card tech support, Google, search MSDN, etc.
--
Craig Ringer
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 14:06 Euler Taveira de Oliveira <euler@timbira.com>
parent: Eduardo Piombino <drakorg@gmail.com>
0 siblings, 1 reply; 25+ messages in thread
From: Euler Taveira de Oliveira @ 2010-01-13 14:06 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: Greg Smith <greg@2ndquadrant.com>; Craig James <craig_james@emolecules.com>; pgsql-performance
Eduardo Piombino escreveu:
> Maybe it does not get logged at all until the ALTER is completed?
>
This feature [1] was implemented a few months ago and it will be available
only in the next PostgreSQL version (8.5).
[1] http://archives.postgresql.org/pgsql-committers/2009-11/msg00018.php
--
Euler Taveira de Oliveira
http://www.timbira.com/
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 15:53 Eduardo Piombino <drakorg@gmail.com>
parent: Euler Taveira de Oliveira <euler@timbira.com>
0 siblings, 0 replies; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-13 15:53 UTC (permalink / raw)
To: Euler Taveira de Oliveira <euler@timbira.com>; +Cc: Greg Smith <greg@2ndquadrant.com>; Craig James <craig_james@emolecules.com>; pgsql-performance
With that said, I assume my current version of pgsql DOES make all this
heavy work go through WAL logging.
Curious thing is that I remember (of course) reviewing logs of the crash
times, and I didn't see anything strange, not even the famous warning "you
are making checkpoints too often. maybe you should consider using extending
the checkpoint_segments parameter".
I will check it again.
Besides, I will try to gather as much information on the system itself (RAID
controllers, disk vendors, etc).
Thank you, will keep you posted.
On Wed, Jan 13, 2010 at 11:06 AM, Euler Taveira de Oliveira <
euler@timbira.com> wrote:
> Eduardo Piombino escreveu:
> > Maybe it does not get logged at all until the ALTER is completed?
> >
> This feature [1] was implemented a few months ago and it will be available
> only in the next PostgreSQL version (8.5).
>
> [1] http://archives.postgresql.org/pgsql-committers/2009-11/msg00018.php
>
>
> --
> Euler Taveira de Oliveira
> http://www.timbira.com/
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 16:23 Robert Haas <robertmhaas@gmail.com>
parent: Eduardo Piombino <drakorg@gmail.com>
1 sibling, 2 replies; 25+ messages in thread
From: Robert Haas @ 2010-01-13 16:23 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: pgsql-performance
On Wed, Jan 13, 2010 at 2:03 AM, Eduardo Piombino <drakorg@gmail.com> wrote:
> Excellent, lots of useful information in your message.
> I will follow your advices, and keep you posted on any progress. I have yet
> to confirm you with some technical details of my setup, but I'm pretty sure
> you hit the nail in any case.
>
> One last question, this IO issue I'm facing, do you think it is just a
> matter of RAID configuration speed, or a matter of queue gluttony (and not
> leaving time for other processes to get into the IO queue in a reasonable
> time)?
>
> Because if it was just a matter of speed, ok, with my actual RAID
> configuration lets say it takes 10 minutes to process the ALTER TABLE
> (leaving no space to other IOs until the ALTER TABLE is done), lets say then
> i put the fastest possible RAID setup, or even remove RAID for the sake of
> speed, and it completes in lets say again, 10 seconds (an unreal
> assumption). But if my table now grows 60 times, I would be facing the very
> same problem again, even with the best RAID configuration.
>
> The problem would seem to be in the way the OS (or hardware, or someone
> else, or all of them) is/are inserting the IO requests into the queue.
> What can I do to control the order in which these IO requests are finally
> entered into the queue?
> I mean .. what i would like to obtain is:
>
> Considering the ALTER TABLE as a sequence of 100.000 READ/WRITE OPERATIONS
> Considering the SELECT * FROM xxx as a sequence of 100 READ OPERATIONS
> (totally unrelated in disk)
>
> First i run the ALTER TABLE on a thread...
> Lets say by the time it generates 1.000 READ/WRITE OPERATIONS, the other
> thread starts with the SELECT * FROM xxx ...
> I would expect the IO system to give chance to the those 100 READ OPERATIONS
> to execute immediately (with no need to wait for the remaining 990.000
> READ/WRITE OPERATIONS finish), that is, to enter the queue at *almost* the
> very same moment the IO request were issued.
>
> If I can not guarantee that, I'm kinda doomed, because the largest the
> amount of IO operations requested by a "heavy duty operation", the longest
> it will take any other thread to start doing anything.
One thing you can do - although it's a darn lot of work compared to
just running a DDL command - is create a new empty table with the
schema you want and then write a script that copies, say, 1000 records
from the old table to the new table. If your table has a primary key
with a natural sort ordering, it's not too hard to keep track of where
you left off the last time and continue on from there. Then you can
incrementally get all of your data over without swamping the system.
I realize that's a pain in the neck, of course.
I'm kind of surprised that there are disk I/O subsystems that are so
bad that a single thread doing non-stop I/O can take down the whole
server. Is that normal? Does it happen on non-Windows operating
systems? What kind of hardware should I not buy to make sure this
doesn't happen to me?
...Robert
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 16:52 Scott Marlowe <scott.marlowe@gmail.com>
parent: Eduardo Piombino <drakorg@gmail.com>
2 siblings, 1 reply; 25+ messages in thread
From: Scott Marlowe @ 2010-01-13 16:52 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: pgsql-performance
On Tue, Jan 12, 2010 at 9:59 PM, Eduardo Piombino <drakorg@gmail.com> wrote:
...
> Now, with this experience, I tried a simple workaround.
> Created an empty version of "a" named "a_empty", identical in every sense.
> renamed "a" to "a_full", and "a_empty" to "a". This procedure costed me like
> 0 seconds of downtime, and everything kept working smoothly. Maybe a cpl of
> operations could have failed if they tried to write in the very second that
> there was actually no table named "a", but since the operation was
> transactional, the worst scenario was that if the operation should have
> failed, the client application would just inform of the error and ask the
> user for a retry. No big deal.
>
> Now, this table, that is totally unattached to the system in every way (no
> one references this table, its like a dumpster for old records), is not
> begin accessed by no other thread in the system, so an ALTER table on it, to
> turn a char(255) to char(250), should have no effect on the system.
>
> So, with this in mind, I tried the ALTER TABLE this time on the "a_full"
> (totally unrelated) table.
> The system went non-responsive again, and this time it had nothing to do
> with threads waiting for the alter table to complete. The pgAdmin GUI went
> non-responsive, as well as the application's server GUI, whose threads kept
> working on the background, but starting to take more and more time for every
> clients request (up to 25 seconds, which are just ridiculous and completely
> unacceptable in normal conditions).
OK, I'm not entirely sure this table is not still locking something
else. If you make a copy by doing something like:
select * into test_table from a;
and then alter test_table do you still get the same problems? If so,
then it is an IO issue, most likely. If not, then there is some
client connection still referencing this table or something and that
could cause this type of behaviour as well.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 17:54 Eduardo Piombino <drakorg@gmail.com>
parent: Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-13 17:54 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-performance
> OK, I'm not entirely sure this table is not still locking something
> else. If you make a copy by doing something like:
>
> select * into test_table from a;
>
> and then alter test_table do you still get the same problems? If so,
> then it is an IO issue, most likely. If not, then there is some
> client connection still referencing this table or something and that
> could cause this type of behaviour as well.
>
I can guarantee you that the table is not being referenced by any other
thread, table or process, and that it is totally unrelated to everything
else in the system.
Its just a plain table, with 1.8 million records, that no thread knows it
exists. It has no foreign keys that would allow thinking of a possible
"lock" on the parent table, nor it is being referenced by any other table in
the model. It has no triggers associated, and no indexes. It could very well
even be on another database on the same physical server, and still do the
same damage. I did not try this, but I'm pretty sure of the outcome. I
would'nt like to bring the server down just to prove this, but I will do it
if I find it necessary.
The only things that are common to this table and other tables in the
system, as I see are:
RAM, IO, and CPU, at a very low level. One of these is being stressed out by
the thread executing the ALTER, and the other threads (not just pgsql
application threads, but system processes in general) suffer from the lack
of this resource. All the previous discussions tend to induce that the
resource we are talking about is IO.
The fact that the Task Manager does not come up, would also not be explained
by a lock in a client thread.
Besides all that, all the client queries are NO WAIT, thus any lock would
just return immediately, and no retry would be done until the response gets
back to the user and the user confirms it. In that case, all the errors
presented to the final users would be "The element is being processed some
other place", as my default handler to pgsql error code "55P03", instead of
the horrible "Operation timed out", that is what final users got during the
huge slowdown/downtime.
--
> Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-performance
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 18:11 Scott Marlowe <scott.marlowe@gmail.com>
parent: Eduardo Piombino <drakorg@gmail.com>
0 siblings, 0 replies; 25+ messages in thread
From: Scott Marlowe @ 2010-01-13 18:11 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; +Cc: pgsql-performance
On Wed, Jan 13, 2010 at 10:54 AM, Eduardo Piombino <drakorg@gmail.com> wrote:
>
>> OK, I'm not entirely sure this table is not still locking something
>> else. If you make a copy by doing something like:
>>
>> select * into test_table from a;
>>
>> and then alter test_table do you still get the same problems? If so,
>> then it is an IO issue, most likely. If not, then there is some
>> client connection still referencing this table or something and that
>> could cause this type of behaviour as well.
>
> I can guarantee you that the table is not being referenced by any other
> thread, table or process, and that it is totally unrelated to everything
> else in the system.
If you rename a table that WAS being referenced by other threads, then
it might still be being accessed or waited on etc by those threads, as
their transaction would have started earlier.
The only way you can guarantee it's not being reference in some way is
to create it fresh and new as I suggested and test on that. Until
then, your guarantee is based on a belief, not verifiable fact. I too
tend to believe this is an IO problem btw, but claiming that it can't
be a problem with some locks without looking at pg_locks at least, is
a bit premature.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 18:46 Greg Smith <greg@2ndquadrant.com>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 25+ messages in thread
From: Greg Smith @ 2010-01-13 18:46 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Eduardo Piombino <drakorg@gmail.com>; pgsql-performance
Robert Haas wrote:
> I'm kind of surprised that there are disk I/O subsystems that are so
> bad that a single thread doing non-stop I/O can take down the whole
> server. Is that normal? Does it happen on non-Windows operating
> systems? What kind of hardware should I not buy to make sure this
> doesn't happen to me?
>
You can kill any hardware on any OS with the right abusive client.
Create a wide table and insert a few million records into it with
generate_series one day and watch what it does to queries trying to run
in parallel with that.
I think the missing step here to nail down exactly what's happening on
Eduardo's system is that he should open up some of the Windows system
monitoring tools, look at both disk I/O and CPU usage, and then watch
what changes when the troublesome ALTER TABLE shows up.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-13 19:13 Eduardo Piombino <drakorg@gmail.com>
parent: Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-13 19:13 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; pgsql-performance
Greg, I will post more detailed data as soon as I'm able to gather it.
I was trying out if the cancellation of the ALTER cmd worked ok, I might
give the ALTER another try, and see how much CPU, RAM and IO usage gets
involved. I will be doing this monitoring with the process explorer from
sysinternals, but I don't know how I can make it to log the results. Do you
know any tool that you have used that can help me generate this evidence? I
will google a little as soon as possible.
On Wed, Jan 13, 2010 at 3:46 PM, Greg Smith <greg@2ndquadrant.com> wrote:
> Robert Haas wrote:
>
>> I'm kind of surprised that there are disk I/O subsystems that are so
>> bad that a single thread doing non-stop I/O can take down the whole
>> server. Is that normal? Does it happen on non-Windows operating
>> systems? What kind of hardware should I not buy to make sure this
>> doesn't happen to me?
>>
>>
> You can kill any hardware on any OS with the right abusive client. Create
> a wide table and insert a few million records into it with generate_series
> one day and watch what it does to queries trying to run in parallel with
> that.
>
> I think the missing step here to nail down exactly what's happening on
> Eduardo's system is that he should open up some of the Windows system
> monitoring tools, look at both disk I/O and CPU usage, and then watch what
> changes when the troublesome ALTER TABLE shows up.
>
>
> --
> Greg Smith 2ndQuadrant Baltimore, MD
> PostgreSQL Training, Services and Support
> greg@2ndQuadrant.com www.2ndQuadrant.com
>
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 05:36 Craig Ringer <craig@postnewspapers.com.au>
parent: Robert Haas <robertmhaas@gmail.com>
1 sibling, 1 reply; 25+ messages in thread
From: Craig Ringer @ 2010-01-14 05:36 UTC (permalink / raw)
To: Robert Haas <robertmhaas@gmail.com>; +Cc: Eduardo Piombino <drakorg@gmail.com>; pgsql-performance
Robert Haas wrote:
> I'm kind of surprised that there are disk I/O subsystems that are so
> bad that a single thread doing non-stop I/O can take down the whole
> server. Is that normal?
No.
> Does it happen on non-Windows operating
> systems?
Yes. My 3ware 8500-8 on a Debian Sarge box was so awful that launching a
terminal would go from a 1/4 second operation to a 5 minute operation
under heavy write load by one writer. I landed up having to modify the
driver to partially mitigate the issue, but a single user on the
terminal server performing any sort of heavy writing would still
absolutely nuke performance.
I landed up having dramatically better results by disabling the
controller's RAID features, instead exposing each disk to the OS
separately and using Linux's software RAID.
> What kind of hardware should I not buy to make sure this
> doesn't happen to me?
3ware's older cards. Apparently their new ones are a lot better, but I
haven't verified this personally.
Anything in RAID-5 without a BBU.
Anything at all without a BBU, preferably.
--
Craig Ringer
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 14:49 Andy Colson <andy@squeakycode.net>
parent: Craig Ringer <craig@postnewspapers.com.au>
0 siblings, 1 reply; 25+ messages in thread
From: Andy Colson @ 2010-01-14 14:49 UTC (permalink / raw)
To: Craig Ringer <craig@postnewspapers.com.au>; pgsql-performance
On 1/13/2010 11:36 PM, Craig Ringer wrote:
> Robert Haas wrote:
>
>> I'm kind of surprised that there are disk I/O subsystems that are so
>> bad that a single thread doing non-stop I/O can take down the whole
>> server. Is that normal?
>
> No.
>
>> Does it happen on non-Windows operating
>> systems?
>
> Yes. My 3ware 8500-8 on a Debian Sarge box was so awful that launching a
> terminal would go from a 1/4 second operation to a 5 minute operation
> under heavy write load by one writer. I landed up having to modify the
> driver to partially mitigate the issue, but a single user on the
> terminal server performing any sort of heavy writing would still
> absolutely nuke performance.
On a side note, on linux, would using the deadline scheduler resolve that?
-Andy
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 16:16 Pierre Frédéric Caillaud <lists@peufeu.com>
parent: Eduardo Piombino <drakorg@gmail.com>
2 siblings, 0 replies; 25+ messages in thread
From: Pierre Frédéric Caillaud @ 2010-01-14 16:16 UTC (permalink / raw)
To: Eduardo Piombino <drakorg@gmail.com>; Craig James <craig_james@emolecules.com>; +Cc: pgsql-performance
> "high CPU usage" It might very well be "high IO usage".
Try this :
Copy (using explorer, the shell, whatever) a huge file.
This will create load similar to ALTER TABLE.
Measure throughput, how much is it ?
If your server blows up just like it did on ALTER TABLE, you got a IO
system problem.
If everything is smooth, you can look into other things.
How's your fragmentation ? Did the disk ever get full ? What does the
task manager say (swap in/out, disk queue lengthn etc)
PS : try a separate tablespace on another disk.
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 18:07 Greg Smith <greg@2ndquadrant.com>
parent: Andy Colson <andy@squeakycode.net>
0 siblings, 1 reply; 25+ messages in thread
From: Greg Smith @ 2010-01-14 18:07 UTC (permalink / raw)
To: Andy Colson <andy@squeakycode.net>; +Cc: Craig Ringer <craig@postnewspapers.com.au>; pgsql-performance
Andy Colson wrote:
> On 1/13/2010 11:36 PM, Craig Ringer wrote:
>> Yes. My 3ware 8500-8 on a Debian Sarge box was so awful that launching a
>> terminal would go from a 1/4 second operation to a 5 minute operation
>> under heavy write load by one writer. I landed up having to modify the
>> driver to partially mitigate the issue, but a single user on the
>> terminal server performing any sort of heavy writing would still
>> absolutely nuke performance.
>
> On a side note, on linux, would using the deadline scheduler resolve
> that?
I've never seen the deadline scheduler resolve anything. If you're out
of I/O capacity and that's blocking other work, performance is dominated
by the policies of the underlying controller/device caches. Think about
it a minute: disks nowadays can easily have 32MB of buffer in them,
right? And random read/write operations are lucky to clear 2MB/s on
cheap drivers. So once the drive is filled with requests, you can
easily sit there for ten seconds before the scheduler even has any input
on resolving the situation. That's even more true if you've got a
larger controller cache in the mix.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 18:15 Andy Colson <andy@squeakycode.net>
parent: Greg Smith <greg@2ndquadrant.com>
0 siblings, 1 reply; 25+ messages in thread
From: Andy Colson @ 2010-01-14 18:15 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Craig Ringer <craig@postnewspapers.com.au>; pgsql-performance
On 1/14/2010 12:07 PM, Greg Smith wrote:
> Andy Colson wrote:
>> On 1/13/2010 11:36 PM, Craig Ringer wrote:
>>> Yes. My 3ware 8500-8 on a Debian Sarge box was so awful that launching a
>>> terminal would go from a 1/4 second operation to a 5 minute operation
>>> under heavy write load by one writer. I landed up having to modify the
>>> driver to partially mitigate the issue, but a single user on the
>>> terminal server performing any sort of heavy writing would still
>>> absolutely nuke performance.
>>
>> On a side note, on linux, would using the deadline scheduler resolve
>> that?
>
> I've never seen the deadline scheduler resolve anything. If you're out
> of I/O capacity and that's blocking other work, performance is dominated
> by the policies of the underlying controller/device caches. Think about
> it a minute: disks nowadays can easily have 32MB of buffer in them,
> right? And random read/write operations are lucky to clear 2MB/s on
> cheap drivers. So once the drive is filled with requests, you can easily
> sit there for ten seconds before the scheduler even has any input on
> resolving the situation. That's even more true if you've got a larger
> controller cache in the mix.
>
That makes sense. So if there is very little io, or if there is way way
too much, then the scheduler really doesn't matter. So there is a slim
middle ground where the io is within a small percent of the HD capacity
where the scheduler might make a difference?
-Andy
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 18:30 Greg Smith <greg@2ndquadrant.com>
parent: Andy Colson <andy@squeakycode.net>
0 siblings, 0 replies; 25+ messages in thread
From: Greg Smith @ 2010-01-14 18:30 UTC (permalink / raw)
To: Andy Colson <andy@squeakycode.net>; +Cc: Craig Ringer <craig@postnewspapers.com.au>; pgsql-performance
Andy Colson wrote:
> So if there is very little io, or if there is way way too much, then
> the scheduler really doesn't matter. So there is a slim middle ground
> where the io is within a small percent of the HD capacity where the
> scheduler might make a difference?
That's basically how I see it. There seem to be people who run into
workloads in the middle ground where the scheduler makes a world of
difference. I've never seen one myself, and suspect that some of the
reports of deadline being a big improvement just relate to some buginess
in the default CFQ implementation that I just haven't encountered.
--
Greg Smith 2ndQuadrant Baltimore, MD
PostgreSQL Training, Services and Support
greg@2ndQuadrant.com www.2ndQuadrant.com
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 20:49 Eduardo Piombino <drakorg@gmail.com>
parent: Eduardo Piombino <drakorg@gmail.com>
0 siblings, 1 reply; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-14 20:49 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; pgsql-performance
Regarding the hardware the system is running on:
It's an HP Proliant DL-180 G5 server.
Here are the specs... our actual configuration only has one CPU, and 16G of
RAM.
The model of the 2 disks I will post later today, when I get to the server.
I was with many things, sorry.
http://h18000.www1.hp.com/products/quickspecs/12903_na/12903_na.HTML
http://h18004.www1.hp.com/products/quickspecs/DS_00126/DS_00126.pdf
*At A Glance
*The HP ProLiant DL180 G5 is a low cost high capacity storage optimized
2-way server that delivers on a history of design excellence and 2U density
for a variety of rack deployments and applications.
- Processors:
- Supports up to two Quad-Core Intel® Xeon® processors: 5400 sequence
with 12MB Level 2 cache
- Intel® 5100 Chipset
- Memory:
- Up to 32 GB of memory supported by six (6) PC2-5300 (667 MHz) DDR2
memory slots
- Internal Drive Support:
- Supports up to twelve via CTO with controller or up to eight via BTO
with the addition of a controller:
- Hot Plug Serial ATA (SATA) 3.5"hard drives; or
- Hot Plug Serial Attached SCSI (SAS) 3.5"hard drives
*NOTE:* 4 hard drives are supported standard via BTO. 8 hard drive
support requires the addition of a Smart Array or HBA
controller. Hot Plug
and SAS functionality require the addition of a Smart Array or HBA
controller. 12 hard drive support available via CTO only and
requires a SAS
controller that supports expanders.
- Internal storage capacity:
- SATA Models: Up to 12.0TB (12 x 1TB Hot Plug 3.5" hard drives)
- SAS Model: Up to 12.0TB (12 x 1TB Hot Plug 3.5" hard drives)
- Network Controller:
- One integrated NC105i PCI-e Gigabit NIC (embedded) (Wake on LAN and
PXE capable)
- Storage Controllers:
- HP Embedded SATA RAID Controller (up to 4 hard drive support on
standard BTO models)
*NOTE:* Transfer rate 1.5 Gb/s SATA
- Expansion Slots:
- One available Low Profile x8 PCI-Express slot using a Low profile
Riser.
- Two Full Height/ Full Length Riser options
- Option1: 2 full-length/full-height PCI-Express x8 connector slots
(x4 electrical - Standard)
- Option2: full-length/full-height riser with 2 PCI-X
Slots(Optional)
- Infrastructure Management:
- Optional HP Lights Out 100c Remote Management card with Virtual KVM
and Virtual Media support (includes IPMI2.0 and SMASH support)
- USB Ports:
- Seven USB ports (2) front, (4) rear, (1) internal
- Optical Drive:
- Support for one:
- Optional Multi-bay DVD
- Optional Floppy (USB only, USB key)
- Power Supply:
- 750W Power Supply (Optional Redundancy Hot Plug, Autoswitching) CSCI
2007/8
- 1200W High Efficiency Power Supply (Optional Redundancy Hot Plug,
Autoswitching) (Optional) CSCI 2007/8
- *NOTE:* Climate Savers Computing Initiative, 2007-2008 Compliant
- Form Factor:
- 2U rack models
Regarding the SATA RAID controller, on the other spec pages it says that for
the 8 disks model (ours), it comes with a Smart Array E200. I will try to
check out if we are using the original, since I recall hearing something
about that our disks were SAS (Serial Attached SCSI), and I don't know if it
is possible to connect those disks to embedded Smart Array E200 controller.
Would it be possible?
On Wed, Jan 13, 2010 at 4:13 PM, Eduardo Piombino <drakorg@gmail.com> wrote:
> Greg, I will post more detailed data as soon as I'm able to gather it.
>
> I was trying out if the cancellation of the ALTER cmd worked ok, I might
> give the ALTER another try, and see how much CPU, RAM and IO usage gets
> involved. I will be doing this monitoring with the process explorer from
> sysinternals, but I don't know how I can make it to log the results. Do you
> know any tool that you have used that can help me generate this evidence? I
> will google a little as soon as possible.
>
>
>
> On Wed, Jan 13, 2010 at 3:46 PM, Greg Smith <greg@2ndquadrant.com> wrote:
>
>> Robert Haas wrote:
>>
>>> I'm kind of surprised that there are disk I/O subsystems that are so
>>> bad that a single thread doing non-stop I/O can take down the whole
>>> server. Is that normal? Does it happen on non-Windows operating
>>> systems? What kind of hardware should I not buy to make sure this
>>> doesn't happen to me?
>>>
>>>
>> You can kill any hardware on any OS with the right abusive client. Create
>> a wide table and insert a few million records into it with generate_series
>> one day and watch what it does to queries trying to run in parallel with
>> that.
>>
>> I think the missing step here to nail down exactly what's happening on
>> Eduardo's system is that he should open up some of the Windows system
>> monitoring tools, look at both disk I/O and CPU usage, and then watch what
>> changes when the troublesome ALTER TABLE shows up.
>>
>>
>> --
>> Greg Smith 2ndQuadrant Baltimore, MD
>> PostgreSQL Training, Services and Support
>> greg@2ndQuadrant.com www.2ndQuadrant.com
>>
>>
>
^ permalink raw reply [nested|flat] 25+ messages in thread
* Re: a heavy duty operation on an "unused" table kills my server
@ 2010-01-14 21:01 Eduardo Piombino <drakorg@gmail.com>
parent: Eduardo Piombino <drakorg@gmail.com>
0 siblings, 0 replies; 25+ messages in thread
From: Eduardo Piombino @ 2010-01-14 21:01 UTC (permalink / raw)
To: Greg Smith <greg@2ndquadrant.com>; +Cc: Robert Haas <robertmhaas@gmail.com>; pgsql-performance
Regarding the EA-200 card, here are the specs.
It seems it has support for SAS disks, so it is most probably that we are
using the embedded/default controller.
http://h18000.www1.hp.com/products/quickspecs/12460_div/12460_div.html
http://h18000.www1.hp.com/products/quickspecs/12460_div/12460_div.pdf
*Key Features *
- Seamless upgrades from past generations and upgrades to next generation
HP high performance and high capacity Serial Attached SCSI Smart Array
controllers.
- 3G SAS technology delivers high performance and data bandwidth up to
300 MB\s per physical link and contains full compatibility with 1.5G SATA
technology.
- x4 2.5G PCI Express host interface technology delivers high performance
and data bandwidth up to 2 GB/s maximum bandwidth.
- Addition of the battery backed cache upgrade enables BBWC, RAID 5,
Capacity Expansion, RAID migration, and Stripe Size Migration.
- Mix-and-match SAS and SATA hard drives, lets you deploy drive
technology as needed to fit your computing environment.
- Support for up to 2 TB in a single logical drive.
- Software consistency among all Smart Array family products: Array
Configuration Utility (ACU), Option ROM Configuration for Arrays (ORCA),
Systems Insight Manager, Array Diagnostic Utility (ADU) and SmartStart. Some
of these features are not available with ProLiant 100 series platforms.
- The SA-E200 controller supports up to 8 drives. The SA-E200i supports
2-8 drives depending on the server implementation.
*Performance*
HP's High Performance Architecture sets new boundaries of industry
performance expectations!
- 3Gb/s SAS (300MB/s bandwidth per physical link)
- x8 3Gb/s SAS physical links (compatible with 1.5G SATA)
- 64 MB or 128 MB DDR1-266 battery-backed cache provides up to 4.2 GB/s
maximum bandwidth.
- x4 2.5G PCI Express host interface provides 2 GB/s maximum bandwidth.
- MIPS 32-bit Processor
- Read ahead caching
- Write-back caching (with battery-backed write cache upgrade)
*Capacity *
Given the increasing need for high performance and rapid capacity expansion,
the SA-E200 offers:
- Up to 6TB of total storage with 6 x 1TB SATA MDL hard drives (3.5")
*NOTE:* Support for greater than 2TB in a single logical drive.
- Up to 2.4TB of total storage with 8 x 300GB SFF SAS hard drives
On Thu, Jan 14, 2010 at 5:49 PM, Eduardo Piombino <drakorg@gmail.com> wrote:
> Regarding the hardware the system is running on:
>
> It's an HP Proliant DL-180 G5 server.
>
> Here are the specs... our actual configuration only has one CPU, and 16G of
> RAM.
> The model of the 2 disks I will post later today, when I get to the server.
> I was with many things, sorry.
>
> http://h18000.www1.hp.com/products/quickspecs/12903_na/12903_na.HTML
> http://h18004.www1.hp.com/products/quickspecs/DS_00126/DS_00126.pdf
>
> *At A Glance
> *The HP ProLiant DL180 G5 is a low cost high capacity storage optimized
> 2-way server that delivers on a history of design excellence and 2U density
> for a variety of rack deployments and applications.
>
> - Processors:
> - Supports up to two Quad-Core Intel® Xeon® processors: 5400
> sequence with 12MB Level 2 cache
> - Intel® 5100 Chipset
> - Memory:
> - Up to 32 GB of memory supported by six (6) PC2-5300 (667 MHz) DDR2
> memory slots
> - Internal Drive Support:
> - Supports up to twelve via CTO with controller or up to eight via
> BTO with the addition of a controller:
> - Hot Plug Serial ATA (SATA) 3.5"hard drives; or
> - Hot Plug Serial Attached SCSI (SAS) 3.5"hard drives
> *NOTE:* 4 hard drives are supported standard via BTO. 8 hard
> drive support requires the addition of a Smart Array or HBA controller. Hot
> Plug and SAS functionality require the addition of a Smart Array or HBA
> controller. 12 hard drive support available via CTO only and requires a SAS
> controller that supports expanders.
> - Internal storage capacity:
> - SATA Models: Up to 12.0TB (12 x 1TB Hot Plug 3.5" hard drives)
> - SAS Model: Up to 12.0TB (12 x 1TB Hot Plug 3.5" hard drives)
> - Network Controller:
> - One integrated NC105i PCI-e Gigabit NIC (embedded) (Wake on LAN
> and PXE capable)
> - Storage Controllers:
> - HP Embedded SATA RAID Controller (up to 4 hard drive support on
> standard BTO models)
> *NOTE:* Transfer rate 1.5 Gb/s SATA
> - Expansion Slots:
> - One available Low Profile x8 PCI-Express slot using a Low profile
> Riser.
> - Two Full Height/ Full Length Riser options
> - Option1: 2 full-length/full-height PCI-Express x8 connector
> slots (x4 electrical - Standard)
> - Option2: full-length/full-height riser with 2 PCI-X
> Slots(Optional)
> - Infrastructure Management:
> - Optional HP Lights Out 100c Remote Management card with Virtual
> KVM and Virtual Media support (includes IPMI2.0 and SMASH support)
> - USB Ports:
> - Seven USB ports (2) front, (4) rear, (1) internal
> - Optical Drive:
> - Support for one:
> - Optional Multi-bay DVD
> - Optional Floppy (USB only, USB key)
> - Power Supply:
> - 750W Power Supply (Optional Redundancy Hot Plug, Autoswitching)
> CSCI 2007/8
> - 1200W High Efficiency Power Supply (Optional Redundancy Hot Plug,
> Autoswitching) (Optional) CSCI 2007/8
> - *NOTE:* Climate Savers Computing Initiative, 2007-2008
> Compliant
> - Form Factor:
> - 2U rack models
>
>
> Regarding the SATA RAID controller, on the other spec pages it says that
> for the 8 disks model (ours), it comes with a Smart Array E200. I will try
> to check out if we are using the original, since I recall hearing something
> about that our disks were SAS (Serial Attached SCSI), and I don't know if it
> is possible to connect those disks to embedded Smart Array E200 controller.
> Would it be possible?
>
>
> On Wed, Jan 13, 2010 at 4:13 PM, Eduardo Piombino <drakorg@gmail.com>wrote:
>
>> Greg, I will post more detailed data as soon as I'm able to gather it.
>>
>> I was trying out if the cancellation of the ALTER cmd worked ok, I might
>> give the ALTER another try, and see how much CPU, RAM and IO usage gets
>> involved. I will be doing this monitoring with the process explorer from
>> sysinternals, but I don't know how I can make it to log the results. Do you
>> know any tool that you have used that can help me generate this evidence? I
>> will google a little as soon as possible.
>>
>>
>>
>> On Wed, Jan 13, 2010 at 3:46 PM, Greg Smith <greg@2ndquadrant.com> wrote:
>>
>>> Robert Haas wrote:
>>>
>>>> I'm kind of surprised that there are disk I/O subsystems that are so
>>>> bad that a single thread doing non-stop I/O can take down the whole
>>>> server. Is that normal? Does it happen on non-Windows operating
>>>> systems? What kind of hardware should I not buy to make sure this
>>>> doesn't happen to me?
>>>>
>>>>
>>> You can kill any hardware on any OS with the right abusive client.
>>> Create a wide table and insert a few million records into it with
>>> generate_series one day and watch what it does to queries trying to run in
>>> parallel with that.
>>>
>>> I think the missing step here to nail down exactly what's happening on
>>> Eduardo's system is that he should open up some of the Windows system
>>> monitoring tools, look at both disk I/O and CPU usage, and then watch what
>>> changes when the troublesome ALTER TABLE shows up.
>>>
>>>
>>> --
>>> Greg Smith 2ndQuadrant Baltimore, MD
>>> PostgreSQL Training, Services and Support
>>> greg@2ndQuadrant.com www.2ndQuadrant.com
>>>
>>>
>>
>
^ permalink raw reply [nested|flat] 25+ messages in thread
end of thread, other threads:[~2010-01-14 21:01 UTC | newest]
Thread overview: 25+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-01-13 04:59 a heavy duty operation on an "unused" table kills my server Eduardo Piombino <drakorg@gmail.com>
2010-01-13 05:14 ` Craig James <craig_james@emolecules.com>
2010-01-13 05:47 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-13 06:02 ` Craig Ringer <craig@postnewspapers.com.au>
2010-01-13 07:03 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-13 10:58 ` Craig Ringer <craig@postnewspapers.com.au>
2010-01-13 16:23 ` Robert Haas <robertmhaas@gmail.com>
2010-01-13 18:46 ` Greg Smith <greg@2ndquadrant.com>
2010-01-13 19:13 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-14 20:49 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-14 21:01 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-14 05:36 ` Craig Ringer <craig@postnewspapers.com.au>
2010-01-14 14:49 ` Andy Colson <andy@squeakycode.net>
2010-01-14 18:07 ` Greg Smith <greg@2ndquadrant.com>
2010-01-14 18:15 ` Andy Colson <andy@squeakycode.net>
2010-01-14 18:30 ` Greg Smith <greg@2ndquadrant.com>
2010-01-13 07:39 ` Greg Smith <greg@2ndquadrant.com>
2010-01-13 08:27 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-13 14:06 ` Euler Taveira de Oliveira <euler@timbira.com>
2010-01-13 15:53 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-14 16:16 ` Pierre Frédéric Caillaud <lists@peufeu.com>
2010-01-13 05:41 ` Craig Ringer <craig@postnewspapers.com.au>
2010-01-13 16:52 ` Scott Marlowe <scott.marlowe@gmail.com>
2010-01-13 17:54 ` Eduardo Piombino <drakorg@gmail.com>
2010-01-13 18:11 ` Scott Marlowe <scott.marlowe@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox