pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feeddatabase slowdown while a lot of inserts occur
12+ messages / 9 participants
[nested] [flat]
* database slowdown while a lot of inserts occur
@ 2012-03-29 17:59 Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Re: database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:17 ` Re: database slowdown while a lot of inserts occur Deron <fecastle@gmail.com>
2012-03-31 01:21 ` Re: database slowdown while a lot of inserts occur Tomas Vondra <tv@fuzzy.cz>
2012-04-02 07:14 ` Re: database slowdown while a lot of inserts occur Віталій Тимчишин <tivv00@gmail.com>
0 siblings, 4 replies; 12+ messages in thread
From: Campbell, Lance @ 2012-03-29 17:59 UTC (permalink / raw)
To: pgsql-performance
PostgreSQL 9.0.x
We have around ten different applications that use the same database. When one particular application is active it does an enormous number of inserts. Each insert is very small. During this time the database seems to slow down in general. The application in question is inserting into a particular table that is not used by the other applications.
1) What should I do to confirm that the database is the issue and not the applications?
2) How can I identify where the bottle neck is occurring if the issue happens to be with the database?
I have been using PostgreSQL for eight years. It is an amazing database.
Thanks,
Lance Campbell
Software Architect
Web Services at Public Affairs
217-333-0382
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
@ 2012-03-29 18:02 ` Campbell, Lance <lance@illinois.edu>
2012-03-29 18:14 ` Re: database slowdown while a lot of inserts occur Filippos Kalamidas <filippos.kal@gmail.com>
2012-03-29 19:27 ` Re: database slowdown while a lot of inserts occur Bob Lunney <bob_lunney@yahoo.com>
2012-03-29 21:36 ` Re: database slowdown while a lot of inserts occur Scott Marlowe <scott.marlowe@gmail.com>
3 siblings, 3 replies; 12+ messages in thread
From: Campbell, Lance @ 2012-03-29 18:02 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; pgsql-performance
I forgot to mention that the slowdown in particular for other applications is when they are trying to insert or update tables unrelated to the application mentioned in my prior application that does the massive small inserts.
Thanks,
Lance Campbell
Software Architect
Web Services at Public Affairs
217-333-0382
From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Campbell, Lance
Sent: Thursday, March 29, 2012 12:59 PM
To: pgsql-performance@postgresql.org
Subject: [PERFORM] database slowdown while a lot of inserts occur
PostgreSQL 9.0.x
We have around ten different applications that use the same database. When one particular application is active it does an enormous number of inserts. Each insert is very small. During this time the database seems to slow down in general. The application in question is inserting into a particular table that is not used by the other applications.
1) What should I do to confirm that the database is the issue and not the applications?
2) How can I identify where the bottle neck is occurring if the issue happens to be with the database?
I have been using PostgreSQL for eight years. It is an amazing database.
Thanks,
Lance Campbell
Software Architect
Web Services at Public Affairs
217-333-0382
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Re: database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
@ 2012-03-29 18:14 ` Filippos Kalamidas <filippos.kal@gmail.com>
2 siblings, 0 replies; 12+ messages in thread
From: Filippos Kalamidas @ 2012-03-29 18:14 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; +Cc: pgsql-performance
can you post all the configuration parameters related to the I/O activity?
plus, could you post some stats from 'iostat' when this is happening?
thx
On Thu, Mar 29, 2012 at 9:02 PM, Campbell, Lance <lance@illinois.edu> wrote:
> I forgot to mention that the slowdown in particular for other
> applications is when they are trying to insert or update tables unrelated
> to the application mentioned in my prior application that does the massive
> small inserts.****
>
> ** **
>
> ** **
>
> Thanks,****
>
> ** **
>
> Lance Campbell****
>
> Software Architect****
>
> Web Services at Public Affairs****
>
> 217-333-0382****
>
> ** **
>
> *From:* pgsql-performance-owner@postgresql.org [mailto:
> pgsql-performance-owner@postgresql.org] *On Behalf Of *Campbell, Lance
> *Sent:* Thursday, March 29, 2012 12:59 PM
> *To:* pgsql-performance@postgresql.org
> *Subject:* [PERFORM] database slowdown while a lot of inserts occur****
>
> ** **
>
> PostgreSQL 9.0.x****
>
> We have around ten different applications that use the same database.
> When one particular application is active it does an enormous number of
> inserts. Each insert is very small. During this time the database seems
> to slow down in general. The application in question is inserting into a
> particular table that is not used by the other applications.****
>
> ** **
>
> **1) **What should I do to confirm that the database is the issue
> and not the applications?****
>
> **2) **How can I identify where the bottle neck is occurring if the
> issue happens to be with the database?****
>
> ** **
>
> I have been using PostgreSQL for eight years. It is an amazing database.*
> ***
>
> ** **
>
> Thanks,****
>
> ** **
>
> Lance Campbell****
>
> Software Architect****
>
> Web Services at Public Affairs****
>
> 217-333-0382****
>
> ** **
>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Re: database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
@ 2012-03-29 19:27 ` Bob Lunney <bob_lunney@yahoo.com>
2012-03-29 19:39 ` Re: database slowdown while a lot of inserts occur Andrew Dunstan <andrew@dunslane.net>
2012-03-31 01:11 ` Re: database slowdown while a lot of inserts occur Tomas Vondra <tv@fuzzy.cz>
2 siblings, 2 replies; 12+ messages in thread
From: Bob Lunney @ 2012-03-29 19:27 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; pgsql-performance
Lance,
May small inserts cause frequent fsyncs. Is there any way those small inserts can be batched into some larger sets of inserts that use copy to perform the load?
Bob Lunney
________________________________
From: "Campbell, Lance" <lance@illinois.edu>
To: "Campbell, Lance" <lance@illinois.edu>; "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
Sent: Thursday, March 29, 2012 1:02 PM
Subject: Re: [PERFORM] database slowdown while a lot of inserts occur
I forgot to mention that the slowdown in particular for other applications is when they are trying to insert or update tables unrelated to the application mentioned in my prior application that does the massive small inserts.
Thanks,
Lance Campbell
Software Architect
Web Services at Public Affairs
217-333-0382
From:pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Campbell, Lance
Sent: Thursday, March 29, 2012 12:59 PM
To: pgsql-performance@postgresql.org
Subject: [PERFORM] database slowdown while a lot of inserts occur
PostgreSQL 9.0.x
We have around ten different applications that use the same database. When one particular application is active it does an enormous number of inserts. Each insert is very small. During this time the database seems to slow down in general. The application in question is inserting into a particular table that is not used by the other applications.
1) What should I do to confirm that the database is the issue and not the applications?
2) How can I identify where the bottle neck is occurring if the issue happens to be with the database?
I have been using PostgreSQL for eight years. It is an amazing database.
Thanks,
Lance Campbell
Software Architect
Web Services at Public Affairs
217-333-0382
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Re: database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 19:27 ` Re: database slowdown while a lot of inserts occur Bob Lunney <bob_lunney@yahoo.com>
@ 2012-03-29 19:39 ` Andrew Dunstan <andrew@dunslane.net>
1 sibling, 0 replies; 12+ messages in thread
From: Andrew Dunstan @ 2012-03-29 19:39 UTC (permalink / raw)
To: Bob Lunney <bob_lunney@yahoo.com>; +Cc: Campbell, Lance <lance@illinois.edu>; pgsql-performance
On 03/29/2012 03:27 PM, Bob Lunney wrote:
> Lance,
>
> May small inserts cause frequent fsyncs. Is there any way those small
> inserts can be batched into some larger sets of inserts that use copy
> to perform the load?
Or possibly a prepared statement called many times in a single
transaction, if you're not using that already. It's not as efficient as
COPY, but it's often a much less extensive change to the code.
cheers
andrew
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Re: database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 19:27 ` Re: database slowdown while a lot of inserts occur Bob Lunney <bob_lunney@yahoo.com>
@ 2012-03-31 01:11 ` Tomas Vondra <tv@fuzzy.cz>
2012-04-01 02:20 ` Re: database slowdown while a lot of inserts occur Bob Lunney <bob_lunney@yahoo.com>
1 sibling, 1 reply; 12+ messages in thread
From: Tomas Vondra @ 2012-03-31 01:11 UTC (permalink / raw)
To: pgsql-performance
On 29.3.2012 21:27, Bob Lunney wrote:
> Lance,
>
> May small inserts cause frequent fsyncs. Is there any way those small
> inserts can be batched into some larger sets of inserts that use copy to
> perform the load?
Not necessarily - fsync happens at COMMIT time, not when the INSERT is
performed (unless each INSERT stands on it's own).
Tomas
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Re: database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 19:27 ` Re: database slowdown while a lot of inserts occur Bob Lunney <bob_lunney@yahoo.com>
2012-03-31 01:11 ` Re: database slowdown while a lot of inserts occur Tomas Vondra <tv@fuzzy.cz>
@ 2012-04-01 02:20 ` Bob Lunney <bob_lunney@yahoo.com>
0 siblings, 0 replies; 12+ messages in thread
From: Bob Lunney @ 2012-04-01 02:20 UTC (permalink / raw)
To: Tomas Vondra <tv@fuzzy.cz>; pgsql-performance
Tomas,
You are correct. I was assuming that each insert was issued as an implicit transaction, without the benefit of an explicit BEGIN/COMMIT batching many of them together, as I've seen countless times in tight loops trying to pose as a batch insert.
Bob
________________________________
From: Tomas Vondra <tv@fuzzy.cz>
To: pgsql-performance@postgresql.org
Sent: Friday, March 30, 2012 8:11 PM
Subject: Re: [PERFORM] database slowdown while a lot of inserts occur
On 29.3.2012 21:27, Bob Lunney wrote:
> Lance,
>
> May small inserts cause frequent fsyncs. Is there any way those small
> inserts can be batched into some larger sets of inserts that use copy to
> perform the load?
Not necessarily - fsync happens at COMMIT time, not when the INSERT is
performed (unless each INSERT stands on it's own).
Tomas
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Re: database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
@ 2012-03-29 21:36 ` Scott Marlowe <scott.marlowe@gmail.com>
2 siblings, 0 replies; 12+ messages in thread
From: Scott Marlowe @ 2012-03-29 21:36 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; +Cc: pgsql-performance
On Thu, Mar 29, 2012 at 12:02 PM, Campbell, Lance <lance@illinois.edu> wrote:
> I forgot to mention that the slowdown in particular for other applications
> is when they are trying to insert or update tables unrelated to the
> application mentioned in my prior application that does the massive small
> inserts.
It sounds like you're just flooding your IO channels. Can you
throttle your write rate in your main application that's inserting so
many inserts? Something that sleeps 10ms between every hundred rows
or some such?
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
@ 2012-03-29 18:17 ` Deron <fecastle@gmail.com>
2012-03-29 20:17 ` Re: database slowdown while a lot of inserts occur Shrirang Chitnis <Shrirang.Chitnis@hovservices.com>
3 siblings, 1 reply; 12+ messages in thread
From: Deron @ 2012-03-29 18:17 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; +Cc: pgsql-performance
On a Linux system you can use tools like "sar" and "iostat" to watch
disk activity and view the writes/second or I am sure there are other
tools you can use. Watch CPU and memory with "top" If it does appear
to be an I/O issue there are
some things you can do in either hardware or software, or if it is a
CPU/ memory issue building indexes or running updates on triggers
A simple suggestion is:
Move the bulk insert application to run during 'off' or 'slow'
hours if possible.
Some Software suggestions are:
Use the PG "Copy" to do the bulk insert
http://www.postgresql.org/docs/9.0/static/sql-copy.html
(or)
Drop the indexes (or triggers), do the inserts and build indexes
and triggers.
Some Hardware suggestions are dependendent on if it is I/O, CPU, or
memory bottleneck.
Deron
On Thu, Mar 29, 2012 at 11:59 AM, Campbell, Lance <lance@illinois.edu> wrote:
> PostgreSQL 9.0.x
>
> We have around ten different applications that use the same database. When
> one particular application is active it does an enormous number of inserts.
> Each insert is very small. During this time the database seems to slow down
> in general. The application in question is inserting into a particular
> table that is not used by the other applications.
>
>
>
> 1) What should I do to confirm that the database is the issue and not
> the applications?
>
> 2) How can I identify where the bottle neck is occurring if the issue
> happens to be with the database?
>
>
>
> I have been using PostgreSQL for eight years. It is an amazing database.
>
>
>
> Thanks,
>
>
>
> Lance Campbell
>
> Software Architect
>
> Web Services at Public Affairs
>
> 217-333-0382
>
>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:17 ` Re: database slowdown while a lot of inserts occur Deron <fecastle@gmail.com>
@ 2012-03-29 20:17 ` Shrirang Chitnis <Shrirang.Chitnis@hovservices.com>
0 siblings, 0 replies; 12+ messages in thread
From: Shrirang Chitnis @ 2012-03-29 20:17 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; Deron <fecastle@gmail.com>; +Cc: pgsql-performance
Lance,
Have faced the same issue with thousands of small inserts (actually they were inserts/updates) causing the database to slowdown. You have received good suggestions from the list, but listing them as points will make the issue clearer:
1) Disk configuration: RAID 5 was killing the performance after the database grew beyond 100 GB. Getting a RAID 10 with 12 spindles made a world of difference in my case. You can use iostat as Deron has suggested below to get information of latency which should help you find if disks are a bottleneck. Unless server RAM is very small and it also doubles up as application server or has other processes running, the RAM should not be a bottleneck.
Also have separate logging and data disks, which has been suggested in many posts in past.
2) Invoking Batch mode in program: In JDBC, there is a batch insert mode. Invoking the batch mode for a set of records has increased the efficiency of inserts in my case. It would be safe to suggest that use of batch mode in programming language you have used will give improved speeds.
3) Dropping indexes/ triggers: This will not work if the application has multiple instances running at same time OR if the insert is actually an insert/update.
4) You should think of using COPY command since you have mentioned that the table is NOT used by other applications, but caveat of multiple instances mentioned above will still hold true.
5) Enabling autovacuum and autoanalyse : A must. Infact you should force a vacuum and analyze if the insert batch is large.
HTH,
Shrirang Chitnis
The information contained in this message, including any attachments, is attorney privileged and/or confidential information intended only for the use of the individual or entity named as addressee. The review, dissemination, distribution or copying of this communication by or to anyone other than the intended addressee is strictly prohibited. If you have received this communication in error, please immediately notify the sender by replying to the message and destroy all copies of the original message.
-----Original Message-----
From: pgsql-performance-owner@postgresql.org [mailto:pgsql-performance-owner@postgresql.org] On Behalf Of Deron
Sent: Thursday, March 29, 2012 11:47 PM
To: Campbell, Lance
Cc: pgsql-performance@postgresql.org
Subject: Re: [PERFORM] database slowdown while a lot of inserts occur
On a Linux system you can use tools like "sar" and "iostat" to watch
disk activity and view the writes/second or I am sure there are other
tools you can use. Watch CPU and memory with "top" If it does appear
to be an I/O issue there are
some things you can do in either hardware or software, or if it is a
CPU/ memory issue building indexes or running updates on triggers
A simple suggestion is:
Move the bulk insert application to run during 'off' or 'slow'
hours if possible.
Some Software suggestions are:
Use the PG "Copy" to do the bulk insert
http://www.postgresql.org/docs/9.0/static/sql-copy.html
(or)
Drop the indexes (or triggers), do the inserts and build indexes
and triggers.
Some Hardware suggestions are dependendent on if it is I/O, CPU, or
memory bottleneck.
Deron
On Thu, Mar 29, 2012 at 11:59 AM, Campbell, Lance <lance@illinois.edu> wrote:
> PostgreSQL 9.0.x
>
> We have around ten different applications that use the same database. When
> one particular application is active it does an enormous number of inserts.
> Each insert is very small. During this time the database seems to slow down
> in general. The application in question is inserting into a particular
> table that is not used by the other applications.
>
>
>
> 1) What should I do to confirm that the database is the issue and not
> the applications?
>
> 2) How can I identify where the bottle neck is occurring if the issue
> happens to be with the database?
>
>
>
> I have been using PostgreSQL for eight years. It is an amazing database.
>
>
>
> Thanks,
>
>
>
> Lance Campbell
>
> Software Architect
>
> Web Services at Public Affairs
>
> 217-333-0382
>
>
--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
@ 2012-03-31 01:21 ` Tomas Vondra <tv@fuzzy.cz>
3 siblings, 0 replies; 12+ messages in thread
From: Tomas Vondra @ 2012-03-31 01:21 UTC (permalink / raw)
To: pgsql-performance
Hi,
On 29.3.2012 19:59, Campbell, Lance wrote:
> PostgreSQL 9.0.x
>
> We have around ten different applications that use the same database.
> When one particular application is active it does an enormous number of
> inserts. Each insert is very small. During this time the database
> seems to slow down in general. The application in question is inserting
> into a particular table that is not used by the other applications.
Can you provide more info? Show us some vmstat / 'iostat -x' logs so
that we can see what kind of bottleneck are you hitting. Provide more
details about your system (especially I/O subsystem - what drives, what
RAID config etc.)
Also, we need more details about the workload. Is each INSERT a separate
transaction or are they grouped into transactions fo multiple INSERTs?
> 1) What should I do to confirm that the database is the issue and
> not the applications?
Well, usually the application is the culprit. Some applications are
designed so that it's almost certain there was a 'let's poke the
database as hard as possible' goal at the beginning. Not sure it's this
case, though.
Might be a misconfigured database too - what are the basic parameters
(shared buffers, ...)?
> 2) How can I identify where the bottle neck is occurring if the
> issue happens to be with the database?
Watching 'iostat -x' or 'top' will usually point you the right
direction. Is the CPU fully utilized => you're doing something that
needs more CPU time than you have? Is the I/O wait high (say above 50%)?
Well, you have issues with I/O bottlenecks (either random or
sequential). The less visible bottlenecks are usually related to memory,
bus bandwidth etc.
Tomas
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: database slowdown while a lot of inserts occur
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
@ 2012-04-02 07:14 ` Віталій Тимчишин <tivv00@gmail.com>
3 siblings, 0 replies; 12+ messages in thread
From: Віталій Тимчишин @ 2012-04-02 07:14 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; +Cc: pgsql-performance
Few words regarding small inserts and a lot of fsyncs:
If it is your problem, you can fix this by using battery-backed raid card.
Similar effect can be reached by turning synchronious commit off. Note
that the latter may make few last commits lost in case of sudden reboot.
But you can at least test if moving to BBU will help you. (Dunno if this
setting can be changed with SIGHUP without restart).
Note that this may still be a lot of random writes. And in case of RAID5 -
a lot of random reads too. I don't think batching will help other
applications. This is the tool to help application that uses batching. If
you have random writes, look at HOT updates - they may help you if you will
follow requirements.
Check your checkpoints - application writes to commit log first (sequential
write), then during checkpoints data is written to tables (random writes) -
longer checkpoints may make you life easier. Try to increase
checkpoint_segments.
If you have alot of data written - try to move you commit logs to another
drive/partition.
If you have good raid card with memory and BBU, you may try to disable read
cache on it (leaving only write cache). Read cache is usually good at OS
level (with much more memory) and fast writes need BBU-protected write
cache.
Best regards, Vitalii Tymchyshyn
2012/3/29 Campbell, Lance <lance@illinois.edu>
> PostgreSQL 9.0.x****
>
> We have around ten different applications that use the same database.
> When one particular application is active it does an enormous number of
> inserts. Each insert is very small. During this time the database seems
> to slow down in general. The application in question is inserting into a
> particular table that is not used by the other applications.****
>
> ** **
>
> **1) **What should I do to confirm that the database is the issue
> and not the applications?****
>
> **2) **How can I identify where the bottle neck is occurring if the
> issue happens to be with the database?****
>
> ** **
>
> I have been using PostgreSQL for eight years. It is an amazing database.*
> ***
>
> ** **
>
> Thanks,****
>
> ** **
>
> Lance Campbell****
>
> Software Architect****
>
> Web Services at Public Affairs****
>
> 217-333-0382****
>
> ** **
>
--
Best regards,
Vitalii Tymchyshyn
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2012-04-02 07:14 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-03-29 17:59 database slowdown while a lot of inserts occur Campbell, Lance <lance@illinois.edu>
2012-03-29 18:02 ` Campbell, Lance <lance@illinois.edu>
2012-03-29 18:14 ` Filippos Kalamidas <filippos.kal@gmail.com>
2012-03-29 19:27 ` Bob Lunney <bob_lunney@yahoo.com>
2012-03-29 19:39 ` Andrew Dunstan <andrew@dunslane.net>
2012-03-31 01:11 ` Tomas Vondra <tv@fuzzy.cz>
2012-04-01 02:20 ` Bob Lunney <bob_lunney@yahoo.com>
2012-03-29 21:36 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-03-29 18:17 ` Deron <fecastle@gmail.com>
2012-03-29 20:17 ` Shrirang Chitnis <Shrirang.Chitnis@hovservices.com>
2012-03-31 01:21 ` Tomas Vondra <tv@fuzzy.cz>
2012-04-02 07:14 ` Віталій Тимчишин <tivv00@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox