pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedstrangest thing happened
11+ messages / 7 participants
[nested] [flat]
* strangest thing happened
@ 2010-07-07 19:59 John <johnf@jfcomputer.com>
0 siblings, 3 replies; 11+ messages in thread
From: John @ 2010-07-07 19:59 UTC (permalink / raw)
To: pgsql-sql
I am the only developer, DBA etc.. for a small project. Today (yesterday was
everything was perfect) many of the sequence numbers fell behind what is the
actual PK value. For example the invoice PK sequence current value = 1056
but the table PK was 1071. Nobody (other than myself) knows how to
edit/access the postgres server. So
1. Does anyone know how this could have happened?????? Other than human
interaction.
2. Does anyone have a script to reset the sequences to match the tables?
Thanks in advance,
Johnf
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-07 20:38 John <johnf@jfcomputer.com>
parent: Justin Graf <justin@magwerks.com>
1 sibling, 0 replies; 11+ messages in thread
From: John @ 2010-07-07 20:38 UTC (permalink / raw)
To: pgsql-sql
Yes I'm using auto_increment (serial data type). No function, method inserts
PK's anywhere in my code. I'm thinking/guessing it had something to do with
vacumn or the backup. I have been using Postgres for a number of years now
and I never seen this in the past. So I'm really at a loss as how this could
have occurred. The backup is a windows product "exec" and I'm using a
special plug-in from exec for the Linux backup. But I still can't see this
actually happening.
Well I have it running for the moment and I'll have to account black magic as
the cause.
Johnf
On Wednesday 07 July 2010 02:25:13 pm Justin Graf wrote:
> Are you using PG's sequence/auto increment???
>
> If so.
> Once PG fires off the nextval() for the sequence that number is
> considered used and gone even if the transaction that called nextval()
> is rolled back
>
> Depending on how the app is written nextval() might be called, but allow
> the User to cancel the invoice creation before the insert into table is
> completed eating up Invoice numbers
>
> To reset Sequences number call
> Select setval('Sequence_Name', VAlue_To_Set_To);
>
> Most people ignore this kind of annoyance when sequence numbers jump.
> Now if it happens all the time where every X hours eating up Z number of
> sequence numbers then one needs to dig into the logs and figure out what
> is calling nextval()
>
> Search the logs to see what is calling nextval('My_Sequence')
>
> You may need to turn up logging to find it.
>
> On 7/7/2010 2:59 PM, John wrote:
> > I am the only developer, DBA etc.. for a small project. Today (yesterday
> > was everything was perfect) many of the sequence numbers fell behind what
> > is the actual PK value. For example the invoice PK sequence current
> > value = 1056 but the table PK was 1071. Nobody (other than myself) knows
> > how to edit/access the postgres server. So
> >
> > 1. Does anyone know how this could have happened?????? Other than human
> > interaction.
> >
> > 2. Does anyone have a script to reset the sequences to match the tables?
> >
> > Thanks in advance,
> >
> > Johnf
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-07 20:42 Ross J. Reedstrom <reedstrm@rice.edu>
parent: Justin Graf <justin@magwerks.com>
1 sibling, 1 reply; 11+ messages in thread
From: Ross J. Reedstrom @ 2010-07-07 20:42 UTC (permalink / raw)
To: pgsql-sql
On Wed, Jul 07, 2010 at 04:25:13PM -0500, Justin Graf wrote:
> Are you using PG's sequence/auto increment???
>
> If so.
> Once PG fires off the nextval() for the sequence that number is
> considered used and gone even if the transaction that called nextval()
> is rolled back
>
> Depending on how the app is written nextval() might be called, but allow
> the User to cancel the invoice creation before the insert into table is
> completed eating up Invoice numbers
>
> To reset Sequences number call
> Select setval('Sequence_Name', VAlue_To_Set_To);
>
> Most people ignore this kind of annoyance when sequence numbers jump.
> Now if it happens all the time where every X hours eating up Z number of
> sequence numbers then one needs to dig into the logs and figure out what
> is calling nextval()
>
> Search the logs to see what is calling nextval('My_Sequence')
>
> You may need to turn up logging to find it.
Justin, you're missing that John reported that the sequences are
_behind_ the table. This only happens for me if I've been doing
bulk data loads. Then I use:
select setval(sequence_name,max(serial_id_column)) from table_with_serial_id;
You do need to trackdown how this might have happened, though. Any
clever code doing it's own 'serial' incrementing?
Ross
--
Ross Reedstrom, Ph.D. reedstrm@rice.edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
The Connexions Project http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE
>
>
> On 7/7/2010 2:59 PM, John wrote:
> > I am the only developer, DBA etc.. for a small project. Today (yesterday was
> > everything was perfect) many of the sequence numbers fell behind what is the
> > actual PK value. For example the invoice PK sequence current value = 1056
> > but the table PK was 1071. Nobody (other than myself) knows how to
> > edit/access the postgres server. So
> >
> > 1. Does anyone know how this could have happened?????? Other than human
> > interaction.
> >
> > 2. Does anyone have a script to reset the sequences to match the tables?
> >
> > Thanks in advance,
> >
> > Johnf
> >
> >
>
>
>
> All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored.
>
> CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately.
> Thank you.
>
> --
> Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-sql
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-07 20:45 Joe Conway <mail@joeconway.com>
parent: John <johnf@jfcomputer.com>
2 siblings, 1 reply; 11+ messages in thread
From: Joe Conway @ 2010-07-07 20:45 UTC (permalink / raw)
To: John <johnf@jfcomputer.com>; +Cc: pgsql-sql
On 07/07/2010 12:59 PM, John wrote:
> I am the only developer, DBA etc.. for a small project. Today (yesterday was
> everything was perfect) many of the sequence numbers fell behind what is the
> actual PK value. For example the invoice PK sequence current value = 1056
> but the table PK was 1071. Nobody (other than myself) knows how to
> edit/access the postgres server. So
>
> 1. Does anyone know how this could have happened?????? Other than human
> interaction.
I've never heard of this happening. Are you certain nothing bypassed the
sequence and directly inserted a PK value?
> 2. Does anyone have a script to reset the sequences to match the tables?
Not heavily tested, but something like this might do the trick:
8<----------------------
CREATE OR REPLACE FUNCTION adjust_seqs(namespace text)
RETURNS text AS $$
DECLARE
rec record;
startval bigint;
sql text;
seqname text;
BEGIN
FOR rec in EXECUTE 'select table_name, column_name, column_default
from information_schema.columns
where table_schema = ''' || namespace || '''
and column_default like ''nextval%''' LOOP
seqname := pg_get_serial_sequence(rec.table_name, rec.column_name);
sql := 'select max(' || rec.column_name || ') + 1 from ' ||
rec.table_name;
EXECUTE sql INTO startval;
IF startval IS NOT NULL THEN
sql := 'ALTER SEQUENCE ' || seqname || ' RESTART WITH ' ||
startval;
EXECUTE sql;
RAISE NOTICE '%', sql;
END IF;
END LOOP;
RETURN 'OK';
END;
$$ LANGUAGE plpgsql STRICT;
select adjust_seqs('public');
8<----------------------
HTH,
Joe
--
Joe Conway
credativ LLC: http://www.credativ.us
Linux, PostgreSQL, and general Open Source
Training, Service, Consulting, & Support
Attachments:
[application/pgp-signature] signature.asc (898B, ../../4C34E764.3040703@joeconway.com/2-signature.asc)
download
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-07 21:25 Justin Graf <justin@magwerks.com>
parent: John <johnf@jfcomputer.com>
2 siblings, 2 replies; 11+ messages in thread
From: Justin Graf @ 2010-07-07 21:25 UTC (permalink / raw)
To: John <johnf@jfcomputer.com>; pgsql-sql
Are you using PG's sequence/auto increment???
If so.
Once PG fires off the nextval() for the sequence that number is
considered used and gone even if the transaction that called nextval()
is rolled back
Depending on how the app is written nextval() might be called, but allow
the User to cancel the invoice creation before the insert into table is
completed eating up Invoice numbers
To reset Sequences number call
Select setval('Sequence_Name', VAlue_To_Set_To);
Most people ignore this kind of annoyance when sequence numbers jump.
Now if it happens all the time where every X hours eating up Z number of
sequence numbers then one needs to dig into the logs and figure out what
is calling nextval()
Search the logs to see what is calling nextval('My_Sequence')
You may need to turn up logging to find it.
On 7/7/2010 2:59 PM, John wrote:
> I am the only developer, DBA etc.. for a small project. Today (yesterday was
> everything was perfect) many of the sequence numbers fell behind what is the
> actual PK value. For example the invoice PK sequence current value = 1056
> but the table PK was 1071. Nobody (other than myself) knows how to
> edit/access the postgres server. So
>
> 1. Does anyone know how this could have happened?????? Other than human
> interaction.
>
> 2. Does anyone have a script to reset the sequences to match the tables?
>
> Thanks in advance,
>
> Johnf
>
>
All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored.
CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately.
Thank you.
Attachments:
[text/x-vcard] justin.vcf (256B, ../../4C34F0B9.5020902@magwerks.com/2-justin.vcf)
download | inline:
begin:vcard
fn:Justin Graf
n:Graf;Justin
org:Magwerks Corp
adr:;;501 Commerce Drive;Danville ;IN;46122;USA
email;internet:justin@magwerks.com
tel;work:317-241-8011 ext 703
tel;fax:317-241-8015
x-mozilla-html:FALSE
url:www.magwerks.com
version:2.1
end:vcard
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-07 22:14 Justin Graf <justin@magwerks.com>
parent: Ross J. Reedstrom <reedstrm@rice.edu>
0 siblings, 2 replies; 11+ messages in thread
From: Justin Graf @ 2010-07-07 22:14 UTC (permalink / raw)
To: reedstrm@rice.edu, pgsql-sql@postgresql.org, John <johnf@jfcomputer.com>
On 7/7/2010 3:42 PM, Ross J. Reedstrom wrote:
>
> Justin, you're missing that John reported that the sequences are
> _behind_ the table. This only happens for me if I've been doing
> bulk data loads. Then I use:
>
> select setval(sequence_name,max(serial_id_column)) from table_with_serial_id;
>
> You do need to trackdown how this might have happened, though. Any
> clever code doing it's own 'serial' incrementing?
>
> Ross
>
Yes i did miss read his statement, oops =-O
The highest PK value in the table is 1071 but the next sequence is
1056. That's interesting and could be a big problem
Quoteing JonF
------------------------------------------------
I'm thinking/guessing it had something to do with
vacumn or the backup.
The backup is a windows product "exec" and I'm using a
special plug-in from exec for the Linux backup. But I still can't see this
actually happening.
--------------------------------------------------
BakupExec HMMM. Are you doing a file level backup, meaning backing up
PGDATA folder or are you doing pg_dump??
I don't think its a backup issue, unless you have done a restore. Which
this would say there are more problems else where
Are there invoices that use up numbers 1056 to 1071 in that table???
Does the app allow for resetting Sequence in a admin interface??? Many
apps have such features and someone could have accidentally rest the
value???
I would be looking at the log files for the Inserts into that table as a
means to track down what is the cause. If there are no log files or
don't have enough detail, crank up the logging level and wait for it to
happen again???
All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored.
CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately.
Thank you.
Attachments:
[text/x-vcard] justin.vcf (256B, ../../4C34FC50.6070203@magwerks.com/2-justin.vcf)
download | inline:
begin:vcard
fn:Justin Graf
n:Graf;Justin
org:Magwerks Corp
adr:;;501 Commerce Drive;Danville ;IN;46122;USA
email;internet:justin@magwerks.com
tel;work:317-241-8011 ext 703
tel;fax:317-241-8015
x-mozilla-html:FALSE
url:www.magwerks.com
version:2.1
end:vcard
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-07 22:41 John <johnf@jfcomputer.com>
parent: Justin Graf <justin@magwerks.com>
1 sibling, 1 reply; 11+ messages in thread
From: John @ 2010-07-07 22:41 UTC (permalink / raw)
To: pgsql-sql
On Wednesday 07 July 2010 03:14:40 pm Justin Graf wrote:
> I would be looking at the log files for the Inserts into that table as a
> means to track down what is the cause. If there are no log files or
> don't have enough detail, crank up the logging level and wait for it to
> happen again???
That is scary - let it happen again???? I'm not keeping enough info in the
log. I actually turned off most of the info the log files are gathering
because the system has been running for 6-7 months without an issue. I just
got a call around noon telling me something was going wrong. That's when I
discovered the sequences were the wrong values. I'm sure there has to be
some sort of real explanation - but I don't know what it is.
Johnf
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-08 06:36 Viktor Bojović <viktor.bojovic@gmail.com>
parent: John <johnf@jfcomputer.com>
2 siblings, 0 replies; 11+ messages in thread
From: Viktor Bojović @ 2010-07-08 06:36 UTC (permalink / raw)
To: John <johnf@jfcomputer.com>; +Cc: pgsql-sql
On Wed, Jul 7, 2010 at 9:59 PM, John <johnf@jfcomputer.com> wrote:
> I am the only developer, DBA etc.. for a small project. Today (yesterday
> was
> everything was perfect) many of the sequence numbers fell behind what is
> the
> actual PK value. For example the invoice PK sequence current value = 1056
> but the table PK was 1071. Nobody (other than myself) knows how to
> edit/access the postgres server. So
>
> 1. Does anyone know how this could have happened?????? Other than human
> interaction.
>
> 2. Does anyone have a script to reset the sequences to match the tables?
>
> Thanks in advance,
>
> Johnf
>
> --
> Sent via pgsql-sql mailing list (pgsql-sql@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-sql
>
John,
If the insert is performed to triggered table, and that trigger returns null
(doesn't insert) then the sequence will increment, but no data will be
inserted. If needed I will send you some code examples where it happened to
me many times.
Sincerely
--
---------------------------------------
Viktor Bojović
---------------------------------------
Wherever I go, Murphy goes with me
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-08 07:37 Ben Morrow <ben@morrow.me.uk>
parent: Justin Graf <justin@magwerks.com>
1 sibling, 0 replies; 11+ messages in thread
From: Ben Morrow @ 2010-07-08 07:37 UTC (permalink / raw)
To: johnf@jfcomputer.com; pgsql-sql
Quoth johnf@jfcomputer.com (John):
> On Wednesday 07 July 2010 03:14:40 pm Justin Graf wrote:
> > I would be looking at the log files for the Inserts into that table as a
> > means to track down what is the cause. If there are no log files or
> > don't have enough detail, crank up the logging level and wait for it to
> > happen again???
>
>
> That is scary - let it happen again???? I'm not keeping enough info in the
> log. I actually turned off most of the info the log files are gathering
> because the system has been running for 6-7 months without an issue. I just
> got a call around noon telling me something was going wrong. That's when I
> discovered the sequences were the wrong values. I'm sure there has to be
> some sort of real explanation - but I don't know what it is.
There are several possible causes:
- Something somewhere is inserting values directly into the serial
columns, without using the sequence. This can be prevented by
REVOKEing INSERT and UPDATE on the relevant columns for all users.
If this causes problems anywhere in your app, those are good
places to start looking for bugs.
- Something somewhere is manipulating the sequence. This can be
prevented by REVOKEing UPDATE on all sequences for all users. You
may need some additional GRANTs of USAGE on sequences if parts of
the app were relying on UPDATE to call nextval().
Obviously if your app routinely drops and creates tables you will need to
arrange for these permissions to be applied every time.
- The database has become corrupted, perhaps by a badly-done backup
and restore. (I would not expect taking a backup alone to cause
corruption, but if the backup isn't done right the backed-up copy
may be corrupt.) Have you done a restore recently?
- Something I haven't thought of :).
- A bug in Pg. While this is *extremely* unlikely, it must be
mentioned as a possibility.
Ben
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-08 14:52 Justin Graf <justin@magwerks.com>
parent: John <johnf@jfcomputer.com>
0 siblings, 0 replies; 11+ messages in thread
From: Justin Graf @ 2010-07-08 14:52 UTC (permalink / raw)
To: pgsql-sql@postgresql.org, John <johnf@jfcomputer.com>
On 7/7/2010 5:41 PM, John wrote:
> On Wednesday 07 July 2010 03:14:40 pm Justin Graf wrote:
>
>> I would be looking at the log files for the Inserts into that table as a
>> means to track down what is the cause. If there are no log files or
>> don't have enough detail, crank up the logging level and wait for it to
>> happen again???
>>
>
> That is scary - let it happen again???? I'm not keeping enough info in the
> log. I actually turned off most of the info the log files are gathering
> because the system has been running for 6-7 months without an issue. I just
> got a call around noon telling me something was going wrong. That's when I
> discovered the sequences were the wrong values. I'm sure there has to be
> some sort of real explanation - but I don't know what it is.
>
> Johnf
>
>
Sometimes we just don't have a choice but to let things become broke
again to figure out what is the cause.
I had an odd case where Parent records could become deleted every once
in a while. For what appeared to be no rhyme or reason
The app had around 1000 functions in pg/psql not counting triggers, so
figuring out the cause by just staring at the code was not practical.
The log was already was set to record
log_statement (all)
log_line_prefix ('User %u, DB%d, Client%r, PID %p, Time %m, SID %c,
LineCount %l , TID %x);
log_destination ('csvlog')
The logs rotated out every 30 days. So I had good sample of the
commands sent to PG to figure out what went wrong
As the statements are logged all i had to do was search for
Delete from cohead where cohead_id = XXXX
this gave me the transaction ID and the Session ID to start backtracking
to see if a Trigger or Function issued the delete. Once I had figured
out that it was pg/psql procedure, i needed to figure out where in the
application called this seriously miss thought out DELETE Sales Order
function. I threw in a RAISE EXCEPTION in the psql and waited for the
Data entry people to come screaming.
Around 3 weeks later a data entry girl came and found me asking to
explain why her computer ordered her to come find me ASAP and why
nothing else matter but stop and find me.
With that error, I now the call stack from the app to figure out the
serious of events that allowed a Sales Order to be deleted.
After all was said and done there was a total of 50 records deleted out
of 60,000 (not even 0.1%) not a big deal unless you are customer who's
order was deleted.
My experience has taught me never turn off logging because we never when
we may need it.
All legitimate Magwerks Corporation quotations are sent in a .PDF file attachment with a unique ID number generated by our proprietary quotation system. Quotations received via any other form of communication will not be honored.
CONFIDENTIALITY NOTICE: This e-mail, including attachments, may contain legally privileged, confidential or other information proprietary to Magwerks Corporation and is intended solely for the use of the individual to whom it addresses. If the reader of this e-mail is not the intended recipient or authorized agent, the reader is hereby notified that any unauthorized viewing, dissemination, distribution or copying of this e-mail is strictly prohibited. If you have received this e-mail in error, please notify the sender by replying to this message and destroy all occurrences of this e-mail immediately.
Thank you.
Attachments:
[text/x-vcard] justin.vcf (271B, ../../4C35E61B.3010904@magwerks.com/2-justin.vcf)
download | inline:
begin:vcard
fn:Justin Graf
n:Graf;Justin
org:Magwerks Corp
adr:;;501 Commerce Drive;Danville ;IN;46122;USA
email;internet:justin@magwerks.com
tel;work:317-241-8011 ext 703
tel;fax:317-241-8015
x-mozilla-html:FALSE
url:www.magwerks.com
version:2.1
end:vcard
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: strangest thing happened
@ 2010-07-08 14:57 Torsten Zühlsdorff <foo@meisterderspiele.de>
parent: Joe Conway <mail@joeconway.com>
0 siblings, 0 replies; 11+ messages in thread
From: Torsten Zühlsdorff @ 2010-07-08 14:57 UTC (permalink / raw)
To: pgsql-sql
Joe Conway schrieb:
>> I am the only developer, DBA etc.. for a small project. Today (yesterday was
>> everything was perfect) many of the sequence numbers fell behind what is the
>> actual PK value. For example the invoice PK sequence current value = 1056
>> but the table PK was 1071. Nobody (other than myself) knows how to
>> edit/access the postgres server. So
>>
>> 1. Does anyone know how this could have happened?????? Other than human
>> interaction.
>
> I've never heard of this happening. Are you certain nothing bypassed the
> sequence and directly inserted a PK value?
Maybe the sequence is defined with "cycle" and starts again? Very
unlikly, but a possibility.
Otherwise i would expect an bad backup/restore, direct insert of serials
or manipulated them afterwards (by insert, update or an function).
Gretings from Germany,
Torsten
--
http://www.dddbl.de - ein Datenbank-Layer, der die Arbeit mit 8
verschiedenen Datenbanksystemen abstrahiert,
Queries von Applikationen trennt und automatisch die Query-Ergebnisse
auswerten kann.
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2010-07-08 14:57 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2010-07-07 19:59 strangest thing happened John <johnf@jfcomputer.com>
2010-07-07 20:45 ` Joe Conway <mail@joeconway.com>
2010-07-08 14:57 ` Torsten Zühlsdorff <foo@meisterderspiele.de>
2010-07-07 21:25 ` Justin Graf <justin@magwerks.com>
2010-07-07 20:38 ` John <johnf@jfcomputer.com>
2010-07-07 20:42 ` Ross J. Reedstrom <reedstrm@rice.edu>
2010-07-07 22:14 ` Justin Graf <justin@magwerks.com>
2010-07-07 22:41 ` John <johnf@jfcomputer.com>
2010-07-08 14:52 ` Justin Graf <justin@magwerks.com>
2010-07-08 07:37 ` Ben Morrow <ben@morrow.me.uk>
2010-07-08 06:36 ` Viktor Bojović <viktor.bojovic@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox