Subject: Email not searchable in our archives

From: Bruce Momjian <[email-masked]>
Date: Mon, 10 Mar 2008 14:55:44 -0400 (EDT)

Would someone please fine out why the attached email from Heikki is not
appearing in a search of our archives?  I tried the subject and a line
from the email and neither came up as a hit:

  http://search.postgresql.org/search?q=git+patch+review&m=1&l=&d=-1&s=r

-- 
  Bruce Momjian  <[email-masked]>        http://momjian.us
  EnterpriseDB                             http://postgres.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

---

From: Tom Lane <[email-masked]>
Date: Mon, 10 Mar 2008 15:42:21 -0400

Bruce Momjian <[email-masked]> writes:
> appearing in a search of our archives?

I've noticed some curious omissions lately, too.  For instance I
searched for "Include Lists for Text Search" this morning, and
successfully got hits on yesterday's and today's posts with that
title, but not the ones I wanted from last September.  Perhaps there
are chunks of last year that are missing from the tsearch index
for some reason?

			regards, tom lane

---

From: "Dave Page" <[email-masked]>
Date: Mon, 10 Mar 2008 19:50:25 +0000

>   http://search.postgresql.org/search?q=git+patch+review&m=1&l=&d=-1&s=r

Eh? The message you included is at the top of the results when I use
the query above.

-- 
Dave Page
EnterpriseDB UK Ltd: http://www.enterprisedb.com
PostgreSQL UK 2008 Conference: http://www.postgresql.org.uk

---

From: Alvaro Herrera <[email-masked]>
Date: Mon, 10 Mar 2008 17:43:49 -0300

BTW, your mail broke our archiver.  If you go to your message page,
http://archives.postgresql.org/pgsql-www/2008-03/msg00183.php
you'll notice it only displays your part of the message -- not the
attached message.  Then you notice that the date index has no
msg00184.php nearby ... until you go to the end of it and you notice
that there's a message from Heikki dated 23 May 2007.

The problem here is that your message contains a text/plain attachment
with the dreaded "^From " line, which causes Mhonarc to think it's a
separate message.  Not sure if there's something we can do about this.
One idea would be making the separator contain the list address in the
"^From " line.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

---

From: Bruce Momjian <[email-masked]>
Date: Mon, 10 Mar 2008 23:25:29 -0400 (EDT)

> Eh? The message you included is at the top of the results when I use

It shows up now for me too, but did not this morning, about 12 hours
ago.

---

From: Bruce Momjian <[email-masked]>
Date: Mon, 10 Mar 2008 23:29:53 -0400 (EDT)

> One idea would be making the separator contain the list address in the

Yea, I can see how that would happen. Sorry.  I can probably modify my
mailer to escape those "From" lines but that isn't going to fix it for
other posters.

---

From: Tom Lane <[email-masked]>
Date: Mon, 10 Mar 2008 23:37:40 -0400

> mailer to escape those "From" lines but that isn't going to fix it for

The "From " to ">From " kluge is supposed to happen during delivery into
a Unix-format mailbox.  There *is not* any restriction on messages in
flight that they not contain lines starting with "From ".  So if this
broke the archives, the fault is on the archives' side not Bruce's.

http://www.faqs.org/rfcs/rfc822.html

			regards, tom lane

---

From: Andrew Sullivan <[email-masked]>
Date: Mon, 10 Mar 2008 23:45:26 -0400

> http://www.faqs.org/rfcs/rfc822.html

Well, you probably want to look at 2821 and 2822, also, but you're quite
right.  The way the archiver is using From sounds like a filthy hack to me.

A

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 11 Mar 2008 08:10:40 -0300

> broke the archives, the fault is on the archives' side not Bruce's.

Well, I'm not sure the problem is the delivery either, because the
"From " line here occured in an attachment, not the email body itself.
I think this is more a bug in Mhonarc's message separation, which is way
too primitive.

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 11 Mar 2008 08:12:09 -0300

> It shows up now for me too, but did not this morning, about 12 hours

Of course it shows up, but it's the copy added to the 2008-03 mbox.
Note the URL.  The original still doesn't appear.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 11 Mar 2008 09:02:07 -0400 (EDT)

> Note the URL.  The original still doesn't appear.

Oh, I see now, yea.  So who is going to find out why that email is
missing?

---

From: "Dave Page" <[email-masked]>
Date: Tue, 11 Mar 2008 13:08:12 +0000

>  Oh, I see now, yea.  So who is going to find out why that email is

Google seems to find the May 07 version, so it must be in the
archives. Any ideas Magnus - being your code 'n' all :-) ?

---

From: Tom Lane <[email-masked]>
Date: Tue, 11 Mar 2008 10:57:43 -0400

> I think this is more a bug in Mhonarc's message separation, which is way

Whether it's an attachment or not is irrelevant --- the standards for
this don't even know that there is such a thing as an attachment.

			regards, tom lane

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 11 Mar 2008 12:15:23 -0300

> this don't even know that there is such a thing as an attachment.

Oh, RFC2822 clearly does -- it refers to RFC2045 through 20499, which
define MIME.  In any case, that "From " line is not defined by RFC822
either, it is purely an implementation matter.  As such, Mhonarc would
stand better if it followed the MIME standard which says that the
message should be split at the terminators defined in the header.  Only
if no such terminators are defined the "From " line should be used.

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 11 Mar 2008 11:16:45 -0400

> this don't even know that there is such a thing as an attachment.

That's not quite right: MIME knows about message parts, which is really what
we mean by "attachment".  My bet is that the problem (I haven't looked at
the pieces that are doing this) is that something (mhonarc?) isn't handling
MIME message parts correctly (maybe during the transition to mailbox
format?).

A

---

From: Tom Lane <[email-masked]>
Date: Tue, 11 Mar 2008 11:25:35 -0400

> MIME message parts correctly (maybe during the transition to mailbox

Right, the problem is exactly that Unix mbox format knows about "From "
(and nothing else) as a message separator.  Whatever code dumps messages
into such a file *must* escape data lines beginning with "From ".

			regards, tom lane

---

From: Alvaro Herrera <[email-masked]>
Date: Tue, 11 Mar 2008 12:30:25 -0300

> into such a file *must* escape data lines beginning with "From ".

It would be Majordomo's fault then.

However, I think you'd find that if you complain about it to them, they
will tell you that they correctly handle the "From " line in the message
body but they don't touch it inside MIME parts.  And they would be right ...
Still, it would be a good idea to ask.