Subject: Spam filtering on the mailing lists

From: "Greg Sabino Mullane" <[email-masked]>
Date: Wed, 16 Jul 2008 14:14:38 -0000

I'd like to reiterate my request for some (or better?) spam filtering on
the lists. I just rejected over 50 messages to pgsql-bugs, all of them
spam and 95% of them would be easily caught by the worst spam filter
on the market. The chance of the moderators mistakenly rejecting a
legitimate message increases as the volume of spam does. If this is
something the project cannot handle for technical/political reasons,
there are plenty of third-party spam filtering solutions out there,
some of which are using Postgres and would probably be happy to provide
the service to us.

- --
Greg Sabino Mullane [email-masked]
PGP Key: 0x14964AC8 200807161012
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8

---

From: Bruce Momjian <[email-masked]>
Date: Wed, 16 Jul 2008 11:09:27 -0400 (EDT)

> some of which are using Postgres and would probably be happy to provide

I am confused --- we have no spam filter for the bugs list?

-- 
  Bruce Momjian  <[email-masked]>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

---

From: "Joshua D. Drake" <[email-masked]>
Date: Wed, 16 Jul 2008 10:01:13 -0700

> I am confused --- we have no spam filter for the bugs list?

We do I am sure, I just find that @postgresql.org spam filtering is
lacking. I get more spam at my @postgresql.org address than my
@commandprompt.com address by far.

I have just learned to overlook it.

Joshua D. Drake

-- 
The PostgreSQL Company since 1997: http://www.commandprompt.com/ 
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

---

From: Andrew Sullivan <[email-masked]>
Date: Wed, 16 Jul 2008 14:07:38 -0400

> I have just learned to overlook it.

But moderators can't overlook it.  We have to troll through piles of
it.  Much of it is in Cyrillic or CJK, and is just obviously not aimed
at the list.  We still have to process it.

(I say "we" even though I am still unable to send mail from the
account I moderate from, and therefore can't do any useful
moderating.)

A

-- 
Andrew Sullivan
[email-masked]
+1 503 667 4564 x104
http://www.commandprompt.com/

---

From: "Joshua D. Drake" <[email-masked]>
Date: Wed, 16 Jul 2008 15:35:15 -0700

> at the list.  We still have to process it.

Yeah I thought about it afterword, I moderate announce. I get maybe a
dozen a day.

Joshua D. Drake

---

From: "Marc G. Fournier" <[email-masked]>
Date: Wed, 16 Jul 2008 23:17:22 -0300

Its sad how this is such an ongoing problem, but this is the first that I hear 
that ppl are having problems ... looking at the message headers for a random 
few, I notice that they are scoring >4, but just below 5:

X-Spam-Status: No, hits=4.855 tagged_above=0 required=5 tests=AWL=-1.994,
 DCC_CHECK=1.37, DIGEST_MULTIPLE=0.001, HTML_MESSAGE=0.001,
 MIME_HTML_ONLY=1.672, RAZOR2_CHECK=0.5, RCVD_IN_BL_SPAMCOP_NET=2.188,
 RCVD_IN_SORBS_WEB=1.117
X-Spam-Level: ****

I can change the quarantining to be >4 if ppl want, which should greatly reduce 
the # of messages going through ...

Thoughts?

- --On Wednesday, July 16, 2008 14:07:38 -0400 Andrew Sullivan 
<[email-masked]> wrote:

> Sent via pgsql-www mailing list ([email-masked])
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-www

- -- 
Marc G. Fournier        Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . [email-masked]                              MSN . [email-masked]
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664

---

From: "Joshua D. Drake" <[email-masked]>
Date: Wed, 16 Jul 2008 19:58:13 -0700

On Wed, 2008-07-16 at 23:17 -0300, Marc G. Fournier wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> that ppl are having problems ...

I can only assume you mean "this time".

Joshua D. Drake

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 17 Jul 2008 00:01:08 -0300

I just cleared out the AWL ... going through the stuff label'd as 'non-spam', 
am finding alot of stuff scoring a bit low due to the From being in AWL ...

Does anyone know if there is some 'rule' that return mail about 'Undeliverable' 
has to be in English?  I'm finding some that *look* like that sort of thing, 
but in other languages / character sets, and wonder if there is an RFC that is 
meant to direct such stuff 'in English'?  (ie. should I be marking that stuff 
as spam, since *I* can't confirm it isn't?)

- --On Wednesday, July 16, 2008 23:17:22 -0300 "Marc G. Fournier" 
<[email-masked]> wrote:

> Its sad how this is such an ongoing problem, but this is the first that I
> hear  that ppl are having problems ... looking at the message headers for a
> random  few, I notice that they are scoring >4, but just below 5:
>
> I can change the quarantining to be >4 if ppl want, which should greatly
> reduce  the # of messages going through ...
>
> Thoughts?
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2.0.9 (FreeBSD)
>
> iEYEARECAAYFAkh+q7IACgkQ4QvfyHIvDvPYfQCfchdCwNpL+90T5maoAWZUZttL
> 97wAoLMRAkqfszOxXojM7YNbUIZBX5ug
> =9v1h
> -----END PGP SIGNATURE-----
>
>
> --

- --

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 17 Jul 2008 08:45:46 -0400

On Thu, Jul 17, 2008 at 12:01:08AM -0300, Marc G. Fournier wrote:

> has to be in English?  

Nope, there isn't.  The bounce message is intended for human
consumption, and therefore localisation is approproate. 

Note, however, that many of the messages should have a Content-Type:
multipart/report; report-type=delivery-status; header.  See RFC 3462,
which obsoletes RFC 1892.

RFC 3462 requires a machine-parsable delivery report, and one such
report is defined in RFC 3464.  Many mail systems (most?) conform to
this these days, so just detecting thatt here is a delivery-status
header, then parsing the second body-part for "^Action: failed" oughta
work.  The message/delivery-status content-type MUST be 7 bit, so you
can be relatively certain it's not going to be anything but ASCII.
(AFAIK, there are no machines left on the Internet that leak 7-bit
charsets that are not ASCII).

A

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 17 Jul 2008 08:46:53 -0400

On Wed, Jul 16, 2008 at 11:17:22PM -0300, Marc G. Fournier wrote:

> that ppl are having problems 

I believe I mentioned this before.  If you really want me to plough
through the archives, I will.

> the # of messages going through ...

Sounds like a good idea to me.

A

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 17 Jul 2008 12:41:58 -0300 (ADT)

On Thu, 17 Jul 2008, Andrew Sullivan wrote:

>> I can change the quarantining to be >4 if ppl want, which should 
>> greatly reduce the # of messages going through ...
>

'k, I'll put it down to 4 tonight, and increase the frequency I check for 
false-positives accordingly, to make sure that I don't get a big spike as 
a result ...

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 17 Jul 2008 11:54:28 -0400

> false-positives accordingly, to make sure that I don't get a big spike as 

Hmm, I think keeping it at 5 is a good idea, seeing how 2 of those were
being substracted by the AWL and you've just cleared that.

Maybe what we need is a way to remove a particular domain from the AWL
instead?

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

---

From: "Greg Sabino Mullane" <[email-masked]>
Date: Thu, 17 Jul 2008 15:54:41 -0000

> few, I notice that they are scoring >4, but just below 5:
...
> the # of messages going through ...

I think that would be a good start, but there are definitely some other problems.

> MIME_HTML_ONLY=1.672, RAZOR2_CHECK=0.5, RCVD_IN_BL_SPAMCOP_NET=2.188,

A score of 0.001 for HTML_MESSAGE? Might as well not have the check at all. Same
with things like DIGEST_MULTIPLE. I think we need more checks, and much higher
scores for many of them.

I grabbed a few random messages from the bugs list last night. Most interesting
was that some had no X-Spam-Status headers at all - does this mean they slipped
through the spam filtering entirely? Here's one of them:

===
Date: Wed, 16 Jul 2008 18:40:43 GMT
To: [email-masked]
Subject: BUG #4310: PkMERMInZQ
From: "make money on line" <[email-masked]>

The following bug has been logged online:

Bug reference:      4310
Logged by:          make money on line
Email address:      [email-masked]
PostgreSQL version: IUrjkiPgQkQXNgo
Operating system:   aJzBuaSGetA
Description:        PkMERMInZQ
Details:

<a href=" http://www.divinecaroline.com/public/user/profile?user_id=83997
">work at home jobs 101waystoincome.com</a>

====

Did it get whitelisted because it came from our form? I still think we
should scan it  - the "make money on line" is a dead giveaway, and
when I ran a local spamassassin on it, I even found:

 2.0 URIBL_BLACK            Contains an URL listed in the URIBL blacklist
                            [URIs: 101waystoincome.com]

Here's another one from last night that did have a spam header. I apologize
for how long this post is getting, but I'm trying to provide some hard data:
===

Date: Thu, 17 Jul 2008 02:59:41 +0100 (GMT+01:00)
From: "Tajana for(Mrs. Lucy Berg)" <[email-masked]>
Reply-To: [email-masked]
Subject: REMINDER NOTIFICATION
To: undisclosed-recipients:;

REMINDER NOTIFICATION

This email is to notify you that your Email
Address attached to a
Ticket Number(140408) has won an Award Sum of
($500,000.00)(Five
Hundred Thousand Dollars)In an Email Sweepstakes
program held in
The Netherlands these year 2008.Please contact the
claim officer
through the below given contact information.

MR.HANSON
CHRIS.
TEL. +31-643-502-787.
FAX: +31-847-290-539.
E-mail:cpinans@aol.
nl

WINNING INFORMATIONS
Ref Number:Nl50286
lucky Numbers:
07,12,24,36,45
Batch Number:EU-175508
Ticket Number:360208

Please
forward the above stated winning information to your Claim
Agent and do
include the following,

Your Name:
Telephone Number:

Congratulations!!!

Yours Sincerely,
Mrs. Lucy Berg.
Public Relation
Officer.

===

The only spam trigger found by postgresql.org was:

X-Spam-Status: No, hits=1.806 tagged_above=0 required=5
 tests=SUBJ_ALL_CAPS=1.806

There are numerous triggers in the body of the email that should
have boosted the score up. Personally, I'd also like to see
SUBJ_ALL_CAPS raised to 3 or 4.

So, to reiterate, I'd like to request the following:

1) Spam filtering is run on all messages
2) The default to reject is lowered to at least 4
3) The values get raised significantly for some tests
4) More SA tests get added (are we at least cronning sa-update?)
5) If 3 and 4 are too much trouble to maintain, outsource the
filtering to someone who does have the time, or who specializes
in it (economies of scale)

I did #5 myself years ago, after getting tired of updating SA rules,
messing with DNS lookups, blacklists, etc. and now just let
maillaunder.com handle it all.

- --
Greg Sabino Mullane [email-masked]
PGP Key: 0x14964AC8 200807171149
http://biglumber.com/x/web?pk=2529DF6AB8F79407E94445B4BC9B906714964AC8

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 17 Jul 2008 15:11:18 -0300 (ADT)

> Maybe what we need is a way to remove a particular domain from the AWL

Actually, I was thinking of disabling the AWL feature altogether ... it 
might make a few extra false-positives to need to release, but I doubt it 
will make a huge difference ...

Maybe disable AWL and leave score threshold at 5 ... ?  I can adjust 
after, based on # of false positives ... if false positives don't go up, 
then without AWL causes no adverse problems ... then I can look at 
adjusting score threshold down while watching for false-positives ...

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 17 Jul 2008 14:15:56 -0400

> will make a huge difference ...

I think that's a bad idea because it'll delay all email.

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 17 Jul 2008 16:58:27 -0300 (ADT)

> I think that's a bad idea because it'll delay all email.

Wait, i think we are talking about two different things here ... I'm not 
talking about Greylisting AWL, I'm talking abotu the AWL that modifies the 
spamscore ... totally different databases ... I'm not touching 
GreyListing, only spamscore ...

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 17 Jul 2008 16:41:22 -0400

> talking about Greylisting AWL, I'm talking abotu the AWL that modifies 
> the spamscore ... totally different databases ... I'm not touching  

Ah, sorry for the noise then -- I see no problem with that.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 17 Jul 2008 17:48:22 -0300

- --On Thursday, July 17, 2008 15:54:41 +0000 Greg Sabino Mullane 
<[email-masked]> wrote:

> I grabbed a few random messages from the bugs list last night. Most
> interesting was that some had no X-Spam-Status headers at all - does this
> mean they slipped through the spam filtering entirely? Here's one of them:
>
> ===
> Return-Path: <[email-masked]>
> Delivered-To: [email-masked]
> Received: from localhost (unknown [200.46.204.183])
>         by postgresql.org (Postfix) with ESMTP id C3148650275
>         for <[email-masked]>; Wed, 16 Jul 2008
> 15:40:45 -0300 (ADT) Received: from postgresql.org ([200.46.204.86])
>  by localhost (mx1.hub.org [200.46.204.183]) (amavisd-maia, port 10024)
>  with ESMTP id 48600-04-3 for <[email-masked]>;
>  Wed, 16 Jul 2008 15:40:43 -0300 (ADT)
> X-Greylist: from auto-whitelisted by SQLgrey-1.7.6
> Received: from wwwmaster.postgresql.org (wwwmaster.postgresql.org
> [217.196.146.204])         by postgresql.org (Postfix) with ESMTP id
> AB1D565026D
>         for <[email-masked]>; Wed, 16 Jul 2008 15:40:44 -0300 (ADT)
> Received: from wwwmaster.postgresql.org (wwwmaster.postgresql.org
> [217.196.146.204])         by wwwmaster.postgresql.org (8.13.8/8.13.8) with
> ESMTP id m6GIehuA007983         for <[email-masked]>; Wed, 16 Jul
> 2008 18:40:43 GMT         (envelope-from [email-masked])
> Received: (from www@localhost)
>         by wwwmaster.postgresql.org (8.13.8/8.13.8/Submit) id m6GIehIP007982;
>         Wed, 16 Jul 2008 18:40:43 GMT
>         (envelope-from www)
> Message-Id: <[email-masked]>
> Content-Type: text/plain; charset=utf-8
> X-Virus-Scanned: Maia Mailguard 1.0.1

The X-Virus-Scanned indicates it went through Amavis/Maia, not sure why it 
would have bypassed the spam checker though ... let me check into it to make 
sure, as I thought both "outbound" and "inbound" was scanned, but maybe just 
for virus, with the expectation that if you are using a spam scanner on 
incoming, you aren't sending *out* spam ...

- --