Subject: Maia Mailgard down?

From: Josh Berkus <[email-masked]>
Date: Fri, 28 Jul 2006 16:12:18 -0700

Marc,

[email-masked] is getting a spam every 3-6 minutes.  Is our spam 
filter down?

--Josh

---

From: "Marc G. Fournier" <[email-masked]>
Date: Fri, 28 Jul 2006 22:20:12 -0300 (ADT)

On Fri, 28 Jul 2006, Josh Berkus wrote:

> Marc,
>
> [email-masked] is getting a spam every 3-6 minutes.  Is our spam filter 
> down?

Email goes down when maia goes down, so it is definitely up and running 
... but, you need to retrain yourself, as due to a problem with the 
database, we've had to reload the system, so you need to relogin, set your 
settings and re-train the bayes component ... apologies, I thought I had 
sent out a notice to this list about it, but in the rush, appear to have 
forgotten :(

note for *anyone* that has a mailbox @postgresql.org, you will need to 
login to http://webmail.postgresql.org/maia to set this up, including all 
of the various regional press contacts ... Josh, not sure where I would 
send to get ahold of all of these ...

Also, once question ... does *anyone* know who [email-masked] is??? 
If nobody does, I'll delete it, since its got a major full mailbox, and 
doesn't look like any of it has ever been read ...

----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . [email-masked]                              MSN . [email-masked]
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664

---

From: Josh Berkus <[email-masked]>
Date: Mon, 31 Jul 2006 12:22:12 -0700

Marc,

> login to http://webmail.postgresql.org/maia to set this up, including
> all of the various regional press contacts ... Josh, not sure where I
> would send to get ahold of all of these ...

Isn't there some what (via Horde) that you can sent out an e-mail to all 
account-holders on @postgresql.org?

And no, I don't know who david@ belongs to; feel free to delete it.

-- 
--Josh

Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: Josh Berkus <[email-masked]>
Date: Sat, 5 Aug 2006 11:52:03 -0700

Folks,

Is there any way we can return to whatever we were using before Maia mailgard?   
Maia is patently NOT working ... it's fundamentally broken in some way, and 
isn't blocking any spam.  The interface for Baysian filtering is also awkward 
and darned near unusable.  In a couple months we're going to be trying to do 
the PostgreSQL 8.2 release which means that we need the @postgresql.org 
e-mail addresses working ... and right now they're all receiving a flood of 
unfiltered spam, making them unusable.

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: "Marc G. Fournier" <[email-masked]>
Date: Sun, 6 Aug 2006 12:11:48 -0300 (ADT)

On Sat, 5 Aug 2006, Josh Berkus wrote:

> Folks,
>
> Is there any way we can return to whatever we were using before Maia 
> mailgard? Maia is patently NOT working ... it's fundamentally broken in 
> some way, and isn't blocking any spam.  The interface for Baysian 
> filtering is also awkward and darned near unusable.  In a couple months 
> we're going to be trying to do the PostgreSQL 8.2 release which means 
> that we need the @postgresql.org e-mail addresses working ... and right 
> now they're all receiving a flood of unfiltered spam, making them 
> unusable.

Actually, all MAIA is is a front-end over what we were using before 
(amavisd + spamassassin) to allow individual mailboxes to configure 
themselves, instead of one big site-wide setting ... I'm getting a 
*trickle* of spam right now, but I've spent the past week or so going 
through and training the Bayesian database ... in fact, under the old 
system, those 46 messages that are currently quarantined in your cache 
would have been delivered, instead of quarantined ... in the 
@postgresql.org one that I monitor, there are >7k messages currently in 
spam-quarantine that normally would have been passed through to the system 
...

Not sure what you are finding awkward about the interface, but if you go 
in and just turn on all the various checks, but *disable* the 
quarantining, so that it just label's and passing everything over to you 
to filter, you will have exactly the same thing in place that was there 
before we added the interface for per-user settings ...

Please note ... at *no time* did we ever reject spam for any mailbox, 
except those for the mailing lists themselves, which majordomo is setup to 
do ... so if you are seeing more now then before, its because we are being 
hit with more now that are passing through spamassassin ...

Note that unlike the ~60 messages in the cache for your mailbox right now 
(12 unconfirmed non-spam, 46 unconfirmed spam), the stuff that I go 
through is right now sitting at ~10k, and I generally spend about a half 
hour in the evening going through it ... I generally care about 
unconfirmed-spam between 5 and 10, ignoring most above that ... and 
unconfirmed non-spam between 0 and 5 ...

----

---

From: "Marc G. Fournier" <[email-masked]>
Date: Sun, 6 Aug 2006 13:26:31 -0300 (ADT)

Just to give some followup numbers, between both servers that are 
currently doing Spam/Virus/MAIA, here is what it has seen for josh@post 
since midnight:

# grep josh@post /var/log/amavis | awk '{print $8}' | sort | uniq -c
    2 CLEAN,
    4 INFECTED
   20 SPAM,

# grep josh@post /var/log/amavis | awk '{print $8}' | sort | uniq -c
    4 CLEAN,
    4 INFECTED
    4 SPAM,

So, not 100% certain where josh is seeing this "flood" ...

... but, just in case, I've raised the timeout on both the razor2 and 
pyzor checks from 10 sec to 60 sec ... see if that helps any ...

On Sun, 6 Aug 2006, Marc G. Fournier wrote:

>> some way, and isn't blocking any spam.  The interface for Baysian filtering 
>> is also awkward and darned near unusable.  In a couple months we're going 
>> to be trying to do the PostgreSQL 8.2 release which means that we need the 
>> @postgresql.org e-mail addresses working ... and right now they're all 
>> receiving a flood of unfiltered spam, making them unusable.
>
> Actually, all MAIA is is a front-end over what we were using before (amavisd 
> + spamassassin) to allow individual mailboxes to configure themselves, 
> instead of one big site-wide setting ... I'm getting a *trickle* of spam 
> right now, but I've spent the past week or so going through and training the 
> Bayesian database ... in fact, under the old system, those 46 messages that 
> are currently quarantined in your cache would have been delivered, instead of 
> quarantined ... in the @postgresql.org one that I monitor, there are >7k 
> messages currently in spam-quarantine that normally would have been passed 
> through to the system ...
>
> Not sure what you are finding awkward about the interface, but if you go in 
> and just turn on all the various checks, but *disable* the quarantining, so 
> that it just label's and passing everything over to you to filter, you will 
> have exactly the same thing in place that was there before we added the 
> interface for per-user settings ...
>
> Please note ... at *no time* did we ever reject spam for any mailbox, except 
> those for the mailing lists themselves, which majordomo is setup to do ... so 
> if you are seeing more now then before, its because we are being hit with 
> more now that are passing through spamassassin ...
>
> Note that unlike the ~60 messages in the cache for your mailbox right now (12 
> unconfirmed non-spam, 46 unconfirmed spam), the stuff that I go through is 
> right now sitting at ~10k, and I generally spend about a half hour in the 
> evening going through it ... I generally care about unconfirmed-spam between 
> 5 and 10, ignoring most above that ... and unconfirmed non-spam between 0 and 
> 5 ...
>
> ----

----

---

From: Josh Berkus <[email-masked]>
Date: Mon, 7 Aug 2006 08:44:19 -0700

Marc,

> hit with more now that are passing through spamassassin ...

Aha, ok, I see what happened.  Some change you made disabled the spam filters 
I had set up oh Horde.  I need to re-enable them.

I need to find some way to set up a "default" spam configuration with all 
regional contacts to make their accounts usable, which I guess would require 
both settings in Maia and Horde.    Suggestions?

> before we added the interface for per-user settings ...

Got it.  The interface isn't quite intuitive and the help file is unhelpful.   
What I found awkward ... downright unusable, in fact ... is the quarantine 
interface.   Bad, bad webform.  The fields are too small to display useful 
data, it's not UTF-8, and there's no "check all".   Of course, if you've been 
using it, you know this.

What's Maia written in?

Question: does putting stuff in a "spam" folder via IMAP still work?

-- 
Josh Berkus
PostgreSQL @ Sun
San Francisco

---

From: "Marc G. Fournier" <[email-masked]>
Date: Mon, 7 Aug 2006 23:50:55 -0300 (ADT)

> both settings in Maia and Horde.    Suggestions?

To be honest, since using MAIA, I think I've removed my own 'spam filters' 
,,, with quarantine set for all messages scoring >5, those messages never 
reach my filters, but stay in MAIA until I go in an report as spam, or 
non-spam ...

So, in theory, it should just be a matter of enabling the appropriate 
settings in MAIA for each of the regional contacts, and getting them to 
login periodically to 'train' the system ...

If you *really* wanted the responsibility for it, I could link all of the 
regional accounts to yours, which means when you login to 'train' the 
system, you'd be training all of them ... your call on that one :)

> Got it.  The interface isn't quite intuitive and the help file is 
> unhelpful. What I found awkward ... downright unusable, in fact ... is 
> the quarantine interface.  Bad, bad webform.  The fields are too small 
> to display useful data, it's not UTF-8, and there's no "check all".  Of 
> course, if you've been using it, you know this.

'k, can't help with alot of the above ... but there is definitely a 'check 
all' ... look at the very top line, before the first email is listed ... 
each column as a 'checkbox', that if you click it, sets all boxes in that 
column to checked ...

> What's Maia written in?

PHP ...

> Question: does putting stuff in a "spam" folder via IMAP still work?

Not if you are quarantining in MAIA, since it won't pass it to the 
mailbox, but holds it in cache until you release it ... and its expected 
that all you will release is, of course, non-SPAM ... the rest is 
reported/trained and deleted from the web interface ...

----

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 8 Aug 2006 10:35:01 -0400 (EDT)

Because all my [email-masked] email is forwarded to
[email-masked], I don't have any way of training the spam filter.  Is
there a way I can use a global one?

---------------------------------------------------------------------------

Marc G. Fournier wrote:
> ---------------------------(end of broadcast)---------------------------
> TIP 6: explain analyze is your friend

-- 
  Bruce Momjian   [email-masked]
  EnterpriseDB    http://www.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 08 Aug 2006 08:41:50 -0700

> there a way I can use a global one?

I don't think that would work (except for maybe greylisting).

Sincerely,

Joshua D. Drake
-- 

    === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564 || 24x7/Emergency: +1.800.492.2240
    Providing the most comprehensive  PostgreSQL solutions since 1997
              http://www.commandprompt.com/

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 8 Aug 2006 21:58:03 -0300 (ADT)

> there a way I can use a global one?

MAIA is *before* your mailbox, so, before the filtering ... the mail 
system essentially goes:

smtpd -> greylist -> maia -> mailbox -> local store
                                             or
                                      -> forward

so, you can still train the system, just login, go to Settings and set 
your appropriate settings so that it Quarantines stuff, and then train 
away ...

The nice thing about the system is that it doesn't just train/report to 
the Bayes database, but it reports it to Razor2, Spamcop and Pyzor as 
well, but, it does it in the background ... the interface just allows you 
to confirm an email as spam, and a cron job then runs to actually do the 
appropriate processing of the email(s) ...

any non-spam that you 'release' is, of course, released instantly back 
into the system ...

there are a few things planned for v1.0.2 that will be nice ... like the 
ability to just auto-discard anything that scores over X (if someone sends 
me an email taht scores over, say, 20, chances are it isn't something I 
want to read or ever really see) ...

----

---

From: Bruce Momjian <[email-masked]>
Date: Tue, 8 Aug 2006 22:59:06 -0400 (EDT)

OK, I set it up to forward, but not discard from my mailbox, and I will
use that for training.  Good idea.  Thanks.