Subject: Mail setup broken (still/again?)

From: Magnus Hagander <[email-masked]>
Date: Tue, 16 Oct 2007 10:52:09 +0200

I just sent an email to -advocacy where I spelled the list name wrong. I
did not get a bounce. Why? Because the mail system is (again or still)
broken in it's config. What happens is:

1) Tries to deliver to svr1.postgresql.org. This machine response that the
user is unknown, *but does so with a 450 error code indicating that this
is a temporary error*. This is of course wrong, it should be responding
with 550.

1b) Also, that machine is supposedly named "postgresql.org" for some reason
that I don't really understand.  But the MX rcord still points to svr1,
which is an alias. (I don't say this should be fixed, because I don't see
the point in not calling the machine svr1, but that's probably just because
I've forgotten the reason for it :-P But either way, it should be
consistent)

2) Since it got a 450, it tries a secondary MX, in this case mx3.hub.org.
Now:

2a) Why do we even bother with secondary MXes since all they do is relay
back to svr1 anyway? It doesn't actualliy *help* us anything that i can
see, it only makes the configuration more complex.

2b) If we do relay, then the secondary MX must *also* know the list of
users, so it can give a proper bounce. What happens now is that my email is
queued up on mx3.hub.org and will stay there as it retries over and over
for a couple of days, when the bounce will be generated on that system.

2c) mx3 is then *graylisted* by svr1. A backup MX must *NOT* be graylisted
by the primary machine. I know I have mentioned this several times before
wrt other machines.

3) At the risk of soundling like a real broken record again, we really
need *some kind of basic documentation* of this system.

Our mail infrastructure is critical to the project. We simply cannot afford
having one that looks like this.

My suggestion is the same as before - de-couple it from the hub.org
infrastructure, thus making things a lot simpler and less likely to break.
If this is for some reason not acceptable, can we please, and *urgently*,
have these issues listed above fixed?

//Magnus

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 16 Oct 2007 10:07:50 -0400

> is a temporary error*. This is of course wrong, it should be responding

This does appear to be an error.

> which is an alias. (I don't say this should be fixed, because I don't see

Nit: it's not an alias; if it were (i.e. a CNAME or probably a
DNAME), it would be an error, because MX records can't point to
CNAMEs.  It's just another name for the same address, which is
perfectly acceptable.  It does seem a little baroque.

> see, it only makes the configuration more complex.

If svr1 goes down, the secondaries queue up the mail.  This is a
robust answer, and a good one, because it prevents mail from getting
queued up all over the Internet.

> users, so it can give a proper bounce. 

No.  The classic way of relaying through a secondary that you
normally don't use except in emergency is just to relay everything
that arrives on the secondary.  When it arrives at the restrictive
server, that server bounces.  You get into trouble from the soft
error you're encountering, because if the user isn't in the map and
that server is the final destination, then you really do need to
bounce and be done.

> by the primary machine. I know I have mentioned this several times before

Absolutely.  

> need *some kind of basic documentation* of this system.

+1

> infrastructure, thus making things a lot simpler and less likely to break.

I don't think removing the secondaries makes things less likely to
break; mail servers are easily overwhelmed and can produce soft
errors all the time.  Having store-and-forward secondaries is an
extremely good idea, and I'd hate to see that feature abandones. 
Your other remarks are right on the money, though.

A

-- 
Andrew Sullivan  | [email-masked]
A certain description of men are for getting out of debt, yet are
against all taxes for raising money to pay it off.
		--Alexander Hamilton

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 16 Oct 2007 17:02:48 +0200

> On Tue, Oct 16, 2007 at 10:52:09AM +0200, Magnus Hagander wrote:

<snip>

> queued up all over the Internet.

Sure, but does it help us in any way at all? Why do we care where the mail
is queued up, reall?

> that server is the final destination, then you really do need to

If we reject it on the secondary MX, we'll be creating a whole bunch of
bounces for invalid addresses that spammers sent to. If our secondary MX
can just drop them, that never happens since they get a reject at the SMTP
protocol level.

> Your other remarks are right on the money, though.

Well, we clearly don't entirely agree. But if we *do* want
store-and-forward secondaries, I would propose we use dedicated ones so we
can do things like push our own userlists over there.

//Magnus

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 16 Oct 2007 11:22:46 -0400

> Sure, but does it help us in any way at all? Why do we care where the mail

We can't control the policies on all those servers, and some of them
may not queue as long as we like.  Also, it's polite to have more
than one mail server, and not force others to queue mail when you
have an outage.  This is part of the reason one has more than one MX
possible, after all.

> can just drop them, that never happens since they get a reject at the SMTP

You mustn't _ever_ "just drop them".  Yes, I know people are doing
that instead of bouncing, but it's wrong, bad, evil, and completely
in contradiction of the totally plain MUSTs in the relevant RFCs.  

I think you meant refuse, though, which is a different matter.  It's
not actually hard to rsync the user map among the various servers
using postfix (I do it myself), so that seems to me to be an
alternative, yes.  And that can be done with multiple user lists.

There is another thing we could do, BTW, to try to reduce the
spam-induced bounces, and still have multiple servers in place.  What
you do is add an MX with priority 0 that always gives a soft error. 
Most spambots won't try the next MX, so your "real" MX (with, say,
priority 1) doesn't get the spam attempt.

A
-- 
Andrew Sullivan  | [email-masked]
Never get involved in litigation. Your hair will fall out, your bones 
will turn to sand.  And it will still be going on.
		--Tom Waits

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 16 Oct 2007 17:31:33 +0200

> in contradiction of the totally plain MUSTs in the relevant RFCs.  

I meant reject, not drop. But it's better for us to reject them at the SMTP
level than it is to generate our own bounce.

> alternative, yes.  And that can be done with multiple user lists.

As do I, so yeah, it's fairly simple. But if you have to interface with an
external system (in this case, hub.org) that makes things a lot more
complex quickly.

> priority 1) doesn't get the spam attempt.

Hmm. Interesting idea :) But I'm not sure how big of a problem that part
really is.

//Magnus

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 16 Oct 2007 11:52:02 -0400

> level than it is to generate our own bounce.

Yeah, that can be more efficient.

> external system (in this case, hub.org) that makes things a lot more

Well, not a _lot_ more complex, I think.  We just need maps special
to each domain, and while I've never done it before, the examples
seem to me to suggest it's straightforward.  I think the trick would
be to design the postgresql.org site the same way as the hub site,
and then leave parts out on postgresql.org, rather than the other way
around.  It _is_ more complex, but with careful choices of
include-file names, it oughta be managable?  (Especially if we have
some documentation ;-)

> Hmm. Interesting idea :) But I'm not sure how big of a problem that part

We get a remarkable number of spam on the lists I moderate, at least,
and none of those are -general.

A

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 16 Oct 2007 13:03:02 -0300

> This does appear to be an error.

Changed ... postfix's default is 550, but we had:

unknown_local_recipient_reject_code = 450

Funny thing is, looking on google, this isn't that particularly unusual:

"On my Fedora Core box the following settings and comments were in
main.cf

# The default setting is 550 (reject mail) but it is safer to start
# with 450 (try again later) until you are certain that your
# local_recipient_maps settings are OK.
#
#unknown_local_recipient_reject_code = 550
unknown_local_recipient_reject_code = 450"
	- <http://www.webservertalk.com/message1926259.html>

I suspect that that is what I did when I set things up originally, went with 
the 'safter to start with 450', but didn't go back and change it to 550 ...

> perfectly acceptable.  It does seem a little baroque.

Fixed, now MX 0 == mail.postgresql.org, also an A record ...

>> by the primary machine. I know I have mentioned this several times before

Fixed ... I had missed the mx3 IP in the mynetworks file on postgresql.org ... 
the other 3 mx servers should never have been affected, only the offsite one ...

- ----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . [email-masked]                              MSN . [email-masked]
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 16 Oct 2007 18:46:28 +0200

Marc G. Fournier wrote:
> 
> 
> --On Tuesday, October 16, 2007 10:07:50 -0400 Andrew Sullivan 

Great.

> Funny thing is, looking on google, this isn't that particularly unusual:
<snip>
Yeah, IIRC there was a recommendation when this feature was added to
postfix that you do this. And it is a good recommendation - you just
have to remember to switch it back to 550 when you've tested :-) (yes,
I've forgotten that on my servers as well a couple of times)

> Fixed, now MX 0 == mail.postgresql.org, also an A record ...

Good.

> the other 3 mx servers should never have been affected, only the offsite one ...

Ah, that's why it reappeared.

Thanks for the quick fixes!

//Magnus

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 16 Oct 2007 14:00:05 -0300

> Thanks for the quick fixes!

Thanks for pointing them out, specially the 450 -> 550 change ... should reduce 
some of the repeat hits:

4902 <[email-masked]>:
 933 <[email-masked]>:
 876 <[email-masked]>:
 863 <[email-masked]>:
 848 <[email-masked]>:
 845 <[email-masked]>:
 805 <[email-masked]>:
 801 <[email-masked]>:
 793 <[email-masked]>:
 788 <[email-masked]>:

And that was just today :)

- ----

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 16 Oct 2007 19:01:26 +0200

Marc G. Fournier wrote:
> 
> 
> --On Tuesday, October 16, 2007 18:46:28 +0200 Magnus Hagander 

What? Several of those aliases *should* exist, no? ;-)

I wonder how the hell they come up with those... I mean, what's the
percentage that they exist at all. I can understand those that fake
bill@ and joe@ and such addresses, but this...

//Magnus

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 16 Oct 2007 14:08:14 -0300

- --On Tuesday, October 16, 2007 19:01:26 +0200 Magnus Hagander 
<[email-masked]> wrote:

>> Thanks for pointing them out, specially the 450 -> 550 change ... should
>> reduce  some of the repeat hits:
>>

I don't know, but the first one with 4902 attempts comes from mx3.cc.teu.ac.jp 
... so its not even coming from random 'English' sites :)

We've had 48535 distinct names attempted 647009 times so far today ... and two 
of them were for [email-masked], and none for joe :)

- ----

---

From: "Joshua D. Drake" <[email-masked]>
Date: Tue, 16 Oct 2007 10:13:30 -0700

On Tue, 16 Oct 2007 14:08:14 -0300
"Marc G. Fournier" <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> 
> >> Thanks for pointing them out, specially the 450 -> 550 change ...
> >> should reduce  some of the repeat hits:
> >>
> I don't know, but the first one with 4902 attempts comes from

So why haven't we just blocked mx3.cc.teu.ac.jp at the firewall?

Joshua D. Drake

> 
> We've had 48535 distinct names attempted 647009 times so far
> today ... and two of them were for [email-masked], and none for
> joe :)
> 
> 
> - ----
> Marc G. Fournier           Hub.Org Networking Services
> (http://www.hub.org) Email .
> [email-masked]                              MSN . [email-masked]
> -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.4 (FreeBSD)
> 
> iD8DBQFHFO/+4QvfyHIvDvMRAnv4AKDVrabAldtlxsT3mesy2mBte2rQqwCeO7rV
> P9wUG7piSThK69RXma4XPQ8=
> =PUfH
> -----END PGP SIGNATURE-----
> 
> 
> ---------------------------(end of
> broadcast)--------------------------- TIP 7: You can help support the
> PostgreSQL project by donating at
> 
>                 http://www.postgresql.org/about/donate
> 

-- 

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
			UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

---

From: Andrew Sullivan <[email-masked]>
Date: Tue, 16 Oct 2007 13:17:57 -0400

> So why haven't we just blocked mx3.cc.teu.ac.jp at the firewall?

Just because an MX happens to initiate spam doesn't mean it's not
also a legitimate mail source.  On a private domain, I would have no
difficulty blocking it, but on a community service like this, it
seems dodgier to me,

A

-- 
Andrew Sullivan  | [email-masked]
This work was visionary and imaginative, and goes to show that visionary
and imaginative work need not end up well. 
		--Dennis Ritchie

---

From: Tom Lane <[email-masked]>
Date: Tue, 16 Oct 2007 13:19:35 -0400

>> bill@ and joe@ and such addresses, but this...

I've seen a fair amount of spam that has forged return addresses that
look like those.  I'm thinking that spammer A's name-generator has
fooled spammer B's address-harvester ...

			regards, tom lane

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 16 Oct 2007 14:23:50 -0300

> So why haven't we just blocked mx3.cc.teu.ac.jp at the firewall?

Several reasons, but foremost ... until Magnus pointed out the need to change 
450 -> 550, nobody knew they were hitting us ... but, I just checked the log 
file, and that is the only address that that host has tried to get through, so 
would be a bit overkill to block one host for retrying one address repeatedly 
because we didn't tell them it was a permanent failure vs temporary one, no? :)

- ----