Subject: what is up with the PG mailing lists?

From: Tom Lane <[email-masked]>
Date: Wed, 31 Oct 2007 23:26:35 -0400

I've seen enormously erratic performance from the mailing lists all day
today --- delivery time for individual messages is up to several hours,
but completely inconsistent.  Is it just me, or are other people seeing
the same?

			regards, tom lane

---

From: Bruce Momjian <[email-masked]>
Date: Thu, 1 Nov 2007 00:02:20 -0400 (EDT)

> but completely inconsistent.  Is it just me, or are other people seeing

I see the same.  My posting to core took +2 hours to arrive and there
are only a few subscribers.

-- 
  Bruce Momjian  <[email-masked]>        http://momjian.us
  EnterpriseDB                             http://postgres.enterprisedb.com

  + If your life is a hard drive, Christ can be your backup. +

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 01:04:45 -0300

I haven't noticed, but haven't been watchign for it ... can you send me full 
headers of one such affected?  This  one here seemed to be 'within minutes' :(

- --On Wednesday, October 31, 2007 23:26:35 -0400 Tom Lane <[email-masked]> 
wrote:

> ---------------------------(end of broadcast)---------------------------
> TIP 9: In versions below 8.0, the planner will ignore your desire to
>        choose an index scan if your joining column's datatypes do not
>        match

- ----
Marc G. Fournier           Hub.Org Networking Services (http://www.hub.org)
Email . [email-masked]                              MSN . [email-masked]
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 01:14:36 -0300

> are only a few subscribers.

'k, what is useful is that *when* you see this, you send me the full  headers 
so that I can look at, and  hopefully figur eout, where the delay is ... if its 
something on our side, knowing when you see the problem, vs hours later really 
helps ...

I can't fidn something I don't know is broke, and 2+ hours to deliver to -core 
is definitely something being broke ...

Now that I know, I also think I know what the problem was, and have fixed it 
... was doing some cleaning up of the virus/spam checker, and took out one too 
many IPs from DNS, so it was running at half-capacity ... there might still be 
some backlog to go through, so give it a few hours, but if you are still seeing 
issues tomorrow, send me full headers of one that is having the problem and 
I'll look into it further ...

- ----

---

From: Tom Lane <[email-masked]>
Date: Thu, 01 Nov 2007 00:29:20 -0400

> headers of one such affected?  This  one here seemed to be 'within minutes' :(

Here's headers from a message I sent to -core today, which had about an
hour turnaround ...

[ btw, it's pretty funny that -core messages get List-Unsubscribe: and
suchlike decorations ... ya think any of us are about to unsubscribe? ]

			regards, tom lane

------- Forwarded Message

To: Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <[email-masked]>
cc: "[email-masked]" <[email-masked]>
Subject: Re: [CORE] [Fwd: [pgsql-packagers] 8.3 beta2, I am building Solaris binaries] 
Comments: In-reply-to Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= <[email-masked]>
	message dated "Wed, 31 Oct 2007 11:54:41 -0700"
Date: Wed, 31 Oct 2007 17:33:30 -0400
From: Tom Lane <[email-masked]>
Sender: [email-masked]

------- End of Forwarded Message

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 01:31:47 -0300

Ya, Bruce sent me one, and I think its the problem that I just fixed, where the 
scanner was running half-throttle ... let me know if you see any of these 
tomorrow, but I think we should be good now ...

Thanks ...

- --On Thursday, November 01, 2007 00:29:20 -0400 Tom Lane <[email-masked]> 
wrote:

> Received: from mx1.hub.org (mx1.hub.org [200.46.208.251])
> 	by sss.pgh.pa.us (8.14.1/8.14.1) with ESMTP id l9VMaxDa022176
> 	for <[email-masked]>; Wed, 31 Oct 2007 18:37:01 -0400 (EDT)
> Received: from postgresql.org (postgresql.org [200.46.204.71])
> 	by mx1.hub.org (Postfix) with ESMTP id E2D04607491
> 	for <[email-masked]>; Wed, 31 Oct 2007 19:36:59 -0300 (ADT)
> Received: from localhost (unknown [200.46.204.184])
> 	by postgresql.org (Postfix) with ESMTP id 00DCE9FCEDF
> 	for <[email-masked]>; Wed, 31 Oct 2007 19:35:50
> -0300 (ADT) Received: from postgresql.org ([200.46.204.71])
>  by localhost (mx1.hub.org [200.46.204.184]) (amavisd-maia, port 10024)
>  with ESMTP id 11420-01-10 for <[email-masked]>;
>  Wed, 31 Oct 2007 19:35:13 -0300 (ADT)
> X-Greylist: from auto-whitelisted by SQLgrey-1.7.5
> Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
> 	by postgresql.org (Postfix) with ESMTP id 34FA79FCF4F
> 	for <[email-masked]>; Wed, 31 Oct 2007 18:33:32 -0300 (ADT)
> Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
> 	by sss.pgh.pa.us (8.14.1/8.14.1) with ESMTP id l9VLXUxR021063;
> 	Wed, 31 Oct 2007 17:33:30 -0400 (EDT)
> Subject: Re: [CORE] [Fwd: [pgsql-packagers] 8.3 beta2, I am building Solaris
> binaries]  In-reply-to: <[email-masked]>
> References: <[email-masked]>
> Comments: In-reply-to Devrim =?ISO-8859-1?Q?G=DCND=DCZ?=
> <[email-masked]> 	message dated "Wed, 31 Oct 2007 11:54:41 -0700"
> Message-ID: <[email-masked]>
> X-Virus-Scanned: Maia Mailguard 1.0.1
> X-Mailing-List: pgsql-core
> List-Archive: <http://archives.postgresql.org/pgsql-core>
> List-Help: <mailto:[email-masked]?body=help>
> List-ID: <pgsql-core.postgresql.org>
> List-Owner: <mailto:[email-masked]>
> List-Post: <mailto:[email-masked]>
> List-Subscribe: <mailto:[email-masked]?body=sub%20pgsql-core>
> List-Unsubscribe: <mailto:[email-masked]?body=unsub%20pgsql-core>
> Precedence: bulk

- ----

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 08:12:17 +0100

> are only a few subscribers.

This is all fairly normal behavior for our lists :-(

The mails are usually stuck in a queue somewhere at hub.org. Sometimes
it's the antispam queue, somtimes it's just the slow network connection
to the panama server, sometimes somewhere else - I don't think it's been
fully diagnosed since it keeps happening quite often. You can usually
get a decent pointer as to where it's stuck by looking at the
Received-headers in the message.

//Magnus

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 09:59:49 +0100

> Here's headers from a message I sent to -core today, which had about an

That one is clearly sitting waiting to go through the hub.org antispam
for about an hour (between 18:33 ADT and 19:35 ADT).

Oh, and the message you sent with this info just took about over 4 hours
to get here, which happens fairly often. Headers from this one:

01:29:30 -0300 (ADT)

In this case, it took 4 hours to get from postgresql.org to mx2.hub.org,
whereas all other steps were quick. So something's broken in the mail
delivery apart from the antispam.

//Magnus

---

From: Dave Page <[email-masked]>
Date: Thu, 01 Nov 2007 09:16:07 +0000

> to get here, which happens fairly often. Headers from this one:

It sat in the mod queue until I approved it earlier.

/D

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 10:30:38 +0100

> It sat in the mod queue until I approved it earlier.

Ah, that may explain some of the delays I've been seeing. But only those
between those two hops (postgresql -> hub) I guess.

//Magnus

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 08:19:27 -0300

- --On Thursday, November 01, 2007 10:30:38 +0100 Magnus Hagander 
<[email-masked]> wrote:

>>>>> I haven't noticed, but haven't been watchign for it ... can you send me
>>>>> full  headers of one such affected?  This  one here seemed to be 'within
>>>>> minutes' :(

Probably alot of the problems you've been seeing have to do with moderator 
approval times for some posts ... Tom doesn't usually let me go very long with 
big delays before letting me know there "might be a problem" ...

- ----

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 07:36:50 -0400

> suchlike decorations ... ya think any of us are about to unsubscribe? ]

Those headers are described in RFC 2369
<ftp://ftp.rfc-editor.org/in-notes/rfc2369.txt>, which says that if
you have them, you SHOULD have all of them.  I expect the ML manager
just inserts them automatically for every list.

A

-- 
Andrew Sullivan  | [email-masked]
I remember when computers were frustrating because they *did* exactly what 
you told them to.  That actually seems sort of quaint now.
		--J.D. Baldwin

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 07:39:38 -0400

> big delays before letting me know there "might be a problem" ...

That could be -- I have been very busy lately, and having some
trouble keeping up with moderation.  Also, I stopped moderating
-advocacy while Josh has his current rules in place, because I didn't
feel comfortable moderating out a discussion I was part of.  So some
of this might just be moderation time.

A

-- 
Andrew Sullivan  | [email-masked]
The whole tendency of modern prose is away from concreteness.
		--George Orwell

---

From: Tom Lane <[email-masked]>
Date: Thu, 01 Nov 2007 08:13:28 -0400

> It sat in the mod queue until I approved it earlier.

Magnus, don't you get notices from the mail daemon when a post of yours
is held for moderator approval?  I do, so I know the difference between
"slow" and "no moderator handy" ...

(I think actually that this behavior isn't default, which strikes me as
a pretty dang poorly chosen default.)

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 13:16:02 +0100

> a pretty dang poorly chosen default.)

Yeah. For my own posts it's generally not moderation. It happens with
big attachments sometimes, and then I get a notice. So in the cases that
my own mails are delayed to the lists, it's one of the other problems.
Harder to tell about somebody else.

Would be kinda handy if majordomo would put that into the headers
somewhere - e.g. "X-Released-From-Moderation: <date>" or something like
that.

//Magnus

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 08:20:32 -0400

On Thu, Nov 01, 2007 at 07:36:50AM -0400, Andrew Sullivan wrote:

> you have them, you SHOULD have all of them.  

Actually, I just re-read it, and it doesn't say that.  It says that
if you have anything, you should have -help (and at the URL, you SHOULD
have complete information for everything else).  I'd still bet the ML
manager just does these automatically, and I think it'd be a bad idea
to make this list different than the others, since most MUAs can be
configured to suppress certain headers in the normal view.

A

-- 
Andrew Sullivan  | [email-masked]
Everything that happens in the world happens at some place.
		--Jane Jacobs

---

From: "Joshua D. Drake" <[email-masked]>
Date: Thu, 1 Nov 2007 08:09:59 -0700

On Thu, 01 Nov 2007 08:13:28 -0400
Tom Lane <[email-masked]> wrote:

> >> Oh, and the message you sent with this info just took about over 4
> >> hours to get here, which happens fairly often. Headers from this
> >> one:
> 
> Magnus, don't you get notices from the mail daemon when a post of
> yours is held for moderator approval?  I do, so I know the difference
> between "slow" and "no moderator handy" ...
> 
> (I think actually that this behavior isn't default, which strikes me
> as a pretty dang poorly chosen default.)

I find the moderator argument holding zero water. We have a lot of
moderators including myself (recently added) on a couple (if not all?)
lists.

Yesterday Devrim approved several messages that came through that I
didn't see until hours later. Not because I wasn't checking email
but because they never showed. Now, we know the current problem with
that, which Marc just fixed.

However the "mailing list" problem is a constant. Sometimes they work,
sometimes I don't get messages for hours. This is not the first time I
or others have brought up the mailing list issues.

It would be great if the actual sysadmin team had management ability on
the mail servers. I would actually argue that it should be a
requirement and that the fact that the sysadmin team doesn't is a real
problem.

Note we still don't have documentation on this stuff, even though the
request has come through well over a dozen times and been willingly
ignored.

Sincerely,

Joshua D. Drake

> 
> 			regards, tom lane
> 
> ---------------------------(end of
> broadcast)--------------------------- TIP 6: explain analyze is your
> friend
> 

-- 

      === The PostgreSQL Company: Command Prompt, Inc. ===
Sales/Support: +1.503.667.4564   24x7/Emergency: +1.800.492.2240
PostgreSQL solutions since 1997  http://www.commandprompt.com/
			UNIQUE NOT NULL
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate
PostgreSQL Replication: http://www.commandprompt.com/products/

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 11:23:33 -0400

> or others have brought up the mailing list issues.

There are indeed sometimes mailing list latency issues.  But I
caution everyone in being too glib about some of this:

1.	All the mail RFCs are totally clear that latency is to be
expected in the mail system.  Every time I hear complaints about mail
latency that entails delays of merely hours, I worry that people are
treating SMTP as though it's XMPP.  It ain't, and it's designed _not_
to be.

2.	There are plenty of individual relays involved here, and
saying "it's slow" without mail headers is no more helpful in
demystifying mail issues than are posts to -performance without
EXPLAIN ANALYSE output.

3.	We know that sometimes, moderation _does_ cost.  This is
especially true because we've already cranked up a lot of rules to
capture common abuses (spam, common admin keywords) that are far from
free to run on lists with the volume of mail the postgres lists get. 
So we're really paying for two moderations: humans, and machines.

> the mail servers. 

This seems true to me.  More important,

> Note we still don't have documentation on this stuff

I think this is a very serious problem.  Some of the issues have been
perplexing to diagnose because of the poor documentation.  We talked
about this most recently with respect to MX records and
higher-preference-number MXes having the user list from the final
destination, so that we could generate rejects consistently, IIRC.

A

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 16:30:13 +0100

> treating SMTP as though it's XMPP.  It ain't, and it's designed _not_

There's a difference between acceptable delay and what we're often
getting. Sure, SMTP should have latency. But a modern SMTP system
shouldn't take hours to deliver an email.

> demystifying mail issues than are posts to -performance without

Sure. But I can tell you that *every single time* I've looked at
latencies, the problem has been at postgresql.org or hub.org. And in my
own case, there is just one relay on the way, usually with a latency of
<5 seconds.

> So we're really paying for two moderations: humans, and machines.

That's very true.

> destination, so that we could generate rejects consistently, IIRC.

Can't agree more.

//Magnus

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 12:47:30 -0300

> shouldn't take hours to deliver an email.

Well, when  you can guarantee that the network connections between any relay 
server from the originator to the recipient is 100% up 100% of the time ... and 
that the mail server at each of the relays is processing properly 100% of the 
time ... and that there are no network delay issues at any one of the dozen or 
so routers, and ... then talk to me ...

> own case, there is just one relay on the way, usually with a latency of

Wow, I didn't know you were in Panama ... I have about 12 routers I go through 
from here, creating at least 12 failure points right there ... you being on the 
same network definitely should reduce your latency, we should look into that 
... what is your IP, so that I can confirm that from our end, we are seeing you 
as being 1 router away also ... maybe bad routing?

- ----

---

From: "Joshua D. Drake" <[email-masked]>
Date: Thu, 1 Nov 2007 08:47:52 -0700

On Thu, 01 Nov 2007 16:30:13 +0100
Magnus Hagander <[email-masked]> wrote:

> > expected in the mail system.  Every time I hear complaints about
> > mail latency that entails delays of merely hours, I worry that
> > people are treating SMTP as though it's XMPP.  It ain't, and it's
> > designed _not_ to be.
> 

Exactly. It is pretty silly to think that a modern, well engineered
system will take hours to deliver mail. It is like people have just
been brow beaten into accepting the poor performance of the lists.

There are exception of course... I run into Exchange not really likely
greylisting for example.

> latencies, the problem has been at postgresql.org or hub.org. And in
> my own case, there is just one relay on the way, usually with a
> latency of <5 seconds.
> 

This is also the case with me and I just gave up because nobody
actually seems to care about how bad the performance really is.

> >> It would be great if the actual sysadmin team had management
> >> ability on the mail servers. 
> > 
> > I think this is a very serious problem.  Some of the issues have
> > been perplexing to diagnose because of the poor documentation.  We
> > talked about this most recently with respect to MX records and

I wish -core actually realized how good it could be. 

Thousands of people rely on these lists. We advertise them as the form
of community support. They are, outside of the code our most important
feature that we provide to our community. Yet... 

Sincerely,

Joshua D. Drake

> broadcast)--------------------------- TIP 1: if posting/reading
> through Usenet, please send an appropriate subscribe-nomail command
> to [email-masked] so that your message can get through to
> the mailing list cleanly
> 

--

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 16:59:14 +0100

Marc G. Fournier wrote:
> 
> 
> --On Thursday, November 01, 2007 16:30:13 +0100 Magnus Hagander 

No. All those cases are reasons for acceptable delays. But how often
does say network connectivity go away for an hour? If they do, you need
to better hosting provider.

> as being 1 router away also ... maybe bad routing?

Notice, I say 5 *seconds*, not 5 *milliseconds*. And yes, the mx
machines of hub.org *most of the time* deliver in less than 5 seconds,
once it reaches the final hop inside hub.org.

This mail for example hit hub.org 200.46.204.184 (with broken name
lookup, it seems) at 12:48:12 ADT, and arrived on my server at 16:48:14
CET, which means it took *2* seconds.

A couple of minutes delay is perfectly acceptable. A couple of hours is
an indication that something is wrong.

//Magnus

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 13:16:01 -0300

> to better hosting provider.

You really don't have a clue on how an SMTP server works, do you?  If delivery 
fails, it backs up and tries again *later* ... if there is a high volume of 
email going through said server, *later* could very well be 1 hour ... and, in 
fact, its an incremental backup, so it actually works out to be something like:

Try now, fail, try in 5 minutes, fail, try in 10 minutes, fail, try in 20 
minutes, fail, etc ... I'm not sure if its a simple '2x' algorithm, but the 
delay between attempts does get progressively greater, so if it fails after 
trying at '40 minutes', then it will be another hour and a half after *that* 
beofre it will try again, etc ...

> an indication that something is wrong.

Well, when you see a couple of hours delay, then do something *useful* and let 
me know ... the only *useful* reports I've had in the past 24 hours dealt with 
a problem that Tom reported yesterday and that I fixed within minutes of him 
reporting ... the headers that you and Bruce sent me were *from that problem* 
...

- ----

---

From: "Joshua D. Drake" <[email-masked]>
Date: Thu, 1 Nov 2007 09:28:06 -0700

On Thu, 01 Nov 2007 13:16:01 -0300
"Marc G. Fournier" <[email-masked]> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> 
> 
> > does say network connectivity go away for an hour? If they do, you
> > need to better hosting provider.
> 
> You really don't have a clue on how an SMTP server works, do you? 

Let's not turn this into an attack fest please.

> If
> delivery fails, it backs up and tries again *later* ... if there is a

Why did delivery fail is the question.

> high volume of email going through said server, *later* could very
> well be 1 hour ... and, in fact, its an incremental backup, so it
> actually works out to be something like:

O.k. this is true but there is something wrong here. Why is the server
so backed up that this is happening in the first place. If the server
is actually loaded that much, we need to figure out how to get a better
server for mail shipped to panama.

Sincerely,

Joshua D. Drake

--

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 13:42:31 -0300

> Why did delivery fail is the question.

My logs go back 7 days for email ... if we had a message-id, we could look at 
that sort of thing relatively easily ...

> so backed up that this is happening in the first place.

I'm not so sure that it is 'so backed up' ... but, looking at mailq righ tnow, 
there are only 3867 messages in the queue ... some of the error codes are stuff 
like:

"450 mailbox unavailable"
   - great, should be 550, but as its 450, we'll keep retrying

"lost connection with mail.mindef.mil.gt"

"Host or domain name not found"

"451 SPAM not accepted"
   - again, if you aren't going to accept it, why temporary fail it so that I 
try again?

"Greylisting in action"

But, again, it would be interesting to look at a specific one, with message-id, 
and see what the logs show for reasons of delays ...

- ----

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 13:00:32 -0400

> shouldn't take hours to deliver an email.

This isn't automatically true, and is explicitly contradicted by the
relevant RFCs.  I think it shouldn't be the _habit_ on such systems,
but AFAICT it isn't.  

But "hours to deliver an email" is in fact totally reasonable on a
busy system.  I think good mail administrators aim for "in general,
minutes".  The problem here is the perception that it is too often
outside the "in general" assumption.  I think that perhaps it'd be
more useful in this discussion to archive, over (say) six months,
cases where you think the headers are showing unexplainable lag.  I
think there probably _is_ a problem, actually, but I haven't yet
written a procmail recipe to catch all pg-[list] mail that has any
header where the hop time was (say) over one hour.  _That_ is the
sort of catalogue we need.

A
-- 
Andrew Sullivan  | [email-masked]
The fact that technology doesn't work is no bar to success in the marketplace.
		--Philip Greenspun

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 13:02:27 -0400

On Thu, Nov 01, 2007 at 12:47:30PM -0300, Marc G. Fournier wrote:
> 
> Wow, I didn't know you were in Panama ... I have about 12 routers I
> go through from here, creating at least 12 failure points right
> there 

Um, if that's the case, then I _strongly_ suggest you learn the
meaning of "BGP" and implement it.  No ISP should have 12 _actual_
failure points from 12 routers in this day and age.  

A
-- 
Andrew Sullivan  | [email-masked]
The very definition of "news" is "something that hardly ever happens."	
		--Bruce Schneier

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 13:04:51 -0400

> minutes, fail, etc ... I'm not sure if its a simple '2x' algorithm, but the 

No, it's a progressive backoff with some randomisation in every
modern server I know of.  That's because, if there's some periodic
issue that causes DoS, then just backing off the same period would do
it again.  We already _had_ that nightmare on the Internet, and we
hope not to experience it again ;-)

A

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 13:14:52 -0400

On Thu, Nov 01, 2007 at 08:47:52AM -0700, Joshua D. Drake wrote:
> system will take hours to deliver mail. 

You have a strange idea of "well engineered" for very large mail
systems.

I have news for those on this list who do not operate systems with
thousands of simultaneous users: your assumption that a couple
hundred users scales to several thousands by adding hardware is as
wrong in mailing lists and DNS systems as it would be for newbies
implementing PostgreSQL systems.  It don't work that way, folks.

Keep in mind that the mail server relies on DNS servers that are out
there in ISP-land, and not in your-control land.  Since DNS
lookups can result in soft failures, it's not at all impossible that
mail to your specific mailbox will get delayed by annoying DNS
holdups that have more to do with, for instance, random nasty people
trying to prevent resolution of every site under .org.  Or under
.postgresql.org (I'd be interested to see graphs of query volumes
against that domain, BTW.  Because I bet they're all over the place
in unpredictable ways).

This means that, even though the mail has made it into the relay, it
might not make it out anything like as fast as you think, and there's
more than one system that can easily cause the problem.  Now,
multiply by, say, 1000 simultaneous delivery attempts, and you have
significant load issues that no mail system can solve, because the
fundamental Internet infrastructure is kinda broken that way.

So, without clear outlines of _exactly_ where the problem is, which
means logging of months of headers, putting aside only those messages
that hung up, this is going to amount to nothing more than "you did
x" " no I didn't lalalalala" discussion.  You want this to get
better?  Capture your logs, and let's do some analysis.

A

-- 
Andrew Sullivan  | [email-masked]
When my information changes, I alter my conclusions.  What do you do sir?
		--attr. John Maynard Keynes

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 13:36:32 -0400

>    - great, should be 550, but as its 450, we'll keep retrying

Hehe.  We just had this one, no?  ;-)  This is Yet Another Victim of
the "safe" mail delivery rules in postfix.  They're safe for the
recipient, of course, but hazardous to the Net.
 
> "lost connection with mail.mindef.mil.gt"

This is exactly the sort of thing we have to worry about.  On global
lists with subscribers possibly on crappy connections, the cost to
the relaying mail server go up _exponentially_.  Not everyone is in
well-connected locales, and managing backoff queues is bloody
expensive.  Add a few thousand users to the mix, and things go south
very quickly.

> "Host or domain name not found"

This should just bounce, no?

>    - again, if you aren't going to accept it, why temporary fail it so that I 

Indeed.  Some clever anti-spam people think that this is a clever
mechanism -- they "force the spammers to pay extra".  The problem is
when their not-quite-as-clever-as-they-thought filters catch
something the wrong way.  The entire Internet pays.  (This is also,
by the way, the basis for the "no thanks" response we should and
usually do give to people who show up with 80% solutions for
"performance problems" in PostgreSQL.)

A

---

From: Tom Lane <[email-masked]>
Date: Thu, 01 Nov 2007 15:23:12 -0400

> and see what the logs show for reasons of delays ...

Well, the current problems seem to be entirely inside hub.org, and so
all this banter about DNS etc seems not relevant.  A handy example is
this same message I'm replying to, which seems to have been hung up
for 2.5 hours:

Date: Thu, 01 Nov 2007 13:42:31 -0300
From: "Marc G. Fournier" <[email-masked]>
To: "Joshua D. Drake" <[email-masked]>
cc: Magnus Hagander <[email-masked]>,
        Andrew Sullivan <[email-masked]>, [email-masked]
Subject: Re: [pgsql-www] what is up with the PG mailing lists?
Sender: [email-masked]

			regards, tom lane

---

From: Andrew Sullivan <[email-masked]>
Date: Thu, 1 Nov 2007 15:59:03 -0400

> all this banter about DNS etc seems not relevant.  A handy example is

Could be.  The main point is to get examples of the sort you just
provided -- if we can get a nice collection, then it becomes
reasonably likely we'll track the issue.

That said, I'm not suggesting that DNS _is_ the problem, merely that
it could be.  Of course, if Marc is using the same DNS for internal
and external DNS queries, and the externals are under attack, then
his internal DNS is hosed.

Or if there's a reverse tree somewhere in the path that isn't working
100% of the time, and the mail servers are all trying to do reverse
lookups.  (This is a remarkably common source of breakage, I have
learned as a result of my work on a current internet draft.)

Mail gremlins are notorious for their difficulty in debugging partly
because of all the horrible interaction with DNS.  Add a couple
different name servers, 3 MX records, and the occasional bad reverse
path, and you're into serious pain, especially after your spam filter
proceeds to do much of it all over again.  Even a single hop can be
expensive then.  But I do think there's something going on in the
suspicious hop, and only Marc can diagnose it.

A

-- 
Andrew Sullivan  | [email-masked]
"The year's penultimate month" is not in truth a good way of saying
November.
		--H.W. Fowler

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 21:33:01 +0100

> You really don't have a clue on how an SMTP server works, do you?  If delivery 

Well, it's been a couple of years since I last wrote a code patch for a
SMTP server, but yeah, I have a fair clue on how it works. And I do run
servers that deliver some 100,000 mails a day. I know, it's not much,
but I know enough to keep those working, and I've never seen internal
delays like what we're seeing here.

> beofre it will try again, etc ...

That's an implementation detail, that differs wildly between different
SMTP servers. But you already know that of course. Postfix,
specifically, implements a '2x' algorithm. There's also a minimum
backoff time (configurable in new versions, previously fixed at 1000
seconds) and a maximum backoff time (configurable).

The main question remains. As Tom posted again in this thread, the delay
happens *internally between hub.org machines*. By your reasoning, that
means it's getting multiple failures to move mail internally. To me,
that's a clear indication that something is wrong. I'm sorry to hear you
don't agree.

> reporting ... the headers that you and Bruce sent me were *from that problem* 

I have given up. I used to send these, but nothing is fixed. Maybe I
should set up a procmail script to capture them...

Oh, and the headers I sent were because the email was stuck in the
moderation queue.

//Magnus

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 21:35:47 +0100

> relevant RFCs.  I think it shouldn't be the _habit_ on such systems,

AFAICT, it is.
And remember that we're talking about delivery *between two internal
relay machines*. Not delivery to the end user.

//Magnus

---

From: Dave Page <[email-masked]>
Date: Thu, 01 Nov 2007 20:46:46 +0000

> reporting ... the headers that you and Bruce sent me were *from that problem* 

OK, how about this one. This is from the first message I've received
since about 4:45PM GMT (~4 hours ago) - that alone tells me something is
wrong; it's not even that quiet on Christmas day!

[207.173.203.254])
	by postgresql.org (Postfix) with ESMTP id D3AE19FDA94
	for <[email-masked]>; Thu,  1 Nov 2007 13:58:41 -0300 (ADT)
Date: Thu, 1 Nov 2007 09:57:23 -0700
From: "Joshua D. Drake" <[email-masked]>
To: Dave Page <[email-masked]>
Cc: "Marc G. Fournier" <[email-masked]>,
        Postgresql Funds Group
 <[email-masked]>
Subject: Re: [PG-FG] Possible request for funds: PGCon Brazil

/D

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 18:03:22 -0300

> This should just bounce, no?

The full message looks like:

Host or domain name not found. Name service error for name=sigma.fr type=MX: 
Host not found, try again

so it looks like postfix is treating it as a soft failure, vs hard ...

- ----

---

From: Dave Page <[email-masked]>
Date: Thu, 01 Nov 2007 21:05:50 +0000

Dave Page wrote:
> Return-Path: <[email-masked]>

Here's another one that may be of interest as it's not a list message. I
won't send any more unless you ask for them - suffice it to say I have
now had two or three others, all equally late; I assume you fed
something a suitable laxative.

To: [email-masked]
From: [email-masked]
Reply-To:
Subject: Your Apple Order WXXXXXXX Has Been Shipped
Date: Thu,  1 Nov 2007 18:26:07 +0000 (GMT)

/D

---

From: Tom Lane <[email-masked]>
Date: Thu, 01 Nov 2007 17:31:15 -0400

> Oh, and the headers I sent were because the email was stuck in the

Yeah --- one of the worst problems in diagnosing this is that there's
no way for anyone except the moderator to know if a delay was simply
waiting-for-moderation or if it indicates an actual system problem.

What are the chances of getting something into the headers to indicate
whether a message was delayed by moderation?

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 01 Nov 2007 22:52:06 +0100

> whether a message was delayed by moderation?

Not sure - Marc? You're the one who know majordomo...

Anyway, I just ran a script on all the mail sitting in my mailbox from
postgresql mailinglists delivered in the past two hours. It's a total of
22 mails (should be more, I know - but I need to deal with differences
in date before I get there, this is a first rough cut of the script).
Here's the top lines from the query:

select path,count(*),avg(t)::int from data group by path order by 2
desc,3 desc;

on this data. Basically, it's the paths that appear often, and the
average delivery time between them (in seconds). I've only included
paths that appear 7 or more times, plus the two rows for mails posted by
Tom and me (there are others that appear twice as well, but I'm not
including those).

This is all delivery to me, which is why the path
mx2.hub.org->svr2.hagander.net and svr2.hagander.net->svr2.hagander.net
(my antispam) is included.

                                         path
               | count | avg
---------------------------------------------------------------------------------------+-------+------
 postgresql.org -> mx2.hub.org
               |    22 |   57
 svr2.hagander.net -> svr2.hagander.net
               |    22 |    1
 mx2.hub.org -> svr2.hagander.net
               |    22 |    1
 postgresql.org -> localhost (mx1.hub.org [200.46.204.184])
               |    15 | 3335
 localhost (mx1.hub.org [200.46.204.184]) -> postgresql.org
               |    15 |   23
 postgresql.org -> localhost (mx1.hub.org [200.46.204.183])
               |     7 | 2710
 localhost (mx1.hub.org [200.46.204.183]) -> postgresql.org
               |     7 |   17
 sss.pgh.pa.us (8.14.1/8.14.1) -> postgresql.org
               |     3 |    8
 svr2.hagander.net -> postgresql.org
               |     2 |    2

(sorry about the breakup of the table, it's the long lines for
mx1.hub.org. Table attached as well for easier reading)

To me this clearly shows that mail is backed up between postgresql.org
(this is svr1, if I'm not mistaken) and mx1.hub.org - which has two
different IPs, but claims to be localhost (that can't be right, can it?).

FYI, the count measures the difference in time between the Received:
lines in the header. It's not fool-proof, but it works on all the paths
inside hub.org and from there to my machine - it misses some originating
MUAs (it's missed 2 out of the 22 emails parsed here)

//Magnus
                                         path                                          | count | avg  
---------------------------------------------------------------------------------------+-------+------
 postgresql.org -> mx2.hub.org                                                         |    22 |   57
 svr2.hagander.net -> svr2.hagander.net                                                |    22 |    1
 mx2.hub.org -> svr2.hagander.net                                                      |    22 |    1
 postgresql.org -> localhost (mx1.hub.org [200.46.204.184])                            |    15 | 3335
 localhost (mx1.hub.org [200.46.204.184]) -> postgresql.org                            |    15 |   23
 postgresql.org -> localhost (mx1.hub.org [200.46.204.183])                            |     7 | 2710
 localhost (mx1.hub.org [200.46.204.183]) -> postgresql.org                            |     7 |   17
 sss.pgh.pa.us (8.14.1/8.14.1) -> postgresql.org                                       |     3 |    8
 svr2.hagander.net -> postgresql.org                                                   |     2 |    2

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 19:06:03 -0300

> something a suitable laxative.

postfix stop; postfix start

- ----

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 19:16:44 -0300

> whether a message was delayed by moderation?

I've asked on the mj2 list about this when someone mentioned it earlier in the 
thread, and am just awaiting confirmation, but the initial answer seemed to 
indicate that someone would have to do some perl programming for this, as it 
isn't a pre-defined variable that we can access  ...

As I said, I'm waiting to hear back, but if my assessment is right, anyone 
interested in writing some perl code?  Have both a 'X-Approved-Date:' and an 
'X-Approved-Delay:' header would be cool ...

- ----

---

From: Dave Page <[email-masked]>
Date: Thu, 01 Nov 2007 22:25:23 +0000

Marc G. Fournier wrote:
> 
> 
> --On Thursday, November 01, 2007 21:05:50 +0000 Dave Page 

On developer.postgresql.org? Any idea what caused it to hang? Any ideas
if it's a state we can monitor in nagios in any sensible way?

/D

---

From: "Marc G. Fournier" <[email-masked]>
Date: Thu, 01 Nov 2007 19:43:00 -0300

- --On Thursday, November 01, 2007 22:25:23 +0000 Dave Page 
<[email-masked]> wrote:

> On developer.postgresql.org?

yes ...

> Any idea what caused it to hang? Any ideas

I'm finding absolutely zilch in the log files in the way of errors on either 
that server, or 200.46.204.184 ... its as if port 10025 (the report port for 
amavis) just stop'd responding ...

Please do look at the logs on the mail server though (and/or login to 204.184) 
and let me know if you see something that I didn't ... note that on 204.184, 
there are some 10025 related errors at the time I restarted postfix, which do 
make sense, since 10025 would have 'disappeared' then ...

- ----

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Fri, 02 Nov 2007 07:47:30 +0100

Marc G. Fournier wrote:
> 
> 
> --On Thursday, November 01, 2007 13:36:32 -0400 Andrew Sullivan 

a query for the MX of that domain results in SERVFAIL which MTAs usually
treat like a temporary and not a permanent error one.

Stefan

---

From: Magnus Hagander <[email-masked]>
Date: Fri, 02 Nov 2007 08:33:56 +0100

Marc G. Fournier wrote:
> 
> 
> --On Thursday, November 01, 2007 17:31:15 -0400 Tom Lane <[email-masked]> 
> wrote:
> 

Do we have any pointers as to where the code goes? I don't know how to
find our way around the code, but if we can get a pointer to where it
needs to go, someone can perhaps do something about it. But it's
probably as much work for the mj2 folks to tell us where to put it as it
would be for them to put it there themselves :-)

//Magnus

---

From: Andreas 'ads' Scherbaum <[email-masked]>
Date: Tue, 6 Nov 2007 20:44:48 +0100

On Thu, 1 Nov 2007 13:14:52 -0400 Andrew Sullivan wrote:

Hello all,

> there in ISP-land, and not in your-control land.

speaking of DNS, i get a lot of errors the last days:

Nov  6 20:20:41 base postfix/smtpd[15808]: warning: smtpd_peer_init: 200.46.204.187: address not listed for hostname maia-2.hub.org

I tested from different hosts in different networks at different times and
i can't resolve 'maia-2.hub.org.' to an ip-address. So this seems to be a
constant problem.

Maybe someone can fix this?

> better?  Capture your logs, and let's do some analysis.

I just posted one of the problems, let's start here and move forward.

Kind regards

-- 
				Andreas 'ads' Scherbaum
Failure is not an option. It comes bundled with your Microsoft product.
 (Ferenc Mantfeld)

---

From: Tom Lane <[email-masked]>
Date: Tue, 06 Nov 2007 14:50:09 -0500

> i can't resolve 'maia-2.hub.org.' to an ip-address. So this seems to be a

Yeah, I see the same: 200.46.204.187 resolves as maia-2.hub.org
but forward resolution of that name fails.  So, inconsistency in
the DNS tables someplace ...

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 06 Nov 2007 20:51:36 +0100

> the DNS tables someplace ...

Same issue exists for maia-1.hub.org (from the logs I checked yesterday,
so I don't recall the IP. But it should be in Marcs inbox)

//Magnus

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 06 Nov 2007 16:31:27 -0400

'k, I'm confused ... the only name that should be referenced in any config file 
is maia.hub.org, which breaks down as:

Name:   maia.hub.org
Address: 200.46.204.184
Name:   maia.hub.org
Address: 200.46.204.183

So, how/where are ppl seeing 200.46.204.187 :(

Unless postfix is caching it internally ... I just restarted postfix, in case 
that is what is happening, but I just checked the mailq also, and nothing is 
queue'd up with them ... most odd ... I've re-added them to DNS while I look 
into it, to at least quiet the error ...

- --On Tuesday, November 06, 2007 14:50:09 -0500 Tom Lane <[email-masked]> 
wrote:

>> Nov  6 20:20:41 base postfix/smtpd[15808]: warning: smtpd_peer_init:
>> 200.46.204.187: address not listed for hostname maia-2.hub.org
>
> TIP 2: Don't 'kill -9' the postmaster

- ----

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 06 Nov 2007 21:33:08 +0100

> So, how/where are ppl seeing 200.46.204.187 :(

dig -x 200.46.204.187

returns
187.204.46.200.in-addr.arpa. 18000 IN   PTR     maia-2.hub.org.

Similar for -1.

//Magnus

---

From: "Marc G. Fournier" <[email-masked]>
Date: Tue, 06 Nov 2007 17:37:18 -0400

- --On Tuesday, November 06, 2007 21:33:08 +0100 Magnus Hagander 
<[email-masked]> wrote:

> Marc G. Fournier wrote:
>>
>> 'k, I'm confused ... the only name that should be referenced in any config
>> file  is maia.hub.org, which breaks down as:
>>
>> Name:   maia.hub.org
>> Address: 200.46.204.184
>> Name:   maia.hub.org
>> Address: 200.46.204.183
>>

Right, and?

- ----

---

From: Andreas 'ads' Scherbaum <[email-masked]>
Date: Tue, 6 Nov 2007 23:37:23 +0100

> > 187.204.46.200.in-addr.arpa. 18000 IN   PTR     maia-2.hub.org.

ads@iridium:/home/ads > dig -x maia-2.hub.org.
dig: '.org.hub.maia-2.in-addr.arpa.' is not a legal name (empty label)

And where this came from?

Nov  1 20:30:53 base postfix/smtpd[11763]: warning: smtpd_peer_init: 200.46.204.187: address not listed for hostname maia-2.hub.org
Nov  1 20:30:54 base postfix/smtpd[11763]: NOQUEUE: reject: RCPT from unknown[200.46.204.187]: 450 <maia-2.hub.org>: Helo command rejected: Host not found; from=<[email-masked]> to=<[email-masked]> proto=ESMTP helo=<maia-2.hub.org>

a) the ip-address has an invalid (missing) ptr record
b) this server is knowing itself by maia-2.hub.org, which cannot resolved
(but matches the invalid ptr record)

Bye

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Wed, 07 Nov 2007 10:06:31 +0100

> dig: '.org.hub.maia-2.in-addr.arpa.' is not a legal name (empty label)

-x implies a reverse/PTR lookup so you need to supply an ip-address ...

> (but matches the invalid ptr record)

well this problem (200.46.204.187 having a PTR of maia-2.hub.org but 
forward resolution is not correct/missing) seems to be fixed now (and 
you log entry is now nearly a week old)

Stefan

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 7 Nov 2007 08:59:08 -0300

> > 187.204.46.200.in-addr.arpa. 18000 IN   PTR     maia-2.hub.org.

It's an A RR for maia.hub.org.

$ dig maia.hub.org

; <<>> DiG 9.4.1-P1 <<>> maia.hub.org
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53021
;; flags: qr rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 3, ADDITIONAL: 3

;; QUESTION SECTION:
;maia.hub.org.                  IN      A

;; ANSWER SECTION:
maia.hub.org.           86400   IN      A       200.46.204.182
maia.hub.org.           86400   IN      A       200.46.204.183
maia.hub.org.           86400   IN      A       200.46.204.184
maia.hub.org.           86400   IN      A       200.46.204.187
maia.hub.org.           86400   IN      A       200.46.204.191

These numbers all resolve back to a different name:
$ for i in 182 183 184 187 191; do dig +short -x 200.46.204.$i; done
maia-5.hub.org.
maia-4.hub.org.
maia-3.hub.org.
maia-2.hub.org.
maia-1.hub.org.

The names resolve OK to the numbers:

$ for i in 182 183 184 187 191; do dig +short -x 200.46.204.$i; done |
xargs -n 1 dig +short
200.46.204.182
;; Warning: ID mismatch: expected ID 15438, got 50558
200.46.204.183
200.46.204.184
200.46.204.187
200.46.204.191

-- 
Alvaro Herrera                               http://www.PlanetPostgreSQL.org/
"Las cosas son buenas o malas segun las hace nuestra opiniÃ³n" (Lisias)

---

From: Andreas 'ads' Scherbaum <[email-masked]>
Date: Wed, 7 Nov 2007 13:52:00 +0100

> -x implies a reverse/PTR lookup so you need to supply an ip-address ...

Ups, my fault, thanks.
The days ago i mostly used 'host' and got errors back.

> you log entry is now nearly a week old)

I just grepped the monthly log and copied the first message ;-)
The last of this error messages appeared yesterday evening 10pm my time.
Since then it seems to work.

Bye

---

From: "Marc G. Fournier" <[email-masked]>
Date: Wed, 07 Nov 2007 13:13:22 -0400

Actually, what is above is from my fixing the issue ... to date, there has been 
no indication as to *why* th e original problem, as those hosts were taken out 
of the RR, since nothing from someone else's server should even be going 
through those hosts (with or without the RR in place) ...

- --On Wednesday, November 07, 2007 08:59:08 -0300 Alvaro Herrera 
<[email-masked]> wrote:

> TIP 7: You can help support the PostgreSQL project by donating at
>
>                 http://www.postgresql.org/about/donate

- ----