Subject: Insecure DNS servers on PG infrastructure

From: Tom Lane <[email-masked]>
Date: Fri, 25 Jul 2008 11:02:03 -0400

I just noted that cvs.postgresql.org and svr1.postgresql.org are not
running the latest bind release, which means that they are vulnerable to
the DNS cache poisoning attack recently discovered by Dan Kaminsky.
Vixie and co think this is a pretty big deal, so folks might want to
update sooner rather than later.
	http://www.kb.cert.org/vuls/id/800113

BTW, there is an excellent end-to-end test available for whether the
security fix (port randomization) is actually working for you:

	dig @server-to-test porttest.dns-oarc.net in txt

This takes a few seconds (they've arranged it to force multiple queries
from the tested server) and gives you back a readout of how many ports
those queries arrived from and the spread in the port addresses.
A good result looks about like this:

;; ANSWER SECTION:
porttest.dns-oarc.net.  60      IN      CNAME   z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
z.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net. 60 IN TXT "66.207.139.134 is GOOD: 26 queries in 2.3 seconds from 26 ports with std dev 17102.06"

If it says FAIR or POOR then you have an unpatched server or there
is something interfering with the port randomization.  If the server
is behind a NAT firewall then the latter is entirely likely.

			regards, tom lane

---

From: "Joshua D. Drake" <[email-masked]>
Date: Fri, 25 Jul 2008 08:26:01 -0700

> 	http://www.kb.cert.org/vuls/id/800113

Dave and Magnus are on vacation. I believe the only other people that
would have access to those boxes are Stefan and Marc. I have pinged
Stefan.

Joshua D. Drake

-- 
The PostgreSQL Company since 1997: http://www.commandprompt.com/ 
PostgreSQL Community Conference: http://www.postgresqlconference.org/
United States PostgreSQL Association: http://www.postgresql.us/
Donate to the PostgreSQL Project: http://www.postgresql.org/about/donate

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 25 Jul 2008 11:40:49 -0400

> update sooner rather than later.

This is an extremely big deal.  The numbers I've seen suggest windows
somewhere around 10 minutes.  If the systems above are doing
recursion, then they need to be patched right away.  (If they're
running both authority and recursive services in the same BIND
instance, I suggest that the practice be abandoned immediately.)

> those queries arrived from and the spread in the port addresses.

Note the TTL, BTW, so that running it twice in a row gives exactly the
same answer.  You need to wait for the cache to expire to repeat your tests.

> is behind a NAT firewall then the latter is entirely likely.

There's no reason that a NAT should do that, if the device is
competently built: if you randomise source ports on the inside, the
NAT device could just use the same port on the outside.

Much worse is the bad habit many got into recently of nailing the
source port to 53, and then only opening that on the firewall on the
way out.  Some packages have had this behaviour turned on by default.
Setting the source port this way will foil the patch that is included
in BIND.  You need to make your recursive queries from a random port,
and the port needs to be unpredictable by someone watching your
resolver.

If you want to try another recursive-only resolver (one that is much
more recent and so was never vulnerable to this attack), have a look
at unbound (find it at unbound.net).  It can do DNSSEC validation for
you too.

BTW, the strategies that BIND and others are using to foil the
Kaminsky attack merely reduce the liklihood, and don't prevent it
completely.  There's not a lot of source ports over which to
randomize.  The real solution to all this is DNSSEC, but deploying it
isn't trivial, and the root isn't signed yet.  (.ORG will be soon,
though!)

A

-- 
Andrew Sullivan
[email-masked]
+1 503 667 4564 x104
http://www.commandprompt.com/

---

From: Tom Lane <[email-masked]>
Date: Fri, 25 Jul 2008 16:44:32 -0400

> NAT device could just use the same port on the outside.

I'm not convinced that that's true.  If the router is trying to forward
UDP messages arriving from several "inside" IP addresses using only one
"outside" address, it has to deal with the possibility of collisions,
ie two "inside" addresses using the same port number at about the same
time.  So it doesn't surprise me that it rewrites the port numbers.
If it assigned randomly-generated substitute numbers there'd be no
problem, but with no prior knowledge that would be a good idea you can
hardly blame the router authors for not indulging in extra complexity.

What I do know is that my own firewall hardware (a Netopia T1 router
that's two or three years old) *was* rewriting UDP port numbers on
requests from a machine that was sharing a NAT address with others.
After remapping to give that machine its own "outside" IP address,
it stopped doing so.  BTW the porttest.dns-oarc.net service was
invaluable in testing this; I'd probably have thought that just
installing the new BIND made me safe, if I hadn't had a way to test it.

			regards, tom lane

---

From: Andrew Sullivan <[email-masked]>
Date: Fri, 25 Jul 2008 18:04:48 -0400

On Fri, Jul 25, 2008 at 04:44:32PM -0400, Tom Lane wrote:

> time.  

This is true.  They can't arrive at exactly the same time, though,
which means that different strategies can be used.  It's certainly
true, however, that one of the strategies may well be to rewrite port
numbers.

In some sense, rewriting to the same port number makes things quite a
bit worse for the router, because rather than just remembering "oh,
port O1 was port I1 and port O2 was port I2", the router has to
remember which {staticport,Iport} pair belongs with which inside
address.  So more state is needed.  (Now everyone can be amazed at
just how fast a hand can be made to wave.  But this is the gist of the
argument.)

> requests from a machine that was sharing a NAT address with others.

It is a problem, for sure, and the OARC test is a big help.  Yay
OARC (full disclosure: my former employer isa major OARC sponsor).

A

---

From: "Joshua D. Drake" <[email-masked]>
Date: Sun, 27 Jul 2008 11:34:30 -0700

>> NAT device could just use the same port on the outside.

Tom can you check if this has been resolved? If not I am going to start 
paging people.

Joshua D. Drake

---

From: Tom Lane <[email-masked]>
Date: Sun, 27 Jul 2008 14:52:26 -0400

> Tom can you check if this has been resolved? If not I am going to start 

The machines I have access to (cvs and svr1) are still unpatched
according to the "dig porttest" test.  Personally, though, I'd be
more interested in getting someone to work on fixing the mailing
lists.  I've received nothing since about midnight EDT, and I know
that there has been traffic.

			regards, tom lane

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Sun, 27 Jul 2008 21:59:00 +0200

> instance, I suggest that the practice be abandoned immediately.)

cvs.postgresql.org is not running bind at all - what it is using are two 
(purely) recursive resolvers upstream. One of them is only going to get 
upgraded tomorrow(some changes need to be rolled out in a staged 
fashion) the other one was done a while ago - I have simply removed that 
one from the resolv.conf for the time being.

Stefan

---

From: Alvaro Herrera <[email-masked]>
Date: Sun, 27 Jul 2008 19:46:57 -0400

> that there has been traffic.

Maybe the mail servers have been DDoS'ed again?

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.

---

From: "Joshua D. Drake" <[email-masked]>
Date: Sun, 27 Jul 2008 17:26:11 -0700

> Maybe the mail servers have been DDoS'ed again?

I did have a bunch of alerts throw recovery around 2:00am but then where 
is all the traffic?

---

From: "Marc G. Fournier" <[email-masked]>
Date: Mon, 28 Jul 2008 01:26:36 -0300

> I just noted that cvs.postgresql.org and svr1.postgresql.org are not

Was out camping, just saw this ... svr1.postgresql.org's name server is 
shutdown/disabled now, since its not needed ... and all of the primary name 
servers have had recursion disabled since Andrew pointed out to me awhile back 
how to do so ...

- -- 
Marc G. Fournier        Hub.Org Hosting Solutions S.A. (http://www.hub.org)
Email . [email-masked]                              MSN . [email-masked]
Yahoo . yscrappy               Skype: hub.org        ICQ . 7615664