Subject: 403 - Forbidden on gitweb pages

From: Tom Lane <[email-masked]>
Date: Wed, 13 Feb 2013 19:19:17 -0500

Every so often (a few percent of the time, I think) I get "403 -
Forbidden" from following a link on http://git.postgresql.org/gitweb/.
So far, a reload has always worked first time, but there's definitely
something fishy in there.  I don't recall exactly when this started
happening, maybe a month or so back; it hasn't been there all along.

Anybody else see this, or have an idea what broke?

			regards, tom lane

-- 
Sent via pgsql-www mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

---

From: Alvaro Herrera <[email-masked]>
Date: Wed, 13 Feb 2013 21:24:54 -0300

> Anybody else see this, or have an idea what broke?

Yes, I see this also.  I reported it to Magnus, who muttered something
about Varnish possibly misbehaving, but that was the end of it.

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

---

From: Dave Page <[email-masked]>
Date: Thu, 14 Feb 2013 00:35:27 +0000

> about Varnish possibly misbehaving, but that was the end of it.
He told me it was intentional rate limiting, applied because gitweb is so
inefficient.

-- 
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 14 Feb 2013 05:24:59 +0100

inefficient.

Correct. And varnish is not involved at all. I think alvaro is confusing it
with me saying that varnish might be a way for us to actually fix the
problem.

The problem is basically that gitweb is a big perl cgi (yes, cgi only)
which means that for every request it had to reload all of perl and reparse
the whole script. And on top of that the actual implementation of several
features is fairly slow in itself.

This caused easy overload - for example the whole site went down when a
search engine spidered us...

The bandaid solution was to rate limit access per ip. The error you hit is
when you're rate limited.

Now, I was supposed to have on my to do list to make a better fix. But as
this one was implemented a long time ago (measured in years, really, I have
to admit I forgot... I'll put bit back on the list.

/Magnus

---

From: Tom Lane <[email-masked]>
Date: Wed, 13 Feb 2013 23:31:38 -0500

> when you're rate limited.

[ raised eyebrow... ]  I'm fairly sure I've seen it more than once when
performing a *single* page fetch.  In any case, the probability of
failure increased by a couple orders of magnitude sometime in the past
month or so, because I'd never seen it before that.

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 14 Feb 2013 06:06:16 +0100

> month or so, because I'd never seen it before that.

In that case you may be hitting a different issue, and we're writing it off
as the rare limiting by mistake. Because nothing about that one has changed
for quite some time, and should never hit on a single page. Will have to
investigate that further then.

/Magnus

---

From: Dimitri Fontaine <[email-masked]>
Date: Thu, 14 Feb 2013 22:26:43 +0100

> features is fairly slow in itself.

What about finding a more efficient replacement?

  http://gitlabhq.com/
  http://hjemli.net/git/cgit/

We need to check that GitLab can be setup to not offer all of its
features, and the CGIT way of displaying branches looks very much suited
to our project back branches support.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Thu, 14 Feb 2013 22:53:59 +0100

> month or so, because I'd never seen it before that.

any chance you moved to a different browser (or a new version of it) in
that timeframe?
I can trivially reproduce that issue here now because my browser is
employing fairly agressive prefetching techniques that the currently
rate-limiting system is not prepared to deal with, and from looking at
the logs this is actually a fairly common issue :(

Stefan

---

From: Tom Lane <[email-masked]>
Date: Thu, 14 Feb 2013 18:28:38 -0500

> the logs this is actually a fairly common issue :(

Hm.  I usually use Apple's Safari, which is currently at 6.0.2, and
it looks like I installed that update at the beginning of November.
It's possible they instituted aggressive prefetching in the 6.0.1 to
6.0.2 update, but somehow I doubt that.

Would it make sense to just back off the rate limiting a bit?

			regards, tom lane

---

From: Stefan Kaltenbrunner <[email-masked]>
Date: Fri, 15 Feb 2013 15:33:18 +0100

> 6.0.2 update, but somehow I doubt that.

hm ok...

> Would it make sense to just back off the rate limiting a bit?

done that for now until we have a better solution - lets see if it
behaves better now.

Stefan

---

From: Magnus Hagander <[email-masked]>
Date: Sat, 23 Feb 2013 13:51:37 +0100

> done that for now until we have a better solution - lets see if it

I've deployed what I think is a better fix for this now - it's now
limited to two parallel *active* connections from wherever, but any
further requests are placed in a queue rather than being rejected iwth
403.

As a bonus, we now also have caching. That makes particularly the
frontpage quite a lot faster for most people - not likely to have a
big effect on details pages, since there are just too many of them to
get efficient caching.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/