public inbox for [email protected]  

help / color / mirror / Atom feed

From: Oleg Bartunov <[email protected]>
To: [email protected]
Subject: incomplete headers: archives.postgresql.org
Date: Tue, 13 Jan 2004 20:52:52 +0300 (MSK)
Message-ID: <[email protected]> (raw)

Hi there,

crawling of archives.postgresql.org is a pain, because there are no
last-modified information in headers and crawler have to download message
again. For example:

megera@mira:~$ curl -I http://archives.postgresql.org/pgsql-hackers/2004-01/msg00282.php
HTTP/1.1 200 OK
Date: Tue, 13 Jan 2004 17:38:26 GMT
Server: Apache/1.3.28 (Unix) PHP/4.3.3RC1
X-Powered-By: PHP/4.3.3RC1
Content-Type: text/html

Is't possible to add, at least,  header 'Last-Modified', so crawler could
understand if this page should be downloaded again ? It'll save bandwidth
and time to crawle. I think the best way to set 'Last-Modified' header
to date of message from 'Date:' field. Of course, there are should be
proof from 'bad clocks', so default time may be arrival time.

Also, it could be useful to add 'Expires' header.
I think, headers should be added only to pages with individual message, not
to indexes, because index pages are indeed changed.

I don't think it's very difficult, but it help site and people.


btw, I use cacheability to check if page could cached:
http://www.sai.msu.su/admin/cacheability/?query=http%3A%2F%2Farchives.postgresql.org%2Fpgsql-hackers...

http://archives.postgresql.org/pgsql-hackers/2004-01/msg00282.php
Expires   	  -
Cache-Control   	  -
Last-Modified   	  -
ETag   	  -
Content-Length  	  - (actual size: 13277)
Server  	Apache/1.3.28 (Unix) PHP/4.3.3RC1

This object will be considered stale, because it doesn't have any freshness
information assigned. It doesn't have a validator present. It doesn't have a Content-Length header present, so it can't be used in a HTTP/1.0 persistent connection.





	Regards,
		Oleg
_____________________________________________________________
Oleg Bartunov, sci.researcher, hostmaster of AstroNet,
Sternberg Astronomical Institute, Moscow University (Russia)
Internet: [email protected], http://www.sai.msu.su/~megera/
phone: +007(095)939-16-83, +007(095)939-23-83

view thread (2+ messages)  latest in thread

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients using the --to and --cc options:
  reply via email

  To: [email protected]
  Cc: [email protected]
  Subject: Re: incomplete headers: archives.postgresql.org
  In-Reply-To: <[email protected]>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox