Subject: New archives for testing

From: Magnus Hagander <[email-masked]>
Date: Fri, 28 Dec 2012 15:32:21 +0100

After way too much time, in mainly due to long delays between doing
different parts, I think we're finally ready to do some public testing
of the new mailinglist archive site/code. The biggest technical change
is that with this code, the data is now stored in (I know, this is
shocking for a project like ours!) a database, which means we can do a
lot more with the presentation than we could before. The main
differences from before are:

* No more breaking threads at month boundaries. A thread can go on for
any amount of time, containing any amount of messages. Our current
record is 344 messages in a thread. Threading is not perfect, of
course, so there are bound to be some threads - particularly old ones
- that are broken into two. This becomes more visible now that we
don't arbitrarily break *all* threads, but it's fundamentally a
problem with the underlying data not containing all the information we
need.

* A thread is no longer bound to a list. A list is just a "tag" on a
thread, meaning a single thread can be in multiple lists. As soon as a
message is CCed to multiple lists, the whole thread will show up on
them, meaning it's now possible to follow a discussion when it's
moved.

* It's now possible to navigate the whole thread from inside a message
view, using a dropdown listing the whole thing. It's still possible to
navigate with to the next and previous in thread at the bottom of the
message, just like before.

* There's also a "flat view" that shows an entire message thread on
one page, for those who prefer that kind of view. (It works pretty
well in most cases, but becomes a huge page for example on the 344
message thread)

* There is no longer a per-month, per-list sequence number for each
message, that causes havoc if accidentally reset (as has happened a
couple of times in the mhonarc history, when the mbox files have been
edited - intenitionally or not). Instead, the messageid url is the
primary URL for all pages, and it's permanent.

* For SEO reasons mainly, the new archives live under the main website
URL space, starting point is http://www.postgresql.org/list/

* Raw messages and mbox files are now protected with http basic
authentication and a fixed password. In the old archives, the software
worked hard to obfuscate email addresses in the headers and such of
the visible email, but they were all fully visible if you clicked
"raw" or downloaded the mbox, making the antispam measures kind of
pointless. The username and password for the protection is listed in
the password prompt, so it shouldn't be a problem for any manual
navigation, but hopefully enough to keep bots out.

* Contents are no longer updated by a cronjob running every 15
minutes. Instead, email is injected directly into the archives as soon
as it leaves majordomo on the list server, and becomes available
within second(s). It still needs to pass antispam, enter majordomo and
possibly be moderated, so it doesn't mean a second after someone hit
"send" in their MUA, but it should be significantly faster than
before.

Now, having said that, I'd like to see some more people testing it
than the few people I've forced to do it so far. The way to test it is
to go to http://www.postgresql.org/list/ and pick your list. You can
also go to http://www.postgresql.org/message-id/<message-id> to view a
message in a thread - that should also work fine if you just replace
"archives" with "www" in the URL of an existing message in the
archives *if* you were using the message-id based url.

A few things that are known not to work at this point:
* Searching by message-id will not work. This is because the new
search code hasn't been deployed, since it would replace the old one
and can't be launched side by side.

* This also means that searching in general will return search hits on
the old archives site, not the new one, even when searching from the
new one.

* There is no redirect from the old archives. There is code available
so that once we consider it production, all old
archives.postgresql.org URLs will redirect to the new site - directly
to the proper message. But this is not deployed yet, as this is a
partial deploy.

So. Please go test it. And give some feedback.

While there is a lot that can be done to improve the experience, and
we can discuss that endlessly, I would like to ask people for now to
focus on things that *don't work*, or things that are *worse than
before*. Once we've tried to deal with those as well as we can, we can
switch over. Then we can improve things further. But let's not get
bogged down trying to add every new feature now - let's just get it
far enough to make it better than before as a first step.

--
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-www mailing list ([email-masked])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-www

---

From: Bruce Momjian <[email-masked]>
Date: Fri, 28 Dec 2012 18:22:07 -0500

> far enough to make it better than before as a first step.

I am confused about how to read a thread.  If I an on this email
message:

	http://www.postgresql.org/message-id/[email-masked]

how do I read the next email in the thread.  Is it the two "Responses"? 
The old system would just have a "next in thread" option and could just
keeping hitting "next in thread" to see all the thread emails.

-- 
  Bruce Momjian  <[email-masked]>        http://momjian.us
  EnterpriseDB                             http://enterprisedb.com

  + It's impossible for everything to be true. +

---

From: Magnus Hagander <[email-masked]>
Date: Sat, 29 Dec 2012 10:53:49 +0100

> how do I read the next email in the thread.  Is it the two "Responses"?

Yes, "responses" corresponds to what was previously "follow ups".

You can also navigate the thread using the thread dropdown in the
message header.

> keeping hitting "next in thread" to see all the thread emails.

If you keep hitting next in thread, I don't think you get that though
- if the thread splits out into two, don't you just end up walking
down one branch?

If you want to navigate the whole thread at once, using the "flat
view" is probably more efficient - it gives you a lot fewer clicks.
Try clicking the "flat" link in the message header

---

From: Bruce Momjian <[email-masked]>
Date: Sat, 29 Dec 2012 10:52:04 -0500

> Try clicking the "flat" link in the message header

Ah, I like the flat view.  Good.

---

From: Dimitri Fontaine <[email-masked]>
Date: Sun, 30 Dec 2012 16:36:52 +0100

> So. Please go test it. And give some feedback.

I like it better than the old one. The thread navigation feels strange
as a drop down menu. The attachement downloading works better than
before, as it keeps the file name.

Congrats, and +1 for going live with the current version!

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr     PostgreSQL : Expertise, Formation et Support

---

From: Dave Page <[email-masked]>
Date: Sun, 30 Dec 2012 18:28:00 +0000

> navigation, but hopefully enough to keep bots out.

The prompt isn't shown on all browsers, so we should stick it on the
website somewhere too.

--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake

EnterpriseDB UK: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 30 Dec 2012 20:17:09 +0100

> The prompt isn't shown on all browsers, so we should stick it on the

Ugh, that's annoying.

Do you think we can get away without putting it next to every single
link? Because I'm not sure how we can do that without making it look
like crap. But if we don't, are people likely to ever read it?

---

From: Tom Lane <[email-masked]>
Date: Sun, 30 Dec 2012 15:42:33 -0500

> like crap. But if we don't, are people likely to ever read it?

How big a problem is this?  When I checked, both Safari and Firefox
showed the prompt.  If there are just a few little-used browsers that
fail to show it, I'm not convinced that we have to clutter the pages
for everybody to cater to them.  I can think of more than a few other
sites where that prompt is pretty damn essential for usability, so
I would argue that a browser that doesn't show it is broken anyhow.

A bigger issue is that with both Safari and Firefox, the first login
attempt failed and I had to try again.  Now maybe I fat-fingered the
password both times, but I suspect there is something wrong there.

			regards, tom lane

---

From: Dave Page <[email-masked]>
Date: Sun, 30 Dec 2012 20:53:59 +0000

On Sunday, December 30, 2012, Tom Lane wrote:

> Magnus Hagander <[email-masked] <javascript:;>> writes:
> > On Sun, Dec 30, 2012 at 7:28 PM, Dave Page <[email-masked]<javascript:;>>
> wrote:

I don't think it was originally intended as a prompt (it's the security
realm actually), but most browsers showed it anyway and it's been (ab)used
that way for years. FYI, the browser I saw not displaying it was Safari on
iOS, so most definitely not 'little used'.

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 30 Dec 2012 22:32:42 +0100

> iOS, so most definitely not 'little used'.

No, but not showing it makes it a pretty useless browser since it's
supposed to tell the user which password to use when different
sections on a site has different passwords.

That said, it doesn't matter how stupid or useless it is, if it's
reality :) We just have to deal with it. I'm not too worried about
iphone users - i doubt either the raw or the mbox view is very
interesting to them. Same for mbox users on iPad - however, I can
certainly see iPad users who want to get the raw view.

Right now, mobile is about 1.2% of our visitors to the archives.
Safari on ios 0.5%. iPhone is just over 0.3% and iPad just over 0.3%.

So it's a very small portion of our visitors. While the total numbers
are likely going up, I'm not sure those who actually want the raw
and/or mbox files are going to go up.

FWIW, it works fine in Chrome (46%), Firefox (36%) and Safari-desktop
(3%). Unverified at this point are IE (11%) and Opera(2%), the rest
are really far down the list.

So the question is how much effort we want to put into it. If we make
the 401 page itself contain the text, does that show up in safari
after authentication has failed, or does it show some custom page?

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 30 Dec 2012 22:36:41 +0100

> after authentication has failed, or does it show some custom page?

ads just confirmed it works on IE as well.

---

From: Peter Geoghegan <[email-masked]>
Date: Sun, 30 Dec 2012 21:43:39 +0000

> ads just confirmed it works on IE as well.

Have you tried using a service to do this for you?

https://browsershots.org/http://www.postgresql.org/message-id/[email-masked]

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 30 Dec 2012 22:48:56 +0100

> https://browsershots.org/http://www.postgresql.org/message-id/[email-masked]

The point is what happens when you go to the raw url. It won't me
submit any new ones, but I somehow have a feeling it won't work with
the password protected URL...

And also, it doesn't include the mobile browsers anyway, which seem to
be the only place this one has issues.

---

From: Tom Lane <[email-masked]>
Date: Mon, 31 Dec 2012 17:47:14 -0500

> ...

At least on iOS 6, Safari doesn't seem to show any 401 page at all.
When you hit the "raw" link, you get an "Authentication required"
popup with just space for username and password.  If you put in
a wrong value, the popup re-appears.  There's not much you can
do except hit "Cancel".  Not very helpful at all I'd say.  (Now
admittedly, on a phone-size screen it's not clear that there's
room for much of a prompt, but still...)

Having just done the experiment, though, I'd have to say that the
usability of the archives is pretty darn low regardless of this.
Too many very small links too close together --- there's basically
no way to hit what you want accurately without zooming way in first.
(And that was on an iPad; don't even want to think about a phone.)
I can't see anybody really caring about either the mbox or raw links
in that context.

But on the third hand ... could we rig it to accept any old name and
password?  The mere occurrence of a challenge ought to be enough to
discourage most bots.

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Tue, 1 Jan 2013 15:24:40 +0100

> room for much of a prompt, but still...)

Well, the page usually shows up once you hit cancel. It's not very
user friendly, but that page is at least in theory customizable. But I
think a lot of browsers don't show it.

There is plenty of room on the phone screen to do a prompt. At least
android has no problem at all with it. But that doesn't really matter
if a platform that's half of our mobile visitors can't handle it -
because we can't change that. Unless we want to take the same approach
as we do with some of the windows code, which is say "it's good
enough, if people want the better functionality they should pick a
more suitable platform". (which for the access of raw or mbox isn't
entirely unreasonable, really..)

> no way to hit what you want accurately without zooming way in first.

I guess I'm spoiled by a browser that auto-zooms just the links when
you accidentally click next to another one, making that a non-issue.
But probably more useful, we could do with a mobile adapted version
*period*. The whole site, where the archives inherits the style, works
fairly badly on small screens (and really big ones, it only really
works well for medium sized ones).

But is it actually any worse than the old archives? Because they work
pretty bad in mobile as well, don't they? Personally, I find them even
harder since the text of the emails tends to be smaller in comparison
to the header... And if it's not a regression against the new ones, I
think it needs to go on the TODO list rather than being a blocker..

> password?  The mere occurrence of a challenge ought to be enough to

Not easily. We could for the raw links, because that authentication
prompt comes from our app. But the mboxes are served directly by the
webserver, which has a fixed password list. So we'd have to write our
own auth module to do that, which is a piece of work I don't think we
want to take on.

---

From: Tom Lane <[email-masked]>
Date: Tue, 01 Jan 2013 12:13:15 -0500

> Well, the page usually shows up once you hit cancel.

Not in Safari...

> entirely unreasonable, really..)

I don't have a problem with that approach.  As you say, a
mobile-optimized version of the site would be the only real fix for some
of these issues, and it doesn't seem to me that we have the manpower or
interest for that.  (Yet, anyway.)

> But is it actually any worse than the old archives?

Only in this one regard of having an unexpected password prompt show up.

			regards, tom lane

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 3 Jan 2013 13:58:38 -0300

> message, just like before.

After exchanging a couple of comments with Dimitri over IM on this
topic, I wonder if there's a better way to represent the sorrounding
thread contect, rather than just "responses" and "in response to" links.
The dropdown is good to have the whole thing in a single view, but when
Dimitri pointed me to this archive sample:
https://lists.gnu.org/archive/html/emacs-devel/2012-12/msg00692.html
I wonder if we can do something similar.  So we would list:

1. the parent of the current message (a clickable link)
2. all children of the current message's parent, including the current
message itself.  All but the current one would be clickable links.
Current one would be in a different color, and perhaps have some other
"you are here" marker.
3. all children of the current message

This way, we're not drawing a full graph of the thread (which would be
too large), but a graph complete enough to see the immediate context.

(It might be useful to annotate the pruned ancestors and sub-branches
with some numbers to let the user know there are hidden messages there.)

-- 
Álvaro Herrera                http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services

---

From: Alvaro Herrera <[email-masked]>
Date: Thu, 3 Jan 2013 14:18:07 -0300

Alvaro Herrera wrote:
> I wonder if we can do something similar.

FWIW I don't think we should hold migration for this; I'm just pointing
out a possible way to improve this *after* we migrate.

I also think we should *not* hold migration due to the auth issues Tom
pointed out for mbox and raw access.  I mean, what's the use in having
the mbox in a phone or tablet?  The raw message might be useful, if you
have a good enough text editor on the phone; I don't really know if
Emacs has been ported to iPad?

---

From: Dave Page <[email-masked]>
Date: Thu, 3 Jan 2013 17:26:35 +0000

On Thu, Jan 3, 2013 at 5:18 PM, Alvaro Herrera <[email-masked]> wrote:
> I don't really know if

No, why would anyone want that? Vim is available of course :-)

---

From: Dimitri Fontaine <[email-masked]>
Date: Thu, 03 Jan 2013 21:52:47 +0100

> out a possible way to improve this *after* we migrate.

+1 to both.

> Emacs has been ported to iPad?

There is of course Emacs Gestures for touch screens. Still, +1.

Regards,

---

From: Tom Lane <[email-masked]>
Date: Thu, 03 Jan 2013 18:54:07 -0500

> Emacs has been ported to iPad?

FWIW, it wasn't me that pointed those out.  I'm fine with not fixing
them before we migrate (in fact, not sure we need to fix 'em ever).

			regards, tom lane

---

From: Alvaro Herrera <[email-masked]>
Date: Fri, 4 Jan 2013 11:19:42 -0300

> archives *if* you were using the message-id based url.

Actually, hold the presses.  See this message:
http://www.postgresql.org/message-id/[email-masked]
It says "the patch is attached", but there is no attachment.
The old archives do contain the patch.

---

From: Magnus Hagander <[email-masked]>
Date: Sat, 5 Jan 2013 15:05:58 +0100

> The old archives do contain the patch.

Bug found and fixed. The problem was that we missed attachments that
were text/plain, did not have a name, but did have a filename. I first
thought it was another case of the From line in mbox, but if you
looked at the raw output you could see that we actually had the whole
thing in there. So in passing, I also wrote a small tool that lets us
reparse messages that are already in the db, whenever we change/fix
the parsing rules.

---

From: Tom Lane <[email-masked]>
Date: Sun, 06 Jan 2013 14:12:07 -0500

Another minor gripe about the new archives: compare
http://archives.postgresql.org/pgsql-hackers/2013-01/msg00218.php
http://www.postgresql.org/message-id/[email-masked]

Although we're not exactly big on pretty pictures on these lists,
I think it's a usability disimprovement that the new archive fails
to include the image in-line.

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 6 Jan 2013 21:41:49 +0100

> to include the image in-line.

Depends on the image, I think. For this specific image, I agree it is.
But if the image is larger (which is not unusual for example for
screenshots) it breaks the layout pretty badly.

We could inline all images, with that risk. Or maybe we could actually
parse the image to figure out it's size, and show it only when it's
smaller than a certain size. That might be a useful compromise...

---

From: Peter Geoghegan <[email-masked]>
Date: Sun, 6 Jan 2013 20:51:54 +0000

> smaller than a certain size. That might be a useful compromise...

How likely is it in practice that an image is any other type of image?
Ideally, you'd be able to do something with the image metadata to
inline an image if its resolution is below a certain threshold. If
that doesn't work out, I'd just inline png, gif and svg images, and
leave it at that.

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 6 Jan 2013 21:53:45 +0100

> that doesn't work out, I'd just inline png, gif and svg images, and

Well, that's what I'm considering, but inlining screenshots at high
resolutions is going to look Really Bad... But yes, the idea would be
to parse out the resolution from the metadata somehow.

---

From: Greg Stark <[email-masked]>
Date: Sun, 6 Jan 2013 20:59:13 +0000

> to parse out the resolution from the metadata somehow.

I would be more concerned with the file size. You don't want someone
to be able to make an archive page send a hapless viewer a 100M image
file. Luckily file size doesn't require parsing the meta data.

Can you not deal with the image size by just sticking a css attribute
for max-height and max-width on the inlined image?

-- 
greg

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 6 Jan 2013 22:12:44 +0100

> for max-height and max-width on the inlined image?

That's certainly another option - limit it by size that way, and then
limiting it by filesize for whether we include it or not. We can even
give it scrollbars, I guess. That does make even more sense.

---

From: Tom Lane <[email-masked]>
Date: Sun, 06 Jan 2013 16:25:08 -0500

> give it scrollbars, I guess. That does make even more sense.

A file-size-based limit on what we'll inline seems perfectly sensible
from here.  As for scrollbars, if it's big enough to need those, we
probably don't want it inline ...

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 6 Jan 2013 22:46:13 +0100

> probably don't want it inline ...

Yeah, I was thinking scrollbars rather than cut. Turns out the
max-width option actually rescales the image, so we're fine.

I've applied a change that makes it now inline all images <75kb, and
rescaling them down to 600 pixels wide if they are bigger.

The example URL you used have been reloaded from the new code, so it
shows what it looks like. Any other pages that had images on them that
were already viewed are currently cached the old way, but will
automatically update once they expire from the cache (max 4 hours).

---

From: Tom Lane <[email-masked]>
Date: Sun, 06 Jan 2013 18:21:51 -0500

> rescaling them down to 600 pixels wide if they are bigger.

Better, but in
http://www.postgresql.org/message-id/[email-masked]
it seems to have forgotten about the second attachment ...

			regards, tom lane

---

From: Magnus Hagander <[email-masked]>
Date: Wed, 9 Jan 2013 18:09:02 +0100

> it seems to have forgotten about the second attachment ...

Turns out that one was because the second attachment didn't have a
name, and as such it was confusing it with the majordomo-added footer.
Basically, any text/plain parts that didn't have a filename were
ignored. Code updated so it now includes any except the first one
(which is the mail itself) as an attachment.

---

From: Noah Misch <[email-masked]>
Date: Sun, 13 Jan 2013 12:00:43 -0500

> well in most cases, but becomes a huge page for example on the 344

I will use this view all the time.  Thanks.

> authentication and a fixed password.

For at least pgsql-hackers and pgsql-bugs, the mbox files appear to have
frozen on 2012-12-09.  The 201212 files have no messages past that date, and
the 201301 files do not exist.

---

From: Magnus Hagander <[email-masked]>
Date: Sun, 13 Jan 2013 18:08:19 +0100

> the 201301 files do not exist.

Ah, it appears I forgot to enable a cronjob. Should be done now.

---

From: Noah Misch <[email-masked]>
Date: Wed, 16 Jan 2013 22:05:11 -0500

> reparse messages that are already in the db, whenever we change/fix

The following message exhibits similar symptoms; the attachment appears in the
raw and mbox versions only:

http://www.postgresql.org/message-id/[email-masked]

It, too, is text/plain with a "filename" and no "name".  Some attachments I
sent on 2013-01-08, using the same email toolchain, show up fine.

---

From: Magnus Hagander <[email-masked]>
Date: Thu, 17 Jan 2013 10:33:38 +0100

> sent on 2013-01-08, using the same email toolchain, show up fine.

Yeah, Alvaro fixed that one - I just wanted to note it here so you
knew it was fixed :)

We haven't figured out how to auto-reparse all those messages yet, so
right now we're fixing them on a case by case basis when they're
found. So keep reporting them!