From: Heikki Linnakangas <hlinnakangas@vmware.com>
To: Robert Haas <robertmhaas@gmail.com>
Cc: Greg Stark <stark@mit.edu>
Cc: Tan Tran <tankimtran@gmail.com>
Cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: GSoC on WAL-logging hash indexes
Date: Fri, 07 Mar 2014 11:34:39 +0200
Message-ID: <531992AF.2080306@vmware.com> (raw)
In-Reply-To: <CA+TgmoZ=HQDaynaRyppSXfYyfTUKrCbk7arFT6Hati7seNmHvg@mail.gmail.com>
References: <B444B441-397F-4EDA-8A0C-877B9AF7033B@gmail.com>
<54977A8C-4AF5-4BCC-B021-78E9CA978077@gmail.com>
<CA+TgmoY4X7VKyC4dAwujuTuBoyxE2qSJF9aybhWZjxXWoc6J8w@mail.gmail.com>
<CAM-w4HNGN+eyDzZEaZB_XHLfMCGMqZNZ+2GJpty+PdrDEtzeWg@mail.gmail.com>
<531873E8.6050802@vmware.com>
<CA+TgmoZ=HQDaynaRyppSXfYyfTUKrCbk7arFT6Hati7seNmHvg@mail.gmail.com>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
On 03/06/2014 09:34 PM, Robert Haas wrote:
> On Thu, Mar 6, 2014 at 8:11 AM, Heikki Linnakangas
> <hlinnakangas@vmware.com> wrote:
>> I don't think it's necessary to improve concurrency just to get WAL-logging.
>> Better concurrency is a worthy goal of its own, of course, but it's a
>> separate concern.
>
> To some extent, I agree, but only to some extent. To make hash
> indexes generally usable, we really need to solve both problems. When
> I got rid of just some of the heavyweight locking in commit
> 76837c1507cb5a5a0048046233568447729e66dd, the results were pretty
> dramatic at higher levels of concurrency:
>
> http://www.postgresql.org/message-id/CA+Tgmoaf=nOJxLyzGcbrrY+pe-0VLL0vfHi6tjdM3fFtVwsOmw@mail.gmail....
>
> But there was still an awful lot of contention inside the heavyweight
> lock manager, and I don't think hash indexes are going to be ready for
> prime time until we solve that problem.
Hmm. You suggested ensuring that a scan always has at least a pin, and
split takes a vacuum-lock. That ought to work. There's no need for the
more complicated maneuvers you described, ISTM that you can just replace
the heavy-weight share lock with holding a pin on the primary page of
the bucket, and an exclusive lock with a vacuum-lock. Note that
_hash_expandtable already takes the exclusive lock conditionally, ie. if
it doesn't get the lock immediately it just gives up. We could do the
same with the cleanup lock.
Vacuum could also be enhanced. It currently takes an exclusive lock on
the bucket, then removes any dead tuples and finally "squeezes" the
bucket by moving tuples to earlier pages. But you only really need the
exclusive lock for the squeeze-phase. You could do the dead-tuple
removal without the bucket-lock, and only grab for the squeeze-phase.
And the squeezing is optional, so you could just skip that if you can't
get the lock. But that's a separate patch as well.
One more thing we could do to make hash indexes more scalable,
independent of the above: Cache the information in the metapage in
backend-private memory. Then you (usually) wouldn't need to access the
metapage at all when scanning. Store a copy of the bitmask for that
bucket in the primary page, and when scanning, check that it matches the
cached value. If not, refresh the cached metapage and try again.
So, there seems to be a few fairly simple and independent improvements
to be made:
1. Replace the heavy-weight lock with pin & vacuum-lock.
2. Make it crash-safe, by adding WAL-logging
3. Only acquire the exclusive-lock (vacuum-lock after step 1) in VACUUM
for the squeeze phase.
4. Cache the metapage.
We still don't know if it's going to be any better than B-tree after all
that's done, but the only way to find out is to go ahead and implement it.
This seems like a great GSoC project to me. We have a pretty good idea
of what we want to accomplish. It's uncontroversial: I don't think
anyone is going to object to improving hash indexes (one could argue
that it's a waste of time, but that's different from objecting to the
idea). And it consists of a few mostly independent parts, so it's
possible to do incrementally which makes it easier to track progress,
and we'll probably have something useful at the end of the summer even
if it doesn't all get finished.
- Heikki
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-students@postgresql.org
Cc: hlinnakangas@vmware.com, robertmhaas@gmail.com, stark@mit.edu, tankimtran@gmail.com, pgsql-hackers@postgresql.org
Subject: Re: GSoC on WAL-logging hash indexes
In-Reply-To: <531992AF.2080306@vmware.com>
References: <B444B441-397F-4EDA-8A0C-877B9AF7033B@gmail.com> <54977A8C-4AF5-4BCC-B021-78E9CA978077@gmail.com> <CA+TgmoY4X7VKyC4dAwujuTuBoyxE2qSJF9aybhWZjxXWoc6J8w@mail.gmail.com> <CAM-w4HNGN+eyDzZEaZB_XHLfMCGMqZNZ+2GJpty+PdrDEtzeWg@mail.gmail.com> <531873E8.6050802@vmware.com> <CA+TgmoZ=HQDaynaRyppSXfYyfTUKrCbk7arFT6Hati7seNmHvg@mail.gmail.com> <531992AF.2080306@vmware.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox