pg.ddx.io pgsql-committers@postgresql.org mailing list archive
help / color / mirror / Atom feedpgsql: dshash: Add sequential scan support.
11+ messages / 5 participants
[nested] [flat]
Commits from this thread:
2022-07-11 5e7608e81e 7cdd0c2d7c 4f88dbac22 cd26139a30 53df1e28d2 eed959a457 Fix lock assertions in dshash.c.
* pgsql: dshash: Add sequential scan support.
@ 2022-03-11 01:02 Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 11+ messages in thread
From: Andres Freund @ 2022-03-11 01:02 UTC (permalink / raw)
To: pgsql-committers@lists.postgresql.org
dshash: Add sequential scan support.
Add ability to scan all entries sequentially to dshash. The interface is
similar but a bit different both from that of dynahash and simple dshash
search functions. The most significant differences is that dshash's interfac
always needs a call to dshash_seq_term when scan ends. Another is
locking. Dshash holds partition lock when returning an entry,
dshash_seq_next() also holds lock when returning an entry but callers
shouldn't release it, since the lock is essential to continue a scan. The
seqscan interface allows entry deletion while a scan is in progress using
dshash_delete_current().
Reviewed-By: Andres Freund <andres@anarazel.de>
Author: Kyotaro Horiguchi <horikyoga.ntt@gmail.com>
Branch
------
master
Details
-------
https://git.postgresql.org/pg/commitdiff/352d297dc74feb0bf0dcb255cc0dfaaed2b96c1e
Modified Files
--------------
src/backend/lib/dshash.c | 163 ++++++++++++++++++++++++++++++++++++++-
src/include/lib/dshash.h | 23 ++++++
src/tools/pgindent/typedefs.list | 1 +
3 files changed, 186 insertions(+), 1 deletion(-)
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
@ 2022-03-11 01:09 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 11+ messages in thread
From: Tom Lane @ 2022-03-11 01:09 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: pgsql-committers@lists.postgresql.org
Andres Freund <andres@anarazel.de> writes:
> dshash: Add sequential scan support.
> Add ability to scan all entries sequentially to dshash. The interface is
> similar but a bit different both from that of dynahash and simple dshash
> search functions. The most significant differences is that dshash's interfac
> always needs a call to dshash_seq_term when scan ends.
Umm ... what about error recovery? Or have you just cemented the
proposition that long-lived dshashes are unsafe?
regards, tom lane
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
@ 2022-03-11 01:27 ` Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Andres Freund @ 2022-03-11 01:27 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; Thomas Munro <thomas.munro@gmail.com>; +Cc: pgsql-committers@lists.postgresql.org, Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
Hi,
On 2022-03-10 20:09:56 -0500, Tom Lane wrote:
> Andres Freund <andres@anarazel.de> writes:
> > dshash: Add sequential scan support.
> > Add ability to scan all entries sequentially to dshash. The interface is
> > similar but a bit different both from that of dynahash and simple dshash
> > search functions. The most significant differences is that dshash's interfac
> > always needs a call to dshash_seq_term when scan ends.
>
> Umm ... what about error recovery? Or have you just cemented the
> proposition that long-lived dshashes are unsafe?
I don't think this commit made it worse. dshash_seq_term() releases an lwlock
(which will be released in case of an error) and unsets
hash_table->find_[exclusively_]locked. The latter weren't introduced by this
patch, and are also set by dshash_find().
I agree that ->find_[exclusively_]locked are problematic from an error
recovery perspective.
It's per-backend state at least and just used for assertions. We could remove
it. Or stop checking it in places where it could be set wrongly: dshash_find()
and dshash_detach() couldn't check anymore, but the rest of the assertions
would still be valid afaics?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
@ 2022-07-04 02:55 ` Thomas Munro <thomas.munro@gmail.com>
2022-07-04 08:31 ` Re: pgsql: dshash: Add sequential scan support. Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2022-07-04 10:46 ` Re: pgsql: dshash: Add sequential scan support. Zhihong Yu <zyu@yugabyte.com>
2022-07-04 20:54 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
0 siblings, 3 replies; 11+ messages in thread
From: Thomas Munro @ 2022-07-04 02:55 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers <pgsql-hackers@postgresql.org>
[Re-directing to -hackers]
On Fri, Mar 11, 2022 at 2:27 PM Andres Freund <andres@anarazel.de> wrote:
> On 2022-03-10 20:09:56 -0500, Tom Lane wrote:
> > Andres Freund <andres@anarazel.de> writes:
> > > dshash: Add sequential scan support.
> > > Add ability to scan all entries sequentially to dshash. The interface is
> > > similar but a bit different both from that of dynahash and simple dshash
> > > search functions. The most significant differences is that dshash's interfac
> > > always needs a call to dshash_seq_term when scan ends.
> >
> > Umm ... what about error recovery? Or have you just cemented the
> > proposition that long-lived dshashes are unsafe?
>
> I don't think this commit made it worse. dshash_seq_term() releases an lwlock
> (which will be released in case of an error) and unsets
> hash_table->find_[exclusively_]locked. The latter weren't introduced by this
> patch, and are also set by dshash_find().
>
> I agree that ->find_[exclusively_]locked are problematic from an error
> recovery perspective.
Right, as seen in the build farm at [1]. Also reproducible with something like:
@@ -269,6 +269,14 @@ dsm_impl_posix(dsm_op op, dsm_handle handle, Size
request_size,
return false;
}
+ /* XXX random fault injection */
+ if (op == DSM_OP_ATTACH && random() < RAND_MAX / 8)
+ {
+ close(fd);
+ elog(ERROR, "chaos");
+ return false;
+ }
+
I must have thought that it was easy and practical to write no-throw
straight-line code and be sure to reach dshash_release_lock(), but I
concede that it was a bad idea: even dsa_get_address() can throw*, and
you're often likely to need to call that while accessing dshash
elements. For example, in lookup_rowtype_tupdesc_internal(), there is
a sequence dshash_find(), ..., dsa_get_address(), ...,
dshash_release_lock(), and I must have considered the range of code
between find and release to be no-throw, but now I know that it is
not.
> It's per-backend state at least and just used for assertions. We could remove
> it. Or stop checking it in places where it could be set wrongly: dshash_find()
> and dshash_detach() couldn't check anymore, but the rest of the assertions
> would still be valid afaics?
Yeah, it's all for assertions... let's just remove it. Those
assertions were useful to me at some stage in development but won't
hold as well as I thought, at least without widespread PG_FINALLY(),
which wouldn't be nice.
*dsa_get_address() might need to adjust the memory map with system
calls, which might fail. If you think of DSA as not only an allocator
but also a poor man's user level virtual memory scheme to tide us over
until we get threads, then this is a pretty low level kind of
should-not-happen failure that is analogous on some level to SIGBUS or
SIGSEGV or something like that, and we should PANIC. Then we could
claim that dsa_get_address() is no-throw. At least, that was one
argument I had with myself while investigating that strange Solaris
shm_open() failure, but ... I lost the argument. It's quite an
extreme position to take just to support these assertions, which are
of pretty limited value.
[1] https://www.postgresql.org/message-id/20220701232009.jcwxpl45bptaxv5n%40alap3.anarazel.de
Attachments:
0001-Remove-spurious-assertions-from-dshash.c.patch (text/x-patch, 6.5K) download
From e3e87a9ec39d3456a7bf29796102502d3724993e Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 4 Jul 2022 11:56:02 +1200
Subject: [PATCH] Remove spurious assertions from dshash.c.
dshash.c maintained flags to track locking, in order to make assertions
that the dshash API was being used correctly. Unfortunately the
interaction with ereport's non-local exits was not thought through
carefully enough. While lwlocks are released automatically, the flags
can get out of sync. Since they're only used for assertions anyway, we
can just remove them.
This problem was noted by Tom and Andres while reviewing changes to
support the new shared memory stats system in 15, the second user of
dshash.c. On closer inspection, even the earlier typcache.c code is not
guaranteed to meet the asserted conditions, now that it's been
highlighted that even dsa_get_address() can throw (albeit in unlikely
circumstances).
Back-patch to 11, where dshash.c arrived.
Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Reported-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220311012712.botrpsikaufzteyt@alap3.anarazel.de
diff --git a/src/backend/lib/dshash.c b/src/backend/lib/dshash.c
index ec454b4d65..9706e7926b 100644
--- a/src/backend/lib/dshash.c
+++ b/src/backend/lib/dshash.c
@@ -110,8 +110,6 @@ struct dshash_table
dshash_table_control *control; /* Control object in DSM. */
dsa_pointer *buckets; /* Current bucket pointers in DSM. */
size_t size_log2; /* log2(number of buckets) */
- bool find_locked; /* Is any partition lock held by 'find'? */
- bool find_exclusively_locked; /* ... exclusively? */
};
/* Given a pointer to an item, find the entry (user data) it holds. */
@@ -234,9 +232,6 @@ dshash_create(dsa_area *area, const dshash_parameters *params, void *arg)
}
}
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
-
/*
* Set up the initial array of buckets. Our initial size is the same as
* the number of partitions.
@@ -285,8 +280,6 @@ dshash_attach(dsa_area *area, const dshash_parameters *params,
hash_table->params = *params;
hash_table->arg = arg;
hash_table->control = dsa_get_address(area, control);
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
Assert(hash_table->control->magic == DSHASH_MAGIC);
/*
@@ -309,8 +302,6 @@ dshash_attach(dsa_area *area, const dshash_parameters *params,
void
dshash_detach(dshash_table *hash_table)
{
- Assert(!hash_table->find_locked);
-
/* The hash table may have been destroyed. Just free local memory. */
pfree(hash_table);
}
@@ -400,7 +391,6 @@ dshash_find(dshash_table *hash_table, const void *key, bool exclusive)
partition = PARTITION_FOR_HASH(hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
LWLockAcquire(PARTITION_LOCK(hash_table, partition),
exclusive ? LW_EXCLUSIVE : LW_SHARED);
@@ -418,8 +408,6 @@ dshash_find(dshash_table *hash_table, const void *key, bool exclusive)
else
{
/* The caller will free the lock by calling dshash_release_lock. */
- hash_table->find_locked = true;
- hash_table->find_exclusively_locked = exclusive;
return ENTRY_FROM_ITEM(item);
}
}
@@ -449,7 +437,6 @@ dshash_find_or_insert(dshash_table *hash_table,
partition = &hash_table->control->partitions[partition_index];
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
restart:
LWLockAcquire(PARTITION_LOCK(hash_table, partition_index),
@@ -494,8 +481,6 @@ restart:
}
/* The caller must release the lock with dshash_release_lock. */
- hash_table->find_locked = true;
- hash_table->find_exclusively_locked = true;
return ENTRY_FROM_ITEM(item);
}
@@ -514,7 +499,6 @@ dshash_delete_key(dshash_table *hash_table, const void *key)
bool found;
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
hash = hash_key(hash_table, key);
partition = PARTITION_FOR_HASH(hash);
@@ -551,14 +535,10 @@ dshash_delete_entry(dshash_table *hash_table, void *entry)
size_t partition = PARTITION_FOR_HASH(item->hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(hash_table->find_exclusively_locked);
Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition),
LW_EXCLUSIVE));
delete_item(hash_table, item);
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
LWLockRelease(PARTITION_LOCK(hash_table, partition));
}
@@ -572,13 +552,8 @@ dshash_release_lock(dshash_table *hash_table, void *entry)
size_t partition_index = PARTITION_FOR_HASH(item->hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition_index),
- hash_table->find_exclusively_locked
- ? LW_EXCLUSIVE : LW_SHARED));
+ Assert(LWLockHeldByMe(PARTITION_LOCK(hash_table, partition_index)));
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
LWLockRelease(PARTITION_LOCK(hash_table, partition_index));
}
@@ -644,7 +619,6 @@ dshash_seq_next(dshash_seq_status *status)
if (status->curpartition == -1)
{
Assert(status->curbucket == 0);
- Assert(!status->hash_table->find_locked);
status->curpartition = 0;
@@ -702,8 +676,6 @@ dshash_seq_next(dshash_seq_status *status)
status->curitem =
dsa_get_address(status->hash_table->area, next_item_pointer);
- status->hash_table->find_locked = true;
- status->hash_table->find_exclusively_locked = status->exclusive;
/*
* The caller may delete the item. Store the next item in case of
@@ -722,9 +694,6 @@ dshash_seq_next(dshash_seq_status *status)
void
dshash_seq_term(dshash_seq_status *status)
{
- status->hash_table->find_locked = false;
- status->hash_table->find_exclusively_locked = false;
-
if (status->curpartition >= 0)
LWLockRelease(PARTITION_LOCK(status->hash_table, status->curpartition));
}
@@ -743,8 +712,6 @@ dshash_delete_current(dshash_seq_status *status)
Assert(status->exclusive);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(hash_table->find_exclusively_locked);
Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition),
LW_EXCLUSIVE));
@@ -762,7 +729,6 @@ dshash_dump(dshash_table *hash_table)
size_t j;
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
for (i = 0; i < DSHASH_NUM_PARTITIONS; ++i)
{
--
2.36.1
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
@ 2022-07-04 08:31 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2 siblings, 0 replies; 11+ messages in thread
From: Kyotaro Horiguchi @ 2022-07-04 08:31 UTC (permalink / raw)
To: thomas.munro@gmail.com; +Cc: andres@anarazel.de; tgl@sss.pgh.pa.us; horiguchi.kyotaro@lab.ntt.co.jp; pgsql-hackers@postgresql.org
At Mon, 4 Jul 2022 14:55:43 +1200, Thomas Munro <thomas.munro@gmail.com> wrote in
> [Re-directing to -hackers]
>
> On Fri, Mar 11, 2022 at 2:27 PM Andres Freund <andres@anarazel.de> wrote:
> > It's per-backend state at least and just used for assertions. We could remove
> > it. Or stop checking it in places where it could be set wrongly: dshash_find()
> > and dshash_detach() couldn't check anymore, but the rest of the assertions
> > would still be valid afaics?
>
> Yeah, it's all for assertions... let's just remove it. Those
> assertions were useful to me at some stage in development but won't
> hold as well as I thought, at least without widespread PG_FINALLY(),
> which wouldn't be nice.
>
> *dsa_get_address() might need to adjust the memory map with system
> calls, which might fail. If you think of DSA as not only an allocator
> but also a poor man's user level virtual memory scheme to tide us over
> until we get threads, then this is a pretty low level kind of
> should-not-happen failure that is analogous on some level to SIGBUS or
> SIGSEGV or something like that, and we should PANIC. Then we could
> claim that dsa_get_address() is no-throw. At least, that was one
> argument I had with myself while investigating that strange Solaris
> shm_open() failure, but ... I lost the argument. It's quite an
> extreme position to take just to support these assertions, which are
> of pretty limited value.
>
> [1] https://www.postgresql.org/message-id/20220701232009.jcwxpl45bptaxv5n%40alap3.anarazel.de
FWIW, the discussion above is convincing to me and the patch looks good.
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
@ 2022-07-04 10:46 ` Zhihong Yu <zyu@yugabyte.com>
2 siblings, 0 replies; 11+ messages in thread
From: Zhihong Yu @ 2022-07-04 10:46 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Andres Freund <andres@anarazel.de>; Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers <pgsql-hackers@postgresql.org>
On Sun, Jul 3, 2022 at 7:56 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> [Re-directing to -hackers]
>
> On Fri, Mar 11, 2022 at 2:27 PM Andres Freund <andres@anarazel.de> wrote:
> > On 2022-03-10 20:09:56 -0500, Tom Lane wrote:
> > > Andres Freund <andres@anarazel.de> writes:
> > > > dshash: Add sequential scan support.
> > > > Add ability to scan all entries sequentially to dshash. The
> interface is
> > > > similar but a bit different both from that of dynahash and simple
> dshash
> > > > search functions. The most significant differences is that dshash's
> interfac
> > > > always needs a call to dshash_seq_term when scan ends.
> > >
> > > Umm ... what about error recovery? Or have you just cemented the
> > > proposition that long-lived dshashes are unsafe?
> >
> > I don't think this commit made it worse. dshash_seq_term() releases an
> lwlock
> > (which will be released in case of an error) and unsets
> > hash_table->find_[exclusively_]locked. The latter weren't introduced by
> this
> > patch, and are also set by dshash_find().
> >
> > I agree that ->find_[exclusively_]locked are problematic from an error
> > recovery perspective.
>
> Right, as seen in the build farm at [1]. Also reproducible with something
> like:
>
> @@ -269,6 +269,14 @@ dsm_impl_posix(dsm_op op, dsm_handle handle, Size
> request_size,
> return false;
> }
>
> + /* XXX random fault injection */
> + if (op == DSM_OP_ATTACH && random() < RAND_MAX / 8)
> + {
> + close(fd);
> + elog(ERROR, "chaos");
> + return false;
> + }
> +
>
> I must have thought that it was easy and practical to write no-throw
> straight-line code and be sure to reach dshash_release_lock(), but I
> concede that it was a bad idea: even dsa_get_address() can throw*, and
> you're often likely to need to call that while accessing dshash
> elements. For example, in lookup_rowtype_tupdesc_internal(), there is
> a sequence dshash_find(), ..., dsa_get_address(), ...,
> dshash_release_lock(), and I must have considered the range of code
> between find and release to be no-throw, but now I know that it is
> not.
>
> > It's per-backend state at least and just used for assertions. We could
> remove
> > it. Or stop checking it in places where it could be set wrongly:
> dshash_find()
> > and dshash_detach() couldn't check anymore, but the rest of the
> assertions
> > would still be valid afaics?
>
> Yeah, it's all for assertions... let's just remove it. Those
> assertions were useful to me at some stage in development but won't
> hold as well as I thought, at least without widespread PG_FINALLY(),
> which wouldn't be nice.
>
> *dsa_get_address() might need to adjust the memory map with system
> calls, which might fail. If you think of DSA as not only an allocator
> but also a poor man's user level virtual memory scheme to tide us over
> until we get threads, then this is a pretty low level kind of
> should-not-happen failure that is analogous on some level to SIGBUS or
> SIGSEGV or something like that, and we should PANIC. Then we could
> claim that dsa_get_address() is no-throw. At least, that was one
> argument I had with myself while investigating that strange Solaris
> shm_open() failure, but ... I lost the argument. It's quite an
> extreme position to take just to support these assertions, which are
> of pretty limited value.
>
> [1]
> https://www.postgresql.org/message-id/20220701232009.jcwxpl45bptaxv5n%40alap3.anarazel.de
Hi,
In the description,
`new shared memory stats system in 15`
It would be clearer to add `release` before `15`.
Cheers
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
@ 2022-07-04 20:54 ` Andres Freund <andres@anarazel.de>
2022-07-04 23:20 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
2 siblings, 1 reply; 11+ messages in thread
From: Andres Freund @ 2022-07-04 20:54 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers <pgsql-hackers@postgresql.org>
Hi,
On 2022-07-04 14:55:43 +1200, Thomas Munro wrote:
> Right, as seen in the build farm at [1]. Also reproducible with something like:
>
> @@ -269,6 +269,14 @@ dsm_impl_posix(dsm_op op, dsm_handle handle, Size
> request_size,
> return false;
> }
>
> + /* XXX random fault injection */
> + if (op == DSM_OP_ATTACH && random() < RAND_MAX / 8)
> + {
> + close(fd);
> + elog(ERROR, "chaos");
> + return false;
> + }
> +
>
> I must have thought that it was easy and practical to write no-throw
> straight-line code and be sure to reach dshash_release_lock(), but I
> concede that it was a bad idea: even dsa_get_address() can throw*, and
> you're often likely to need to call that while accessing dshash
> elements. For example, in lookup_rowtype_tupdesc_internal(), there is
> a sequence dshash_find(), ..., dsa_get_address(), ...,
> dshash_release_lock(), and I must have considered the range of code
> between find and release to be no-throw, but now I know that it is
> not.
Yea - I'd go as far as saying that it's almost never feasible.
> > It's per-backend state at least and just used for assertions. We could remove
> > it. Or stop checking it in places where it could be set wrongly: dshash_find()
> > and dshash_detach() couldn't check anymore, but the rest of the assertions
> > would still be valid afaics?
>
> Yeah, it's all for assertions... let's just remove it. Those
> assertions were useful to me at some stage in development but won't
> hold as well as I thought, at least without widespread PG_FINALLY(),
> which wouldn't be nice.
Hm. I'd be inclined to at least add a few more
Assert(!LWLockHeldByMe[InMode]()) style assertions. E.g. to
dshash_find_or_insert().
> @@ -572,13 +552,8 @@ dshash_release_lock(dshash_table *hash_table, void *entry)
> size_t partition_index = PARTITION_FOR_HASH(item->hash);
>
> Assert(hash_table->control->magic == DSHASH_MAGIC);
> - Assert(hash_table->find_locked);
> - Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition_index),
> - hash_table->find_exclusively_locked
> - ? LW_EXCLUSIVE : LW_SHARED));
> + Assert(LWLockHeldByMe(PARTITION_LOCK(hash_table, partition_index)));
>
> - hash_table->find_locked = false;
> - hash_table->find_exclusively_locked = false;
> LWLockRelease(PARTITION_LOCK(hash_table, partition_index));
> }
This LWLockHeldByMe() doesn't add much - the LWLockRelease() will error out if
we don't hold the lock.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
2022-07-04 20:54 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
@ 2022-07-04 23:20 ` Thomas Munro <thomas.munro@gmail.com>
2022-07-04 23:25 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 11+ messages in thread
From: Thomas Munro @ 2022-07-04 23:20 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers <pgsql-hackers@postgresql.org>
On Tue, Jul 5, 2022 at 8:54 AM Andres Freund <andres@anarazel.de> wrote:
> On 2022-07-04 14:55:43 +1200, Thomas Munro wrote:
> > > It's per-backend state at least and just used for assertions. We could remove
> > > it. Or stop checking it in places where it could be set wrongly: dshash_find()
> > > and dshash_detach() couldn't check anymore, but the rest of the assertions
> > > would still be valid afaics?
> >
> > Yeah, it's all for assertions... let's just remove it. Those
> > assertions were useful to me at some stage in development but won't
> > hold as well as I thought, at least without widespread PG_FINALLY(),
> > which wouldn't be nice.
>
> Hm. I'd be inclined to at least add a few more
> Assert(!LWLockHeldByMe[InMode]()) style assertions. E.g. to
> dshash_find_or_insert().
Yeah, I was wondering about that, but it needs to check the whole 128
element lock array. Hmm, yeah that seems OK for assertion builds.
Since there were 6 places with I-hold-no-lock assertions, I shoved the
loop into a function so I could do:
- Assert(!status->hash_table->find_locked);
+ assert_no_lock_held_by_me(hash_table);
> > + Assert(LWLockHeldByMe(PARTITION_LOCK(hash_table, partition_index)));
> >
> > - hash_table->find_locked = false;
> > - hash_table->find_exclusively_locked = false;
> > LWLockRelease(PARTITION_LOCK(hash_table, partition_index));
> This LWLockHeldByMe() doesn't add much - the LWLockRelease() will error out if
> we don't hold the lock.
Duh. Removed.
Attachments:
v2-0001-Fix-lock-assertions-in-dshash.c.patch (text/x-patch, 7.9K) download
From da650f5dc0dda2155d088f56e2c062f7d10dec44 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 4 Jul 2022 11:56:02 +1200
Subject: [PATCH v2] Fix lock assertions in dshash.c.
dshash.c maintained flags to track locking, in order to make assertions
that the dshash API was being used correctly. Unfortunately the
interaction with ereport's non-local exits was not thought through
carefully enough and the flag could get out of sync with reality. Since
these flags were primarily used for assertions that no lock was held, we
can get most of the intended effect with a loop over all LWLocks instead
(in assertion builds only).
This problem was noted by Tom and Andres while reviewing changes to
support the new shared memory stats system in release 15, the second
user of dshash.c. On closer inspection, even the earlier typcache.c
code was not guaranteed to meet the asserted conditions, now that it's
been highlighted that even dsa_get_address() can throw (albeit in
unlikely circumstances).
Back-patch to 11, where dshash.c arrived.
Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Reported-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
Reviewed-by: Zhihong Yu <zyu@yugabyte.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220311012712.botrpsikaufzteyt@alap3.anarazel.de
---
src/backend/lib/dshash.c | 54 +++++++++++++++-------------------------
1 file changed, 20 insertions(+), 34 deletions(-)
diff --git a/src/backend/lib/dshash.c b/src/backend/lib/dshash.c
index ec454b4d65..90255a5eb9 100644
--- a/src/backend/lib/dshash.c
+++ b/src/backend/lib/dshash.c
@@ -110,8 +110,6 @@ struct dshash_table
dshash_table_control *control; /* Control object in DSM. */
dsa_pointer *buckets; /* Current bucket pointers in DSM. */
size_t size_log2; /* log2(number of buckets) */
- bool find_locked; /* Is any partition lock held by 'find'? */
- bool find_exclusively_locked; /* ... exclusively? */
};
/* Given a pointer to an item, find the entry (user data) it holds. */
@@ -167,6 +165,7 @@ struct dshash_table
BUCKET_INDEX_FOR_HASH_AND_SIZE(hash, \
hash_table->size_log2)])
+static inline void assert_no_lock_held_by_me(dshash_table *hash_table);
static void delete_item(dshash_table *hash_table,
dshash_table_item *item);
static void resize(dshash_table *hash_table, size_t new_size);
@@ -234,9 +233,6 @@ dshash_create(dsa_area *area, const dshash_parameters *params, void *arg)
}
}
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
-
/*
* Set up the initial array of buckets. Our initial size is the same as
* the number of partitions.
@@ -285,8 +281,6 @@ dshash_attach(dsa_area *area, const dshash_parameters *params,
hash_table->params = *params;
hash_table->arg = arg;
hash_table->control = dsa_get_address(area, control);
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
Assert(hash_table->control->magic == DSHASH_MAGIC);
/*
@@ -309,7 +303,7 @@ dshash_attach(dsa_area *area, const dshash_parameters *params,
void
dshash_detach(dshash_table *hash_table)
{
- Assert(!hash_table->find_locked);
+ assert_no_lock_held_by_me(hash_table);
/* The hash table may have been destroyed. Just free local memory. */
pfree(hash_table);
@@ -400,7 +394,7 @@ dshash_find(dshash_table *hash_table, const void *key, bool exclusive)
partition = PARTITION_FOR_HASH(hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ assert_no_lock_held_by_me(hash_table);
LWLockAcquire(PARTITION_LOCK(hash_table, partition),
exclusive ? LW_EXCLUSIVE : LW_SHARED);
@@ -418,8 +412,6 @@ dshash_find(dshash_table *hash_table, const void *key, bool exclusive)
else
{
/* The caller will free the lock by calling dshash_release_lock. */
- hash_table->find_locked = true;
- hash_table->find_exclusively_locked = exclusive;
return ENTRY_FROM_ITEM(item);
}
}
@@ -449,7 +441,7 @@ dshash_find_or_insert(dshash_table *hash_table,
partition = &hash_table->control->partitions[partition_index];
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ assert_no_lock_held_by_me(hash_table);
restart:
LWLockAcquire(PARTITION_LOCK(hash_table, partition_index),
@@ -494,8 +486,6 @@ restart:
}
/* The caller must release the lock with dshash_release_lock. */
- hash_table->find_locked = true;
- hash_table->find_exclusively_locked = true;
return ENTRY_FROM_ITEM(item);
}
@@ -514,7 +504,7 @@ dshash_delete_key(dshash_table *hash_table, const void *key)
bool found;
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ assert_no_lock_held_by_me(hash_table);
hash = hash_key(hash_table, key);
partition = PARTITION_FOR_HASH(hash);
@@ -551,14 +541,10 @@ dshash_delete_entry(dshash_table *hash_table, void *entry)
size_t partition = PARTITION_FOR_HASH(item->hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(hash_table->find_exclusively_locked);
Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition),
LW_EXCLUSIVE));
delete_item(hash_table, item);
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
LWLockRelease(PARTITION_LOCK(hash_table, partition));
}
@@ -572,13 +558,7 @@ dshash_release_lock(dshash_table *hash_table, void *entry)
size_t partition_index = PARTITION_FOR_HASH(item->hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition_index),
- hash_table->find_exclusively_locked
- ? LW_EXCLUSIVE : LW_SHARED));
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
LWLockRelease(PARTITION_LOCK(hash_table, partition_index));
}
@@ -644,7 +624,7 @@ dshash_seq_next(dshash_seq_status *status)
if (status->curpartition == -1)
{
Assert(status->curbucket == 0);
- Assert(!status->hash_table->find_locked);
+ assert_no_lock_held_by_me(status->hash_table);
status->curpartition = 0;
@@ -702,8 +682,6 @@ dshash_seq_next(dshash_seq_status *status)
status->curitem =
dsa_get_address(status->hash_table->area, next_item_pointer);
- status->hash_table->find_locked = true;
- status->hash_table->find_exclusively_locked = status->exclusive;
/*
* The caller may delete the item. Store the next item in case of
@@ -722,9 +700,6 @@ dshash_seq_next(dshash_seq_status *status)
void
dshash_seq_term(dshash_seq_status *status)
{
- status->hash_table->find_locked = false;
- status->hash_table->find_exclusively_locked = false;
-
if (status->curpartition >= 0)
LWLockRelease(PARTITION_LOCK(status->hash_table, status->curpartition));
}
@@ -743,8 +718,6 @@ dshash_delete_current(dshash_seq_status *status)
Assert(status->exclusive);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(hash_table->find_exclusively_locked);
Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition),
LW_EXCLUSIVE));
@@ -762,7 +735,7 @@ dshash_dump(dshash_table *hash_table)
size_t j;
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ assert_no_lock_held_by_me(hash_table);
for (i = 0; i < DSHASH_NUM_PARTITIONS; ++i)
{
@@ -806,6 +779,19 @@ dshash_dump(dshash_table *hash_table)
LWLockRelease(PARTITION_LOCK(hash_table, i));
}
+/*
+ * Assertion used to enforce rule that you can't lock more than one partition
+ * at a time, to avoid risk of deadlock.
+ */
+static inline void
+assert_no_lock_held_by_me(dshash_table *hash_table)
+{
+ int i;
+
+ for (i = 0; i < DSHASH_NUM_PARTITIONS; ++i)
+ Assert(!LWLockHeldByMe(PARTITION_LOCK(hash_table, i)));
+}
+
/*
* Delete a locked item to which we have a pointer.
*/
--
2.36.1
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
2022-07-04 20:54 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 23:20 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
@ 2022-07-04 23:25 ` Andres Freund <andres@anarazel.de>
2022-07-05 03:21 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Andres Freund @ 2022-07-04 23:25 UTC (permalink / raw)
To: Thomas Munro <thomas.munro@gmail.com>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers <pgsql-hackers@postgresql.org>
Hi,
On 2022-07-05 11:20:54 +1200, Thomas Munro wrote:
> On Tue, Jul 5, 2022 at 8:54 AM Andres Freund <andres@anarazel.de> wrote:
> > > Yeah, it's all for assertions... let's just remove it. Those
> > > assertions were useful to me at some stage in development but won't
> > > hold as well as I thought, at least without widespread PG_FINALLY(),
> > > which wouldn't be nice.
> >
> > Hm. I'd be inclined to at least add a few more
> > Assert(!LWLockHeldByMe[InMode]()) style assertions. E.g. to
> > dshash_find_or_insert().
>
> Yeah, I was wondering about that, but it needs to check the whole 128
> element lock array.
I think it'd be ok to just check the current partition - yes, it'd not catch
cases where we're still holding a lock on another partition, but that's imo
not too bad?
> Hmm, yeah that seems OK for assertion builds.
> Since there were 6 places with I-hold-no-lock assertions, I shoved the
> loop into a function so I could do:
>
> - Assert(!status->hash_table->find_locked);
> + assert_no_lock_held_by_me(hash_table);
I am a *bit* wary about the costs of that, even in assert builds - each of the
partition checks in the loop will in turn need to iterate through
held_lwlocks. But I guess we can also just later weaken them if it turns out
to be a problem.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
2022-07-04 20:54 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 23:20 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
2022-07-04 23:25 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
@ 2022-07-05 03:21 ` Thomas Munro <thomas.munro@gmail.com>
2022-07-11 04:50 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
0 siblings, 1 reply; 11+ messages in thread
From: Thomas Munro @ 2022-07-05 03:21 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers <pgsql-hackers@postgresql.org>
On Tue, Jul 5, 2022 at 11:25 AM Andres Freund <andres@anarazel.de> wrote:
> On 2022-07-05 11:20:54 +1200, Thomas Munro wrote:
> > Since there were 6 places with I-hold-no-lock assertions, I shoved the
> > loop into a function so I could do:
> >
> > - Assert(!status->hash_table->find_locked);
> > + assert_no_lock_held_by_me(hash_table);
>
> I am a *bit* wary about the costs of that, even in assert builds - each of the
> partition checks in the loop will in turn need to iterate through
> held_lwlocks. But I guess we can also just later weaken them if it turns out
> to be a problem.
Maybe we should add assertion support for arrays of locks, so we don't
need two levels of loop? Something like the attached?
Attachments:
v3-0001-Fix-lock-assertions-in-dshash.c.patch (text/x-patch, 9.4K) download
From d98874ab15fc568b3da8a6395e0a0ef02855a880 Mon Sep 17 00:00:00 2001
From: Thomas Munro <thomas.munro@gmail.com>
Date: Mon, 4 Jul 2022 11:56:02 +1200
Subject: [PATCH v3] Fix lock assertions in dshash.c.
dshash.c maintained flags to track locking, in order to make assertions
that the dshash API was being used correctly. Unfortunately the
interaction with ereport's non-local exits was not thought through
carefully enough and the flag could get out of sync with reality. Since
these flags were primarily used for assertions that no lock was held, we
can get most of the intended effect by making an assertion about the
partition locks directly.
To avoid N*M complexity while asserting that we hold no partition locks,
add a new debugging-only interface LWLockAnyHeldByMe(), which can check
if any lock in an array is held without the need for a loop within a
loop.
This problem was noted by Tom and Andres while reviewing changes to
support the new shared memory stats system in release 15, the second
user of dshash.c. On closer inspection, even the earlier typcache.c
code was not guaranteed to meet the asserted conditions, now that it's
been highlighted that even dsa_get_address() can throw (albeit in
unlikely circumstances).
Back-patch to 11, where dshash.c arrived.
Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Reported-by: Andres Freund <andres@anarazel.de>
Reviewed-by: Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>
Reviewed-by: Zhihong Yu <zyu@yugabyte.com>
Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20220311012712.botrpsikaufzteyt@alap3.anarazel.de
---
src/backend/lib/dshash.c | 44 +++++++------------------------
src/backend/storage/lmgr/lwlock.c | 26 ++++++++++++++++++
src/include/storage/lwlock.h | 1 +
3 files changed, 37 insertions(+), 34 deletions(-)
diff --git a/src/backend/lib/dshash.c b/src/backend/lib/dshash.c
index ec454b4d65..c5c032a593 100644
--- a/src/backend/lib/dshash.c
+++ b/src/backend/lib/dshash.c
@@ -110,8 +110,6 @@ struct dshash_table
dshash_table_control *control; /* Control object in DSM. */
dsa_pointer *buckets; /* Current bucket pointers in DSM. */
size_t size_log2; /* log2(number of buckets) */
- bool find_locked; /* Is any partition lock held by 'find'? */
- bool find_exclusively_locked; /* ... exclusively? */
};
/* Given a pointer to an item, find the entry (user data) it holds. */
@@ -194,6 +192,10 @@ static inline bool equal_keys(dshash_table *hash_table,
#define PARTITION_LOCK(hash_table, i) \
(&(hash_table)->control->partitions[(i)].lock)
+#define ASSERT_NO_PARTITION_LOCKS_HELD_BY_ME(hash_table) \
+ Assert(!LWLockAnyHeldByMe(&(hash_table)->control->partitions[0].lock, \
+ DSHASH_NUM_PARTITIONS, sizeof(dshash_partition)))
+
/*
* Create a new hash table backed by the given dynamic shared area, with the
* given parameters. The returned object is allocated in backend-local memory
@@ -234,9 +236,6 @@ dshash_create(dsa_area *area, const dshash_parameters *params, void *arg)
}
}
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
-
/*
* Set up the initial array of buckets. Our initial size is the same as
* the number of partitions.
@@ -285,8 +284,6 @@ dshash_attach(dsa_area *area, const dshash_parameters *params,
hash_table->params = *params;
hash_table->arg = arg;
hash_table->control = dsa_get_address(area, control);
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
Assert(hash_table->control->magic == DSHASH_MAGIC);
/*
@@ -309,7 +306,7 @@ dshash_attach(dsa_area *area, const dshash_parameters *params,
void
dshash_detach(dshash_table *hash_table)
{
- Assert(!hash_table->find_locked);
+ ASSERT_NO_PARTITION_LOCKS_HELD_BY_ME(hash_table);
/* The hash table may have been destroyed. Just free local memory. */
pfree(hash_table);
@@ -400,7 +397,7 @@ dshash_find(dshash_table *hash_table, const void *key, bool exclusive)
partition = PARTITION_FOR_HASH(hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ ASSERT_NO_PARTITION_LOCKS_HELD_BY_ME(hash_table);
LWLockAcquire(PARTITION_LOCK(hash_table, partition),
exclusive ? LW_EXCLUSIVE : LW_SHARED);
@@ -418,8 +415,6 @@ dshash_find(dshash_table *hash_table, const void *key, bool exclusive)
else
{
/* The caller will free the lock by calling dshash_release_lock. */
- hash_table->find_locked = true;
- hash_table->find_exclusively_locked = exclusive;
return ENTRY_FROM_ITEM(item);
}
}
@@ -449,7 +444,7 @@ dshash_find_or_insert(dshash_table *hash_table,
partition = &hash_table->control->partitions[partition_index];
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ ASSERT_NO_PARTITION_LOCKS_HELD_BY_ME(hash_table);
restart:
LWLockAcquire(PARTITION_LOCK(hash_table, partition_index),
@@ -494,8 +489,6 @@ restart:
}
/* The caller must release the lock with dshash_release_lock. */
- hash_table->find_locked = true;
- hash_table->find_exclusively_locked = true;
return ENTRY_FROM_ITEM(item);
}
@@ -514,7 +507,7 @@ dshash_delete_key(dshash_table *hash_table, const void *key)
bool found;
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ ASSERT_NO_PARTITION_LOCKS_HELD_BY_ME(hash_table);
hash = hash_key(hash_table, key);
partition = PARTITION_FOR_HASH(hash);
@@ -551,14 +544,10 @@ dshash_delete_entry(dshash_table *hash_table, void *entry)
size_t partition = PARTITION_FOR_HASH(item->hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(hash_table->find_exclusively_locked);
Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition),
LW_EXCLUSIVE));
delete_item(hash_table, item);
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
LWLockRelease(PARTITION_LOCK(hash_table, partition));
}
@@ -572,13 +561,7 @@ dshash_release_lock(dshash_table *hash_table, void *entry)
size_t partition_index = PARTITION_FOR_HASH(item->hash);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition_index),
- hash_table->find_exclusively_locked
- ? LW_EXCLUSIVE : LW_SHARED));
- hash_table->find_locked = false;
- hash_table->find_exclusively_locked = false;
LWLockRelease(PARTITION_LOCK(hash_table, partition_index));
}
@@ -644,7 +627,7 @@ dshash_seq_next(dshash_seq_status *status)
if (status->curpartition == -1)
{
Assert(status->curbucket == 0);
- Assert(!status->hash_table->find_locked);
+ ASSERT_NO_PARTITION_LOCKS_HELD_BY_ME(status->hash_table);
status->curpartition = 0;
@@ -702,8 +685,6 @@ dshash_seq_next(dshash_seq_status *status)
status->curitem =
dsa_get_address(status->hash_table->area, next_item_pointer);
- status->hash_table->find_locked = true;
- status->hash_table->find_exclusively_locked = status->exclusive;
/*
* The caller may delete the item. Store the next item in case of
@@ -722,9 +703,6 @@ dshash_seq_next(dshash_seq_status *status)
void
dshash_seq_term(dshash_seq_status *status)
{
- status->hash_table->find_locked = false;
- status->hash_table->find_exclusively_locked = false;
-
if (status->curpartition >= 0)
LWLockRelease(PARTITION_LOCK(status->hash_table, status->curpartition));
}
@@ -743,8 +721,6 @@ dshash_delete_current(dshash_seq_status *status)
Assert(status->exclusive);
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(hash_table->find_locked);
- Assert(hash_table->find_exclusively_locked);
Assert(LWLockHeldByMeInMode(PARTITION_LOCK(hash_table, partition),
LW_EXCLUSIVE));
@@ -762,7 +738,7 @@ dshash_dump(dshash_table *hash_table)
size_t j;
Assert(hash_table->control->magic == DSHASH_MAGIC);
- Assert(!hash_table->find_locked);
+ ASSERT_NO_PARTITION_LOCKS_HELD_BY_ME(hash_table);
for (i = 0; i < DSHASH_NUM_PARTITIONS; ++i)
{
diff --git a/src/backend/storage/lmgr/lwlock.c b/src/backend/storage/lmgr/lwlock.c
index 8aef909037..1ebb6c7464 100644
--- a/src/backend/storage/lmgr/lwlock.c
+++ b/src/backend/storage/lmgr/lwlock.c
@@ -1925,6 +1925,32 @@ LWLockHeldByMe(LWLock *l)
return false;
}
+/*
+ * LWLockHeldByMe - test whether my process holds any of n locks in any mode
+ *
+ * This is meant as debug support only.
+ */
+bool
+LWLockAnyHeldByMe(LWLock *l, int nlocks, size_t stride)
+{
+ char *held_lock_addr;
+ char *begin;
+ char *end;
+ int i;
+
+ begin = (char *) l;
+ end = begin + nlocks * stride;
+ for (i = 0; i < num_held_lwlocks; i++)
+ {
+ held_lock_addr = (char *) held_lwlocks[i].lock;
+ if (held_lock_addr >= begin &&
+ held_lock_addr < end &&
+ (held_lock_addr - begin) % stride == 0)
+ return true;
+ }
+ return false;
+}
+
/*
* LWLockHeldByMeInMode - test whether my process holds a lock in given mode
*
diff --git a/src/include/storage/lwlock.h b/src/include/storage/lwlock.h
index e8c91139f8..aeddd80987 100644
--- a/src/include/storage/lwlock.h
+++ b/src/include/storage/lwlock.h
@@ -120,6 +120,7 @@ extern void LWLockRelease(LWLock *lock);
extern void LWLockReleaseClearVar(LWLock *lock, uint64 *valptr, uint64 val);
extern void LWLockReleaseAll(void);
extern bool LWLockHeldByMe(LWLock *lock);
+extern bool LWLockAnyHeldByMe(LWLock *lock, int nlocks, size_t stride);
extern bool LWLockHeldByMeInMode(LWLock *lock, LWLockMode mode);
extern int LWLockHeldCount(void);
--
2.36.1
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: pgsql: dshash: Add sequential scan support.
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Re: pgsql: dshash: Add sequential scan support. Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
2022-07-04 20:54 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-04 23:20 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
2022-07-04 23:25 ` Re: pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-07-05 03:21 ` Re: pgsql: dshash: Add sequential scan support. Thomas Munro <thomas.munro@gmail.com>
@ 2022-07-11 04:50 ` Thomas Munro <thomas.munro@gmail.com>
0 siblings, 0 replies; 11+ messages in thread
From: Thomas Munro @ 2022-07-11 04:50 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; Kyotaro HORIGUCHI <horiguchi.kyotaro@lab.ntt.co.jp>; pgsql-hackers <pgsql-hackers@postgresql.org>
On Tue, Jul 5, 2022 at 3:21 PM Thomas Munro <thomas.munro@gmail.com> wrote:
> On Tue, Jul 5, 2022 at 11:25 AM Andres Freund <andres@anarazel.de> wrote:
> > On 2022-07-05 11:20:54 +1200, Thomas Munro wrote:
> > > Since there were 6 places with I-hold-no-lock assertions, I shoved the
> > > loop into a function so I could do:
> > >
> > > - Assert(!status->hash_table->find_locked);
> > > + assert_no_lock_held_by_me(hash_table);
> >
> > I am a *bit* wary about the costs of that, even in assert builds - each of the
> > partition checks in the loop will in turn need to iterate through
> > held_lwlocks. But I guess we can also just later weaken them if it turns out
> > to be a problem.
>
> Maybe we should add assertion support for arrays of locks, so we don't
> need two levels of loop? Something like the attached?
Pushed.
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2022-07-11 04:50 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-03-11 01:02 pgsql: dshash: Add sequential scan support. Andres Freund <andres@anarazel.de>
2022-03-11 01:09 ` Tom Lane <tgl@sss.pgh.pa.us>
2022-03-11 01:27 ` Andres Freund <andres@anarazel.de>
2022-07-04 02:55 ` Thomas Munro <thomas.munro@gmail.com>
2022-07-04 08:31 ` Kyotaro Horiguchi <horikyota.ntt@gmail.com>
2022-07-04 10:46 ` Zhihong Yu <zyu@yugabyte.com>
2022-07-04 20:54 ` Andres Freund <andres@anarazel.de>
2022-07-04 23:20 ` Thomas Munro <thomas.munro@gmail.com>
2022-07-04 23:25 ` Andres Freund <andres@anarazel.de>
2022-07-05 03:21 ` Thomas Munro <thomas.munro@gmail.com>
2022-07-11 04:50 ` Thomas Munro <thomas.munro@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox