public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 1/4] Remove entries that haven't been used for a certain time
28+ messages / 7 participants
[nested] [flat]
* [PATCH 2/4] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
This also can put a hard limit on the number of catcache entries.
---
doc/src/sgml/config.sgml | 38 +++++
src/backend/access/transam/xact.c | 5 +
src/backend/utils/cache/catcache.c | 205 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 63 ++++++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/utils/catcache.h | 33 ++++-
6 files changed, 338 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9b7a7388d5..d0d2374944 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1662,6 +1662,44 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-syscache-memory-target" xreflabel="syscache_memory_target">
+ <term><varname>syscache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_memory_target</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which syscache is expanded
+ without pruning. The value defaults to 0, indicating that pruning is
+ always considered. After exceeding this size, syscache pruning is
+ considered according to
+ <xref linkend="guc-syscache-prune-min-age"/>. If you need to keep
+ certain amount of syscache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-syscache-prune-min-age" xreflabel="syscache_prune_min_age">
+ <term><varname>syscache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_prune_min_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ syscache entry is considered to be removed. -1 indicates that syscache
+ pruning is disabled at all. The value defaults to 600 seconds
+ (<literal>10 minutes</literal>). The syscache entries that are not
+ used for the duration can be removed to prevent syscache bloat. This
+ behavior is suppressed until the size of syscache exceeds
+ <xref linkend="guc-syscache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 92bda87804..ddc433c59e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -734,7 +734,12 @@ void
SetCurrentStatementStartTimestamp(void)
{
if (!IsParallelWorker())
+ {
stmtStartTimestamp = GetCurrentTimestamp();
+
+ /* Set this timestamp as aproximated current time */
+ SetCatCacheClock(stmtStartTimestamp);
+ }
else
Assert(stmtStartTimestamp != 0);
}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..c70ce3b745 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -71,9 +71,38 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int cache_memory_target = 0;
+
+
+/*
+ * GUC for entry limit. Entries are removed when the number of them goes above
+ * cache_entry_limit by the ratio specified by cache_entry_limit_prune_ratio
+ */
+int cache_entry_limit = 0;
+double cache_entry_limit_prune_ratio = 0.8;
+
+/* GUC variable to define the minimum age of entries that will be cosidered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int cache_prune_min_age = 600;
+
+/*
+ * Ignorance interval between two success move of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define LRU_IGNORANCE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Timestamp used for any operation on caches. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -481,6 +510,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -490,6 +520,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_tupsize -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -841,7 +872,9 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_tupsize = 0;
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
* debugging information, if appropriate.
@@ -858,9 +891,133 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initilize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had no access in the same duration.
+ */
+#define PRUNE_BY_AGE 0x01
+#define PRUNE_BY_NUMBER 0x02
+
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ size_t hash_size;
+ int nelems_before = cp->cc_ntup;
+ int ndelelems = 0;
+ int action = 0;
+ dlist_mutable_iter iter;
+
+ if (cache_prune_min_age >= 0)
+ {
+ /* prune only if the size of the hash is above the target */
+
+ hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+ if (hash_size + cp->cc_tupsize > (Size) cache_memory_target * 1024L)
+ action |= PRUNE_BY_AGE;
+ }
+
+ if (cache_entry_limit > 0 && nelems_before >= cache_entry_limit)
+ {
+ ndelelems = nelems_before -
+ (int) (cache_entry_limit * cache_entry_limit_prune_ratio);
+
+ if (ndelelems < 256)
+ ndelelems = 256;
+ if (ndelelems > nelems_before)
+ ndelelems = nelems_before;
+
+ action |= PRUNE_BY_NUMBER;
+ }
+
+ /* Return immediately if no pruning is wanted */
+ if (action == 0)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ bool remove_this = false;
+
+ /* We don't remove referenced entry */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /* check against age */
+ if (action & PRUNE_BY_AGE)
+ {
+ long entry_age;
+ int us;
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < cache_prune_min_age)
+ {
+ /* no longer have a business with further entries, exit */
+ action &= ~PRUNE_BY_AGE;
+ break;
+ }
+
+ /*
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ remove_this = true;
+ }
+
+ /* check against entry number */
+ if (action & PRUNE_BY_NUMBER)
+ {
+ if (nremoved < ndelelems)
+ remove_this = true;
+ else
+ action &= ~PRUNE_BY_NUMBER; /* satisfied */
+ }
+
+ /* exit if finished */
+ if (action == 0)
+ break;
+
+ /* do the work */
+ if (remove_this)
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, nelems_before);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1274,6 +1431,21 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * We don't want too frequent update of LRU. cache_prune_min_age can
+ * be changed on-session so we need to maintan the LRU regardless of
+ * cache_prune_min_age.
+ */
+ if (catcacheclock - ct->lastaccess > LRU_IGNORANCE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1819,11 +1991,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ int tupsize = 0;
/* negative entries have no tuple associated */
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,13 +2016,14 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
ct->tuple.t_data = (HeapTupleHeader)
MAXALIGN(((char *) ct) + sizeof(CatCTup));
+ ct->size = tupsize;
/* copy tuple contents */
memcpy((char *) ct->tuple.t_data,
(const char *) dtp->t_data,
@@ -1876,8 +2051,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
{
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup));
-
+ tupsize = sizeof(CatCTup);
+ ct = (CatCTup *) palloc(tupsize);
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,18 +2073,34 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
+ ct->size = tupsize;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ cache->cc_tupsize += tupsize;
+
+ /* increase refcount so that this survives pruning */
+ ct->refcount++;
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ /* we may still want to prune by entry number, check it */
+ else if (cache_entry_limit > 0 && cache->cc_ntup > cache_entry_limit)
+ CatCacheCleanupOldEntries(cache);
+
+ ct->refcount--;
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8681ada33a..d4df841982 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2204,6 +2205,58 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Cache is not pruned before exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &cache_prune_min_age,
+ 600, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -3368,6 +3421,16 @@ static struct config_real ConfigureNamesReal[] =
NULL, NULL, NULL
},
+ {
+ {"cache_entry_limit_prune_ratio", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit_prune_ratio,
+ 0.8, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0.0, 0.0, 0.0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c7f53470df..108d332f2c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#cache_memory_target = 0kB # in kB
+#cache_prune_min_age = 600s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..3c6842e272 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,9 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
+ int cc_tupsize; /* total amount of catcache tuples */
+ int cc_nfreeent; /* # of entries currently not referenced */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +123,10 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +196,30 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int cache_prune_min_age;
+extern int cache_memory_target;
+extern int cache_entry_limit;
+extern double cache_entry_limit_prune_ratio;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Thu_Feb_07_21_18_45_2019_504)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v13-0003-Syscache-usage-tracking-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 1/4] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
---
doc/src/sgml/config.sgml | 38 ++++++
src/backend/access/transam/xact.c | 5 +
src/backend/utils/cache/catcache.c | 168 ++++++++++++++++++++++++--
src/backend/utils/misc/guc.c | 23 ++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/utils/catcache.h | 28 ++++-
6 files changed, 256 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9b7a7388d5..d0d2374944 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1662,6 +1662,44 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-syscache-memory-target" xreflabel="syscache_memory_target">
+ <term><varname>syscache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_memory_target</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which syscache is expanded
+ without pruning. The value defaults to 0, indicating that pruning is
+ always considered. After exceeding this size, syscache pruning is
+ considered according to
+ <xref linkend="guc-syscache-prune-min-age"/>. If you need to keep
+ certain amount of syscache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-syscache-prune-min-age" xreflabel="syscache_prune_min_age">
+ <term><varname>syscache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_prune_min_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ syscache entry is considered to be removed. -1 indicates that syscache
+ pruning is disabled at all. The value defaults to 600 seconds
+ (<literal>10 minutes</literal>). The syscache entries that are not
+ used for the duration can be removed to prevent syscache bloat. This
+ behavior is suppressed until the size of syscache exceeds
+ <xref linkend="guc-syscache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 92bda87804..ddc433c59e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -734,7 +734,12 @@ void
SetCurrentStatementStartTimestamp(void)
{
if (!IsParallelWorker())
+ {
stmtStartTimestamp = GetCurrentTimestamp();
+
+ /* Set this timestamp as aproximated current time */
+ SetCatCacheClock(stmtStartTimestamp);
+ }
else
Assert(stmtStartTimestamp != 0);
}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..5106ed896a 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -71,9 +71,24 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int cache_memory_target = 0;
+
+/* GUC variable to define the minimum age of entries that will be cosidered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int cache_prune_min_age = 600;
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Timestamp used for any operation on caches. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -490,6 +505,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_tupsize -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -841,6 +857,7 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_tupsize = 0;
/*
* new cache is initialized as far as we can go for now. print some
@@ -858,9 +875,127 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initilize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had no access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int i;
+ int nremoved = 0;
+ size_t hash_size;
+#ifdef CATCACHE_STATS
+ /* These variables are only for debugging purpose */
+ int ntotal = 0;
+ /*
+ * nth element in nentries stores the number of cache entries that have
+ * lived unaccessed for corresponding multiple in ageclass of
+ * cache_prune_min_age. The index of nremoved_entry is the value of the
+ * clock-sweep counter, which takes from 0 up to 2.
+ */
+ double ageclass[] = {0.05, 0.1, 1.0, 2.0, 3.0, 0.0};
+ int nentries[] = {0, 0, 0, 0, 0, 0};
+ int nremoved_entry[3] = {0, 0, 0};
+ int j;
+#endif
+
+ /* Return immediately if no pruning is wanted */
+ if (cache_prune_min_age < 0)
+ return false;
+
+ /*
+ * Return without pruning if the size of the hash is below the target.
+ */
+ hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+ if (hash_size + cp->cc_tupsize < (Size) cache_memory_target * 1024L)
+ return false;
+
+ /* Search the whole hash for entries to remove */
+ for (i = 0; i < cp->cc_nbuckets; i++)
+ {
+ dlist_mutable_iter iter;
+
+ dlist_foreach_modify(iter, &cp->cc_bucket[i])
+ {
+ CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+ long entry_age;
+ int us;
+
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+#ifdef CATCACHE_STATS
+ /* count catcache entries for each age class */
+ ntotal++;
+ for (j = 0 ;
+ ageclass[j] != 0.0 &&
+ entry_age > cache_prune_min_age * ageclass[j] ;
+ j++);
+ if (ageclass[j] == 0.0) j--;
+ nentries[j]++;
+#endif
+
+ /*
+ * Try to remove entries older than cache_prune_min_age seconds.
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (entry_age > cache_prune_min_age)
+ {
+#ifdef CATCACHE_STATS
+ Assert (ct->naccess >= 0 && ct->naccess <= 2);
+ nremoved_entry[ct->naccess]++;
+#endif
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else if (ct->refcount == 0 &&
+ (!ct->c_list || ct->c_list->refcount == 0))
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+ }
+ }
+
+#ifdef CATCACHE_STATS
+ ereport(DEBUG1,
+ (errmsg ("removed %d/%d, age(-%.0fs:%d, -%.0fs:%d, *-%.0fs:%d, -%.0fs:%d, -%.0fs:%d) naccessed(0:%d, 1:%d, 2:%d)",
+ nremoved, ntotal,
+ ageclass[0] * cache_prune_min_age, nentries[0],
+ ageclass[1] * cache_prune_min_age, nentries[1],
+ ageclass[2] * cache_prune_min_age, nentries[2],
+ ageclass[3] * cache_prune_min_age, nentries[3],
+ ageclass[4] * cache_prune_min_age, nentries[4],
+ nremoved_entry[0], nremoved_entry[1], nremoved_entry[2]),
+ errhidestmt(true)));
+#endif
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1274,6 +1409,11 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+ ct->lastaccess = catcacheclock;
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1819,11 +1959,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ int tupsize = 0;
/* negative entries have no tuple associated */
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,13 +1984,14 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
ct->tuple.t_data = (HeapTupleHeader)
MAXALIGN(((char *) ct) + sizeof(CatCTup));
+ ct->size = tupsize;
/* copy tuple contents */
memcpy((char *) ct->tuple.t_data,
(const char *) dtp->t_data,
@@ -1876,8 +2019,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
{
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup));
-
+ tupsize = sizeof(CatCTup);
+ ct = (CatCTup *) palloc(tupsize);
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,19 +2041,30 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ ct->size = tupsize;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ cache->cc_tupsize += tupsize;
+ /* increase refcount so that this survives pruning */
+ ct->refcount++;
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8681ada33a..06c589f725 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2204,6 +2205,28 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Cache is not pruned before exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &cache_prune_min_age,
+ 600, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c7f53470df..108d332f2c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#cache_memory_target = 0kB # in kB
+#cache_prune_min_age = 600s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..5d24809900 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,7 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ int cc_tupsize; /* total amount of catcache tuples */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +121,9 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +193,28 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int cache_prune_min_age;
+extern int cache_memory_target;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Wed_Feb_06_17_37_04_2019_764)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v11-0002-Syscache-usage-tracking-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 3/5] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
This also can put a hard limit on the number of catcache entries.
---
doc/src/sgml/config.sgml | 40 ++++
src/backend/tcop/postgres.c | 13 ++
src/backend/utils/cache/catcache.c | 283 +++++++++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 11 +
src/backend/utils/misc/guc.c | 43 ++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/miscadmin.h | 1 +
src/include/utils/catcache.h | 50 ++++-
src/include/utils/timeout.h | 1 +
10 files changed, 436 insertions(+), 9 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 07b847a8e9..4749ad61a9 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1661,6 +1661,46 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The catalog cache entries that are not used for
+ the duration can be removed to prevent it from being filled up with
+ useless entries. This behaviour is muted until the size of a catalog
+ cache exceeds <xref linkend="guc-catalog-cache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-catalog-cache-memory-target" xreflabel="catalog_cache_memory_target">
+ <term><varname>catalog_cache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_memory_target</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which a system catalog cache
+ can expand without pruning in kilobytes. The value defaults to 0,
+ indicating that age-based pruning is always considered. After
+ exceeding this size, catalog cache starts pruning according to
+ <xref linkend="guc-catalog-cache-prune-min-age"/>. If you need to keep
+ certain amount of catalog cache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 36cfd507b2..f192ee2ca6 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2584,6 +2585,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
@@ -3159,6 +3161,14 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (CatcacheClockTimeoutPending)
+ {
+ CatcacheClockTimeoutPending = 0;
+
+ /* Update timetamp then set up the next timeout */
+ UpdateCatCacheClock();
+ }
}
@@ -4021,6 +4031,9 @@ PostgresMain(int argc, char *argv[],
QueryCancelPending = false; /* second to avoid race condition */
stmt_timeout_active = false;
+ /* get sync with the timer state */
+ catcache_clock_timeout_active = false;
+
/* Not reading from the client anymore. */
DoingCommandRead = false;
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..04a60a490a 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -39,6 +39,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -71,9 +72,43 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/* GUC variable to define the minimum age of entries that will be considered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int catalog_cache_prune_min_age = 300;
+
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int catalog_cache_memory_target = 0;
+
+/*
+ * GUC for limit by the number of entries. Entries are removed when the number
+ * of them goes above catalog_cache_entry_limit and leaving newer entries by
+ * the ratio specified by catalog_cache_prune_ratio.
+ */
+int catalog_cache_entry_limit = 0;
+double catalog_cache_prune_ratio = 0.8;
+
+/*
+ * Flag to keep track of whether catcache clock timer is active.
+ */
+bool catcache_clock_timeout_active = false;
+
+/*
+ * Minimum interval between two success move of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define MIN_LRU_UPDATE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock used to record the last accessed time of a catcache record. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -481,6 +516,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -490,6 +526,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_memusage -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -779,6 +816,7 @@ InitCatCache(int id,
MemoryContext oldcxt;
size_t sz;
int i;
+ uint64 base_size;
/*
* nbuckets is the initial number of hash buckets to use in this catcache.
@@ -821,8 +859,12 @@ InitCatCache(int id,
*
* Note: we rely on zeroing to initialize all the dlist headers correctly
*/
+ base_size = MemoryContextGetConsumption(CacheMemoryContext);
sz = sizeof(CatCache) + PG_CACHE_LINE_SIZE;
cp = (CatCache *) CACHELINEALIGN(palloc0(sz));
+ cp->cc_head_alloc_size =
+ MemoryContextGetConsumption(CacheMemoryContext) - base_size;
+
cp->cc_bucket = palloc0(nbuckets * sizeof(dlist_head));
/*
@@ -842,6 +884,11 @@ InitCatCache(int id,
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ /* cc_head_alloc_size + consumed size for cc_bucket */
+ cp->cc_memusage =
+ MemoryContextGetConsumption(CacheMemoryContext) - base_size;
+
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
* debugging information, if appropriate.
@@ -858,9 +905,185 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * helper routine for SetCatCacheClock and UpdateCatCacheClockTimer.
+ *
+ * We need to maintain the catcache clock during a long query.
+ */
+void
+SetupCatCacheClockTimer(void)
+{
+ long delay;
+
+ /* stop timer if not needed */
+ if (catalog_cache_prune_min_age == 0)
+ {
+ catcache_clock_timeout_active = false;
+ return;
+ }
+
+ /* One 10th of the variable, in milliseconds */
+ delay = catalog_cache_prune_min_age * 1000/10;
+
+ /* Lower limit is 1 second */
+ if (delay < 1000)
+ delay = 1000;
+
+ enable_timeout_after(CATCACHE_CLOCK_TIMEOUT, delay);
+
+ catcache_clock_timeout_active = true;
+}
+
+/*
+ * Update catcacheclock: this is intended to be called from
+ * CATCACHE_CLOCK_TIMEOUT. The interval is expected more than 1 second (see
+ * above), so GetCurrentTime() doesn't harm.
+ */
+void
+UpdateCatCacheClock(void)
+{
+ catcacheclock = GetCurrentTimestamp();
+ SetupCatCacheClockTimer();
+}
+
+/*
+ * It may take an unexpectedly long time before the next clock update when
+ * catalog_cache_prune_min_age gets shorter. Disabling the current timer let
+ * the next update happen at the expected interval. We don't necessariry
+ * require this for increase the age but we don't need to avoid to disable
+ * either.
+ */
+void
+assign_catalog_cache_prune_min_age(int newval, void *extra)
+{
+ if (catcache_clock_timeout_active)
+ disable_timeout(CATCACHE_CLOCK_TIMEOUT, false);
+
+ catcache_clock_timeout_active = false;
+}
+
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had less access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ int nelems_before = cp->cc_ntup;
+ int ndelelems = 0;
+ bool prune_by_age = false;
+ bool prune_by_number = false;
+ dlist_mutable_iter iter;
+
+ /* prune only if the size of the hash is above the target */
+ if (catalog_cache_prune_min_age >= 0 &&
+ cp->cc_memusage > (Size) catalog_cache_memory_target * 1024L)
+ prune_by_age = true;
+
+ if (catalog_cache_entry_limit > 0 &&
+ nelems_before >= catalog_cache_entry_limit)
+ {
+ ndelelems = nelems_before -
+ (int) (catalog_cache_entry_limit * catalog_cache_prune_ratio);
+
+ /* an arbitrary lower limit.. */
+ if (ndelelems < 256)
+ ndelelems = 256;
+ if (ndelelems > nelems_before)
+ ndelelems = nelems_before;
+
+ prune_by_number = true;
+ }
+
+ /* Return immediately if no pruning is wanted */
+ if (!prune_by_age && !prune_by_number)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ bool remove_this = false;
+
+ /* We don't remove referenced entry */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /* check against age */
+ if (prune_by_age)
+ {
+ long entry_age;
+ int us;
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ {
+ /* no longer have a business with further entries, exit */
+ prune_by_age = false;
+ break;
+ }
+ /*
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ remove_this = true;
+ }
+
+ /* check against entry number */
+ if (prune_by_number)
+ {
+ if (nremoved < ndelelems)
+ remove_this = true;
+ else
+ prune_by_number = false; /* we're satisfied */
+ }
+
+ /* exit immediately if all finished */
+ if (!prune_by_age && !prune_by_number)
+ break;
+
+ /* do the work */
+ if (remove_this)
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, nelems_before);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -870,6 +1093,7 @@ RehashCatCache(CatCache *cp)
dlist_head *newbucket;
int newnbuckets;
int i;
+ uint64 base_size = MemoryContextGetConsumption(CacheMemoryContext);
elog(DEBUG1, "rehashing catalog cache id %d for %s; %d tups, %d buckets",
cp->id, cp->cc_relname, cp->cc_ntup, cp->cc_nbuckets);
@@ -878,6 +1102,10 @@ RehashCatCache(CatCache *cp)
newnbuckets = cp->cc_nbuckets * 2;
newbucket = (dlist_head *) MemoryContextAllocZero(CacheMemoryContext, newnbuckets * sizeof(dlist_head));
+ /* recalculate memory usage from the first */
+ cp->cc_memusage = cp->cc_head_alloc_size +
+ MemoryContextGetConsumption(CacheMemoryContext) - base_size;
+
/* Move all entries from old hash table to new. */
for (i = 0; i < cp->cc_nbuckets; i++)
{
@@ -890,6 +1118,7 @@ RehashCatCache(CatCache *cp)
dlist_delete(iter.cur);
dlist_push_head(&newbucket[hashIndex], &ct->cache_elem);
+ cp->cc_memusage += ct->size;
}
}
@@ -1274,6 +1503,21 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * We don't want too frequent update of
+ * LRU. catalog_cache_prune_min_age can be changed on-session so we
+ * need to maintain the LRU regardless of catalog_cache_prune_min_age.
+ */
+ if (catcacheclock - ct->lastaccess > MIN_LRU_UPDATE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1709,6 +1953,11 @@ SearchCatCacheList(CatCache *cache,
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
+
+ /*
+ * Don't waste a time by counting the list in catcache memory usage,
+ * since it doesn't live a long life.
+ */
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
@@ -1819,11 +2068,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ uint64 base_size = MemoryContextGetConsumption(CacheMemoryContext);
/* negative entries have no tuple associated */
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,8 +2093,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
@@ -1877,7 +2128,6 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
ct = (CatCTup *) palloc(sizeof(CatCTup));
-
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,18 +2148,36 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ ct->size = MemoryContextGetConsumption(CacheMemoryContext) - base_size;
+ cache->cc_memusage += ct->size;
+
+ /* increase refcount so that this survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ /* we may still want to prune by entry number, check it */
+ else if (catalog_cache_entry_limit > 0 &&
+ cache->cc_ntup > catalog_cache_entry_limit)
+ CatCacheCleanupOldEntries(cache);
+
+ ct->refcount--;
return ct;
}
@@ -1940,7 +2208,7 @@ CatCacheFreeKeys(TupleDesc tupdesc, int nkeys, int *attnos, Datum *keys)
/*
* Helper routine that copies the keys in the srckeys array into the dstkeys
* one, guaranteeing that the datums are fully allocated in the current memory
- * context.
+ * context. Returns allocated memory size.
*/
static void
CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
@@ -1976,7 +2244,6 @@ CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
att->attbyval,
att->attlen);
}
-
}
/*
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index fd51934aaf..0e8b972a29 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -32,6 +32,7 @@ volatile sig_atomic_t QueryCancelPending = false;
volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
+volatile sig_atomic_t CatcacheClockTimeoutPending = false;
volatile sig_atomic_t ConfigReloadPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index a5ee209f91..9eb50e9676 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -72,6 +72,7 @@ static void ShutdownPostgres(int code, Datum arg);
static void StatementTimeoutHandler(void);
static void LockTimeoutHandler(void);
static void IdleInTransactionSessionTimeoutHandler(void);
+static void CatcacheClockTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -628,6 +629,8 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
RegisterTimeout(LOCK_TIMEOUT, LockTimeoutHandler);
RegisterTimeout(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IdleInTransactionSessionTimeoutHandler);
+ RegisterTimeout(CATCACHE_CLOCK_TIMEOUT,
+ CatcacheClockTimeoutHandler);
}
/*
@@ -1238,6 +1241,14 @@ IdleInTransactionSessionTimeoutHandler(void)
SetLatch(MyLatch);
}
+static void
+CatcacheClockTimeoutHandler(void)
+{
+ CatcacheClockTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
+
/*
* Returns true if at least one role is defined in this database cluster.
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 41d477165c..c62d5ad8b8 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2205,6 +2206,38 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Catalog cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, assign_catalog_cache_prune_min_age, NULL
+ },
+
+ {
+ {"catalog_cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Time-based cache pruning starts working after exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &catalog_cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"catalog_cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &catalog_cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -3368,6 +3401,16 @@ static struct config_real ConfigureNamesReal[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_ratio", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Reduce ratio of pruning caused by catalog_cache_entry_limit."),
+ NULL
+ },
+ &catalog_cache_prune_ratio,
+ 0.8, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0.0, 0.0, 0.0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ad6c436f93..aeb5968e75 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_memory_target = 0kB # in kB
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index c9e35003a5..33b800e80f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -82,6 +82,7 @@ extern PGDLLIMPORT volatile sig_atomic_t InterruptPending;
extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t CatcacheClockTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ConfigReloadPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..0a714bf514 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,11 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
+ int cc_head_alloc_size;/* consumed memory to allocate this struct */
+ int cc_memusage; /* memory usage of this catcache (excluding
+ * header part) */
+ int cc_nfreeent; /* # of entries currently not referenced */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +125,10 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +198,45 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+extern int catalog_cache_memory_target;
+extern int catalog_cache_entry_limit;
+extern double catalog_cache_prune_ratio;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * Flag to keep track of whether catcache timestamp timer is active.
+ */
+extern bool catcache_clock_timeout_active;
+
+/* catcache prune time helper functions */
+extern void SetupCatCacheClockTimer(void);
+extern void UpdateCatCacheClock(void);
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record and start
+ * maintenance timer if needed. We keep to update the clock even while pruning
+ * is disable so that we are not confused by bogus clock value.
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+
+ if (!catcache_clock_timeout_active && catalog_cache_prune_min_age > 0)
+ SetupCatCacheClockTimer();
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
+extern void assign_catalog_cache_prune_min_age(int newval, void *extra);
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 9244a2a7b7..b2d97b4f7b 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -31,6 +31,7 @@ typedef enum TimeoutId
STANDBY_TIMEOUT,
STANDBY_LOCK_TIMEOUT,
IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
+ CATCACHE_CLOCK_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
/* Maximum number of timeout reasons */
--
2.16.3
----Next_Part(Wed_Feb_13_15_31_14_2019_467)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v15-0004-Syscache-usage-tracking-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 2/3] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
This also can put a hard limit on the number of catcache entries.
---
doc/src/sgml/config.sgml | 41 ++++
src/backend/tcop/postgres.c | 13 ++
src/backend/utils/cache/catcache.c | 285 +++++++++++++++++++++++++-
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 11 +
src/backend/utils/misc/guc.c | 43 ++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/miscadmin.h | 1 +
src/include/utils/catcache.h | 49 ++++-
src/include/utils/timeout.h | 1 +
10 files changed, 440 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 07b847a8e9..71d784b6fe 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1661,6 +1661,47 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+
+ Specifies the minimum amount of unused time in seconds at which a
+ catalog cache entry is considered to be removed. -1 indicates that
+ this feature is disabled at all. The value defaults to 300 seconds
+ (<literal>5 minutes</literal>). The catalog cache entries that are
+ not used for the duration can be removed to prevent bloat. This
+ behavior is suppressed until the size of a catalog cache exceeds
+ <xref linkend="guc-catalog-cache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-catalog-cache-memory-target" xreflabel="catalog_cache_memory_target">
+ <term><varname>catalog_cache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_memory_target</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which syscache is expanded
+ without pruning in kilobytes. The value defaults to 0, indicating that
+ pruning is always considered. After exceeding this size, catalog cache
+ pruning is considered according to
+ <xref linkend="guc-catalog-cache-prune-min-age"/>. If you need to keep
+ certain amount of catalog cache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 36cfd507b2..f192ee2ca6 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2584,6 +2585,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
@@ -3159,6 +3161,14 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (CatcacheClockTimeoutPending)
+ {
+ CatcacheClockTimeoutPending = 0;
+
+ /* Update timetamp then set up the next timeout */
+ UpdateCatCacheClock();
+ }
}
@@ -4021,6 +4031,9 @@ PostgresMain(int argc, char *argv[],
QueryCancelPending = false; /* second to avoid race condition */
stmt_timeout_active = false;
+ /* get sync with the timer state */
+ catcache_clock_timeout_active = false;
+
/* Not reading from the client anymore. */
DoingCommandRead = false;
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..0195e19976 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -39,6 +39,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -71,9 +72,43 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/* GUC variable to define the minimum age of entries that will be considered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int catalog_cache_prune_min_age = 300;
+
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int catalog_cache_memory_target = 0;
+
+/*
+ * GUC for limit by the number of entries. Entries are removed when the number
+ * of them goes above catalog_cache_entry_limit and leaving newer entries by
+ * the ratio specified by catalog_cache_prune_ratio.
+ */
+int catalog_cache_entry_limit = 0;
+double catalog_cache_prune_ratio = 0.8;
+
+/*
+ * Flag to keep track of whether catcache clock timer is active.
+ */
+bool catcache_clock_timeout_active = false;
+
+/*
+ * Minimum interval between two success move of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define MIN_LRU_UPDATE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock used to record the last accessed time of a catcache record. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -481,6 +516,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -490,6 +526,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_memusage -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -841,7 +878,13 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_memusage =
+ CacheMemoryContext->methods->get_chunk_space(CacheMemoryContext,
+ cp) +
+ CacheMemoryContext->methods->get_chunk_space(CacheMemoryContext,
+ cp->cc_bucket);
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
* debugging information, if appropriate.
@@ -858,9 +901,191 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * helper routine for SetCatCacheClock and UpdateCatCacheClockTimer.
+ *
+ * We need to maintain the catcache clock during a long query.
+ */
+void
+SetupCatCacheClockTimer(void)
+{
+ long delay;
+
+ /* stop timer if not needed */
+ if (catalog_cache_prune_min_age == 0)
+ {
+ catcache_clock_timeout_active = false;
+ return;
+ }
+
+ /* One 10th of the variable, in milliseconds */
+ delay = catalog_cache_prune_min_age * 1000/10;
+
+ /* Lower limit is 1 second */
+ if (delay < 1000)
+ delay = 1000;
+
+ enable_timeout_after(CATCACHE_CLOCK_TIMEOUT, delay);
+
+ catcache_clock_timeout_active = true;
+}
+
+/*
+ * Update catcacheclock: this is intended to be called from
+ * CATCACHE_CLOCK_TIMEOUT. The interval is expected more than 1 second (see
+ * above), so GetCurrentTime() doesn't harm.
+ */
+void
+UpdateCatCacheClock(void)
+{
+ catcacheclock = GetCurrentTimestamp();
+ SetupCatCacheClockTimer();
+}
+
+/*
+ * It may take an unexpectedly long time before the next clock update when
+ * catalog_cache_prune_min_age gets shorter. Disabling the current timer let
+ * the next update happen at the expected interval. We don't necessariry
+ * require this for increase the age but we don't need to avoid to disable
+ * either.
+ */
+void
+assign_catalog_cache_prune_min_age(int newval, void *extra)
+{
+ if (catcache_clock_timeout_active)
+ disable_timeout(CATCACHE_CLOCK_TIMEOUT, false);
+
+ catcache_clock_timeout_active = false;
+}
+
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had less access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ size_t hash_size;
+ int nelems_before = cp->cc_ntup;
+ int ndelelems = 0;
+ bool prune_by_age = false;
+ bool prune_by_number = false;
+ dlist_mutable_iter iter;
+
+ if (catalog_cache_prune_min_age >= 0)
+ {
+ /* prune only if the size of the hash is above the target */
+
+ hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+ if (hash_size + cp->cc_memusage >
+ (Size) catalog_cache_memory_target * 1024L)
+ prune_by_age = true;
+ }
+
+ if (catalog_cache_entry_limit > 0 &&
+ nelems_before >= catalog_cache_entry_limit)
+ {
+ ndelelems = nelems_before -
+ (int) (catalog_cache_entry_limit * catalog_cache_prune_ratio);
+
+ /* an arbitrary lower limit.. */
+ if (ndelelems < 256)
+ ndelelems = 256;
+ if (ndelelems > nelems_before)
+ ndelelems = nelems_before;
+
+ prune_by_number = true;
+ }
+
+ /* Return immediately if no pruning is wanted */
+ if (!prune_by_age && !prune_by_number)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ bool remove_this = false;
+
+ /* We don't remove referenced entry */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /* check against age */
+ if (prune_by_age)
+ {
+ long entry_age;
+ int us;
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ {
+ /* no longer have a business with further entries, exit */
+ prune_by_age = false;
+ break;
+ }
+ /*
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ remove_this = true;
+ }
+
+ /* check against entry number */
+ if (prune_by_number)
+ {
+ if (nremoved < ndelelems)
+ remove_this = true;
+ else
+ prune_by_number = false; /* we're satisfied */
+ }
+
+ /* exit immediately if all finished */
+ if (!prune_by_age && !prune_by_number)
+ break;
+
+ /* do the work */
+ if (remove_this)
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, nelems_before);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -878,6 +1103,13 @@ RehashCatCache(CatCache *cp)
newnbuckets = cp->cc_nbuckets * 2;
newbucket = (dlist_head *) MemoryContextAllocZero(CacheMemoryContext, newnbuckets * sizeof(dlist_head));
+ /* recalculate memory usage from the first */
+ cp->cc_memusage =
+ CacheMemoryContext->methods->get_chunk_space(CacheMemoryContext,
+ cp) +
+ CacheMemoryContext->methods->get_chunk_space(CacheMemoryContext,
+ newbucket);
+
/* Move all entries from old hash table to new. */
for (i = 0; i < cp->cc_nbuckets; i++)
{
@@ -890,6 +1122,7 @@ RehashCatCache(CatCache *cp)
dlist_delete(iter.cur);
dlist_push_head(&newbucket[hashIndex], &ct->cache_elem);
+ cp->cc_memusage += ct->size;
}
}
@@ -1274,6 +1507,21 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * We don't want too frequent update of
+ * LRU. catalog_cache_prune_min_age can be changed on-session so we
+ * need to maintain the LRU regardless of catalog_cache_prune_min_age.
+ */
+ if (catcacheclock - ct->lastaccess > MIN_LRU_UPDATE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1709,6 +1957,11 @@ SearchCatCacheList(CatCache *cache,
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
+
+ /*
+ * Don't waste a time by counting the list in catcache memory usage,
+ * since it doesn't live a long life.
+ */
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
@@ -1824,6 +2077,7 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,8 +2096,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
@@ -1877,7 +2131,6 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
ct = (CatCTup *) palloc(sizeof(CatCTup));
-
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,18 +2151,38 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ ct->size =
+ CacheMemoryContext->methods->get_chunk_space(CacheMemoryContext,
+ ct);
+ cache->cc_memusage += ct->size;
+
+ /* increase refcount so that this survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ /* we may still want to prune by entry number, check it */
+ else if (catalog_cache_entry_limit > 0 &&
+ cache->cc_ntup > catalog_cache_entry_limit)
+ CatCacheCleanupOldEntries(cache);
+
+ ct->refcount--;
return ct;
}
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index fd51934aaf..0e8b972a29 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -32,6 +32,7 @@ volatile sig_atomic_t QueryCancelPending = false;
volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
+volatile sig_atomic_t CatcacheClockTimeoutPending = false;
volatile sig_atomic_t ConfigReloadPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index a5ee209f91..9eb50e9676 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -72,6 +72,7 @@ static void ShutdownPostgres(int code, Datum arg);
static void StatementTimeoutHandler(void);
static void LockTimeoutHandler(void);
static void IdleInTransactionSessionTimeoutHandler(void);
+static void CatcacheClockTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -628,6 +629,8 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
RegisterTimeout(LOCK_TIMEOUT, LockTimeoutHandler);
RegisterTimeout(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IdleInTransactionSessionTimeoutHandler);
+ RegisterTimeout(CATCACHE_CLOCK_TIMEOUT,
+ CatcacheClockTimeoutHandler);
}
/*
@@ -1238,6 +1241,14 @@ IdleInTransactionSessionTimeoutHandler(void)
SetLatch(MyLatch);
}
+static void
+CatcacheClockTimeoutHandler(void)
+{
+ CatcacheClockTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
+
/*
* Returns true if at least one role is defined in this database cluster.
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 41d477165c..c62d5ad8b8 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2205,6 +2206,38 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Catalog cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, assign_catalog_cache_prune_min_age, NULL
+ },
+
+ {
+ {"catalog_cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Time-based cache pruning starts working after exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &catalog_cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"catalog_cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &catalog_cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -3368,6 +3401,16 @@ static struct config_real ConfigureNamesReal[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_ratio", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Reduce ratio of pruning caused by catalog_cache_entry_limit."),
+ NULL
+ },
+ &catalog_cache_prune_ratio,
+ 0.8, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0.0, 0.0, 0.0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index ad6c436f93..aeb5968e75 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_memory_target = 0kB # in kB
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index c9e35003a5..33b800e80f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -82,6 +82,7 @@ extern PGDLLIMPORT volatile sig_atomic_t InterruptPending;
extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t CatcacheClockTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ConfigReloadPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..0425fc0786 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,10 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
+ int cc_memusage; /* memory usage of this catcache (excluding
+ * header part) */
+ int cc_nfreeent; /* # of entries currently not referenced */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +124,10 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +197,45 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+extern int catalog_cache_memory_target;
+extern int catalog_cache_entry_limit;
+extern double catalog_cache_prune_ratio;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * Flag to keep track of whether catcache timestamp timer is active.
+ */
+extern bool catcache_clock_timeout_active;
+
+/* catcache prune time helper functions */
+extern void SetupCatCacheClockTimer(void);
+extern void UpdateCatCacheClock(void);
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record and start
+ * maintenance timer if needed. We keep to update the clock even while pruning
+ * is disable so that we are not confused by bogus clock value.
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+
+ if (!catcache_clock_timeout_active && catalog_cache_prune_min_age > 0)
+ SetupCatCacheClockTimer();
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
+extern void assign_catalog_cache_prune_min_age(int newval, void *extra);
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 9244a2a7b7..b2d97b4f7b 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -31,6 +31,7 @@ typedef enum TimeoutId
STANDBY_TIMEOUT,
STANDBY_LOCK_TIMEOUT,
IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
+ CATCACHE_CLOCK_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
/* Maximum number of timeout reasons */
--
2.16.3
----Next_Part(Tue_Feb_12_20_36_28_2019_578)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v14-0003-Syscache-usage-tracking-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 1/4] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
---
doc/src/sgml/config.sgml | 38 ++++++
src/backend/access/transam/xact.c | 5 +
src/backend/utils/cache/catcache.c | 166 ++++++++++++++++++++++++--
src/backend/utils/misc/guc.c | 23 ++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/utils/catcache.h | 28 ++++-
6 files changed, 254 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9b7a7388d5..d0d2374944 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1662,6 +1662,44 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-syscache-memory-target" xreflabel="syscache_memory_target">
+ <term><varname>syscache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_memory_target</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which syscache is expanded
+ without pruning. The value defaults to 0, indicating that pruning is
+ always considered. After exceeding this size, syscache pruning is
+ considered according to
+ <xref linkend="guc-syscache-prune-min-age"/>. If you need to keep
+ certain amount of syscache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-syscache-prune-min-age" xreflabel="syscache_prune_min_age">
+ <term><varname>syscache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_prune_min_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ syscache entry is considered to be removed. -1 indicates that syscache
+ pruning is disabled at all. The value defaults to 600 seconds
+ (<literal>10 minutes</literal>). The syscache entries that are not
+ used for the duration can be removed to prevent syscache bloat. This
+ behavior is suppressed until the size of syscache exceeds
+ <xref linkend="guc-syscache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 92bda87804..ddc433c59e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -734,7 +734,12 @@ void
SetCurrentStatementStartTimestamp(void)
{
if (!IsParallelWorker())
+ {
stmtStartTimestamp = GetCurrentTimestamp();
+
+ /* Set this timestamp as aproximated current time */
+ SetCatCacheClock(stmtStartTimestamp);
+ }
else
Assert(stmtStartTimestamp != 0);
}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..2a996d740a 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -71,9 +71,24 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int cache_memory_target = 0;
+
+/* GUC variable to define the minimum age of entries that will be cosidered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int cache_prune_min_age = 600;
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Timestamp used for any operation on caches. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -490,6 +505,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_tupsize -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -841,6 +857,7 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_tupsize = 0;
/*
* new cache is initialized as far as we can go for now. print some
@@ -858,9 +875,129 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initilize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had no access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int i;
+ int nremoved = 0;
+ size_t hash_size;
+#ifdef CATCACHE_STATS
+ /* These variables are only for debugging purpose */
+ int ntotal = 0;
+ /*
+ * nth element in nentries stores the number of cache entries that have
+ * lived unaccessed for corresponding multiple in ageclass of
+ * cache_prune_min_age. The index of nremoved_entry is the value of the
+ * clock-sweep counter, which takes from 0 up to 2.
+ */
+ double ageclass[] = {0.05, 0.1, 1.0, 2.0, 3.0, 0.0};
+ int nentries[] = {0, 0, 0, 0, 0, 0};
+ int nremoved_entry[3] = {0, 0, 0};
+ int j;
+#endif
+
+ /* Return immediately if no pruning is wanted */
+ if (cache_prune_min_age < 0)
+ return false;
+
+ /*
+ * Return without pruning if the size of the hash is below the target.
+ */
+ hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+ if (hash_size + cp->cc_tupsize < (Size) cache_memory_target * 1024L)
+ return false;
+
+ /* Search the whole hash for entries to remove */
+ for (i = 0; i < cp->cc_nbuckets; i++)
+ {
+ dlist_mutable_iter iter;
+
+ dlist_foreach_modify(iter, &cp->cc_bucket[i])
+ {
+ CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+ long entry_age;
+ int us;
+
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+#ifdef CATCACHE_STATS
+ /* count catcache entries for each age class */
+ ntotal++;
+ for (j = 0 ;
+ ageclass[j] != 0.0 &&
+ entry_age > cache_prune_min_age * ageclass[j] ;
+ j++);
+ if (ageclass[j] == 0.0) j--;
+ nentries[j]++;
+#endif
+
+ /*
+ * Try to remove entries older than cache_prune_min_age seconds.
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (entry_age > cache_prune_min_age)
+ {
+#ifdef CATCACHE_STATS
+ Assert (ct->naccess >= 0 && ct->naccess <= 2);
+ nremoved_entry[ct->naccess]++;
+#endif
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ if (!ct->c_list || ct->c_list->refcount == 0)
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+ }
+ }
+ }
+
+#ifdef CATCACHE_STATS
+ ereport(DEBUG1,
+ (errmsg ("removed %d/%d, age(-%.0fs:%d, -%.0fs:%d, *-%.0fs:%d, -%.0fs:%d, -%.0fs:%d) naccessed(0:%d, 1:%d, 2:%d)",
+ nremoved, ntotal,
+ ageclass[0] * cache_prune_min_age, nentries[0],
+ ageclass[1] * cache_prune_min_age, nentries[1],
+ ageclass[2] * cache_prune_min_age, nentries[2],
+ ageclass[3] * cache_prune_min_age, nentries[3],
+ ageclass[4] * cache_prune_min_age, nentries[4],
+ nremoved_entry[0], nremoved_entry[1], nremoved_entry[2]),
+ errhidestmt(true)));
+#endif
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1274,6 +1411,11 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+ ct->lastaccess = catcacheclock;
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1819,11 +1961,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ int tupsize = 0;
/* negative entries have no tuple associated */
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,13 +1986,14 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
ct->tuple.t_data = (HeapTupleHeader)
MAXALIGN(((char *) ct) + sizeof(CatCTup));
+ ct->size = tupsize;
/* copy tuple contents */
memcpy((char *) ct->tuple.t_data,
(const char *) dtp->t_data,
@@ -1876,8 +2021,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
{
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup));
-
+ tupsize = sizeof(CatCTup);
+ ct = (CatCTup *) palloc(tupsize);
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,17 +2043,24 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ ct->size = tupsize;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ cache->cc_tupsize += tupsize;
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
return ct;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8681ada33a..06c589f725 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2204,6 +2205,28 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Cache is not pruned before exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &cache_prune_min_age,
+ 600, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c7f53470df..108d332f2c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#cache_memory_target = 0kB # in kB
+#cache_prune_min_age = 600s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..5d24809900 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,7 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ int cc_tupsize; /* total amount of catcache tuples */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +121,9 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +193,28 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int cache_prune_min_age;
+extern int cache_memory_target;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Wed_Feb_06_14_43_34_2019_896)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v9-0002-Syscache-usage-tracking-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 2/2] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
This also can put a hard limit on the number of catcache entries.
---
doc/src/sgml/config.sgml | 40 +++++
src/backend/tcop/postgres.c | 13 ++
src/backend/utils/cache/catcache.c | 243 ++++++++++++++++++++++++--
src/backend/utils/init/globals.c | 1 +
src/backend/utils/init/postinit.c | 11 ++
src/backend/utils/misc/guc.c | 23 +++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/miscadmin.h | 1 +
src/include/utils/catcache.h | 43 ++++-
src/include/utils/timeout.h | 1 +
10 files changed, 364 insertions(+), 14 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 8bd57f376b..7a93aef659 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1661,6 +1661,46 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The catalog cache entries that are not used for
+ the duration can be removed to prevent it from being filled up with
+ useless entries. This behaviour is muted until the size of a catalog
+ cache exceeds <xref linkend="guc-catalog-cache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-catalog-cache-memory-target" xreflabel="catalog_cache_memory_target">
+ <term><varname>catalog_cache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_memory_target</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which a system catalog cache
+ can expand without pruning in kilobytes. The value defaults to 0,
+ indicating that age-based pruning is always considered. After
+ exceeding this size, catalog cache starts pruning according to
+ <xref linkend="guc-catalog-cache-prune-min-age"/>. If you need to keep
+ certain amount of catalog cache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8b4d94c9a1..d9a54ed37f 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2584,6 +2585,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
@@ -3159,6 +3161,14 @@ ProcessInterrupts(void)
if (ParallelMessagePending)
HandleParallelMessages();
+
+ if (CatcacheClockTimeoutPending)
+ {
+ CatcacheClockTimeoutPending = false;
+
+ /* Update timestamp then set up the next timeout */
+ UpdateCatCacheClock();
+ }
}
@@ -4021,6 +4031,9 @@ PostgresMain(int argc, char *argv[],
QueryCancelPending = false; /* second to avoid race condition */
stmt_timeout_active = false;
+ /* get sync with the timer state */
+ catcache_clock_timeout_active = false;
+
/* Not reading from the client anymore. */
DoingCommandRead = false;
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 78dd5714fa..30ab710aaa 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -39,6 +39,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -61,9 +62,35 @@
#define CACHE_elog(...)
#endif
+/* GUC variable to define the minimum age of entries that will be considered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int catalog_cache_prune_min_age = 300;
+
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int catalog_cache_memory_target = 0;
+
+/*
+ * Flag to keep track of whether catcache clock timer is active.
+ */
+bool catcache_clock_timeout_active = false;
+
+/*
+ * Minimum interval between two success move of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define MIN_LRU_UPDATE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock used to record the last accessed time of a catcache record. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -97,7 +124,7 @@ static CatCTup *CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp,
static void CatCacheFreeKeys(TupleDesc tupdesc, int nkeys, int *attnos,
Datum *keys);
-static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
+static size_t CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
Datum *srckeys, Datum *dstkeys);
@@ -469,6 +496,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -478,6 +506,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_memusage -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -811,7 +840,9 @@ InitCatCache(int id,
*/
sz = sizeof(CatCache) + PG_CACHE_LINE_SIZE;
cp = (CatCache *) CACHELINEALIGN(palloc0(sz));
- cp->cc_bucket = palloc0(nbuckets * sizeof(dlist_head));
+ cp->cc_head_size = sz;
+ sz = nbuckets * sizeof(dlist_head);
+ cp->cc_bucket = palloc0(sz);
/*
* initialize the cache's relation information for the relation
@@ -830,6 +861,9 @@ InitCatCache(int id,
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_memusage = cp->cc_head_size + sz;
+
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
* debugging information, if appropriate.
@@ -846,9 +880,143 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * helper routine for SetCatCacheClock and UpdateCatCacheClockTimer.
+ *
+ * We need to maintain the catcache clock during a long query.
+ */
+void
+SetupCatCacheClockTimer(void)
+{
+ long delay;
+
+ /* stop timer if not needed */
+ if (catalog_cache_prune_min_age == 0)
+ {
+ catcache_clock_timeout_active = false;
+ return;
+ }
+
+ /* One 10th of the variable, in milliseconds */
+ delay = catalog_cache_prune_min_age * 1000/10;
+
+ /* Lower limit is 1 second */
+ if (delay < 1000)
+ delay = 1000;
+
+ enable_timeout_after(CATCACHE_CLOCK_TIMEOUT, delay);
+
+ catcache_clock_timeout_active = true;
+}
+
+/*
+ * Update catcacheclock: this is intended to be called from
+ * CATCACHE_CLOCK_TIMEOUT. The interval is expected more than 1 second (see
+ * above), so GetCurrentTime() doesn't harm.
+ */
+void
+UpdateCatCacheClock(void)
+{
+ catcacheclock = GetCurrentTimestamp();
+ SetupCatCacheClockTimer();
+}
+
+/*
+ * It may take an unexpectedly long time before the next clock update when
+ * catalog_cache_prune_min_age gets shorter. Disabling the current timer let
+ * the next update happen at the expected interval. We don't necessariry
+ * require this for increase the age but we don't need to avoid to disable
+ * either.
+ */
+void
+assign_catalog_cache_prune_min_age(int newval, void *extra)
+{
+ if (catcache_clock_timeout_active)
+ disable_timeout(CATCACHE_CLOCK_TIMEOUT, false);
+
+ catcache_clock_timeout_active = false;
+}
+
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had less access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ dlist_mutable_iter iter;
+
+ /* Return immediately if no pruning is wanted */
+ if (catalog_cache_prune_min_age == 0 ||
+ cp->cc_memusage <= (Size) catalog_cache_memory_target * 1024L)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ long entry_age;
+ int us;
+
+ /* We don't remove referenced entry */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /*
+ * Calculate the duration from the time from the last access to
+ * the "current" time. catcacheclock is updated per-statement
+ * basis and additionaly udpated periodically during a long
+ * running query.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ {
+ /*
+ * no longer have a business with further entries, exit. At least
+ * one removal is enough to prevent rehashing this time.
+ */
+ return nremoved > 0;
+ }
+
+ /*
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ /* remove this entry */
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -858,13 +1026,18 @@ RehashCatCache(CatCache *cp)
dlist_head *newbucket;
int newnbuckets;
int i;
+ size_t sz;
elog(DEBUG1, "rehashing catalog cache id %d for %s; %d tups, %d buckets",
cp->id, cp->cc_relname, cp->cc_ntup, cp->cc_nbuckets);
/* Allocate a new, larger, hash table. */
newnbuckets = cp->cc_nbuckets * 2;
- newbucket = (dlist_head *) MemoryContextAllocZero(CacheMemoryContext, newnbuckets * sizeof(dlist_head));
+ sz = newnbuckets * sizeof(dlist_head);
+ newbucket = (dlist_head *) MemoryContextAllocZero(CacheMemoryContext, sz);
+
+ /* reset memory usage */
+ cp->cc_memusage = cp->cc_head_size + sz;
/* Move all entries from old hash table to new. */
for (i = 0; i < cp->cc_nbuckets; i++)
@@ -878,6 +1051,7 @@ RehashCatCache(CatCache *cp)
dlist_delete(iter.cur);
dlist_push_head(&newbucket[hashIndex], &ct->cache_elem);
+ cp->cc_memusage += ct->size;
}
}
@@ -1260,6 +1434,21 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * We don't want too frequent update of
+ * LRU. catalog_cache_prune_min_age can be changed on-session so we
+ * need to maintain the LRU regardless of catalog_cache_prune_min_age.
+ */
+ if (catcacheclock - ct->lastaccess > MIN_LRU_UPDATE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1695,6 +1884,11 @@ SearchCatCacheList(CatCache *cache,
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
+
+ /*
+ * Don't waste a time by counting the list in catcache memory usage,
+ * since it doesn't live a long life.
+ */
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
@@ -1805,6 +1999,7 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ int tupsize;
/* negative entries have no tuple associated */
if (ntp)
@@ -1828,8 +2023,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
@@ -1862,14 +2057,16 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
{
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup));
+ tupsize = sizeof(CatCTup);
+ ct = (CatCTup *) palloc(tupsize);
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
*/
- CatCacheCopyKeys(cache->cc_tupdesc, cache->cc_nkeys, cache->cc_keyno,
- arguments, ct->keys);
+ tupsize +=
+ CatCacheCopyKeys(cache->cc_tupdesc, cache->cc_nkeys,
+ cache->cc_keyno, arguments, ct->keys);
MemoryContextSwitchTo(oldcxt);
}
@@ -1884,19 +2081,33 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ ct->size = tupsize;
+ cache->cc_memusage += ct->size;
+
+ /* increase refcount so that this survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
@@ -1926,13 +2137,14 @@ CatCacheFreeKeys(TupleDesc tupdesc, int nkeys, int *attnos, Datum *keys)
/*
* Helper routine that copies the keys in the srckeys array into the dstkeys
* one, guaranteeing that the datums are fully allocated in the current memory
- * context.
+ * context. Returns allocated memory size.
*/
-static void
+static size_t
CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
Datum *srckeys, Datum *dstkeys)
{
int i;
+ size_t sz = 0;
/*
* XXX: memory and lookup performance could possibly be improved by
@@ -1961,8 +2173,13 @@ CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
dstkeys[i] = datumCopy(src,
att->attbyval,
att->attlen);
+
+ /* approximate size */
+ if (!att->attbyval)
+ sz += VARHDRSZ + att->attlen;
}
+ return sz;
}
/*
diff --git a/src/backend/utils/init/globals.c b/src/backend/utils/init/globals.c
index fd51934aaf..0e8b972a29 100644
--- a/src/backend/utils/init/globals.c
+++ b/src/backend/utils/init/globals.c
@@ -32,6 +32,7 @@ volatile sig_atomic_t QueryCancelPending = false;
volatile sig_atomic_t ProcDiePending = false;
volatile sig_atomic_t ClientConnectionLost = false;
volatile sig_atomic_t IdleInTransactionSessionTimeoutPending = false;
+volatile sig_atomic_t CatcacheClockTimeoutPending = false;
volatile sig_atomic_t ConfigReloadPending = false;
volatile uint32 InterruptHoldoffCount = 0;
volatile uint32 QueryCancelHoldoffCount = 0;
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index a5ee209f91..9eb50e9676 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -72,6 +72,7 @@ static void ShutdownPostgres(int code, Datum arg);
static void StatementTimeoutHandler(void);
static void LockTimeoutHandler(void);
static void IdleInTransactionSessionTimeoutHandler(void);
+static void CatcacheClockTimeoutHandler(void);
static bool ThereIsAtLeastOneRole(void);
static void process_startup_options(Port *port, bool am_superuser);
static void process_settings(Oid databaseid, Oid roleid);
@@ -628,6 +629,8 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username,
RegisterTimeout(LOCK_TIMEOUT, LockTimeoutHandler);
RegisterTimeout(IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
IdleInTransactionSessionTimeoutHandler);
+ RegisterTimeout(CATCACHE_CLOCK_TIMEOUT,
+ CatcacheClockTimeoutHandler);
}
/*
@@ -1238,6 +1241,14 @@ IdleInTransactionSessionTimeoutHandler(void)
SetLatch(MyLatch);
}
+static void
+CatcacheClockTimeoutHandler(void)
+{
+ CatcacheClockTimeoutPending = true;
+ InterruptPending = true;
+ SetLatch(MyLatch);
+}
+
/*
* Returns true if at least one role is defined in this database cluster.
*/
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 156d147c85..d863c8dec8 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2205,6 +2206,28 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Catalog cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, assign_catalog_cache_prune_min_age, NULL
+ },
+
+ {
+ {"catalog_cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Time-based cache pruning starts working after exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &catalog_cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 194f312096..7c82b0eca7 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_memory_target = 0kB # in kB
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/miscadmin.h b/src/include/miscadmin.h
index c9e35003a5..33b800e80f 100644
--- a/src/include/miscadmin.h
+++ b/src/include/miscadmin.h
@@ -82,6 +82,7 @@ extern PGDLLIMPORT volatile sig_atomic_t InterruptPending;
extern PGDLLIMPORT volatile sig_atomic_t QueryCancelPending;
extern PGDLLIMPORT volatile sig_atomic_t ProcDiePending;
extern PGDLLIMPORT volatile sig_atomic_t IdleInTransactionSessionTimeoutPending;
+extern PGDLLIMPORT volatile sig_atomic_t CatcacheClockTimeoutPending;
extern PGDLLIMPORT volatile sig_atomic_t ConfigReloadPending;
extern PGDLLIMPORT volatile sig_atomic_t ClientConnectionLost;
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..1ae49b4819 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,10 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
+ int cc_head_size; /* memory usage of catcache header */
+ int cc_memusage; /* total memory usage of this catcache */
+ int cc_nfreeent; /* # of entries currently not referenced */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +124,10 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +197,39 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+extern int catalog_cache_memory_target;
+extern int catalog_cache_entry_limit;
+extern double catalog_cache_prune_ratio;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * Flag to keep track of whether catcache timestamp timer is active.
+ */
+extern bool catcache_clock_timeout_active;
+
+/* catcache prune time helper functions */
+extern void SetupCatCacheClockTimer(void);
+extern void UpdateCatCacheClock(void);
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record and start
+ * maintenance timer if needed. We keep to update the clock even while pruning
+ * is disable so that we are not confused by bogus clock value.
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+
+ if (!catcache_clock_timeout_active && catalog_cache_prune_min_age > 0)
+ SetupCatCacheClockTimer();
+}
+
+extern void assign_catalog_cache_prune_min_age(int newval, void *extra);
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
diff --git a/src/include/utils/timeout.h b/src/include/utils/timeout.h
index 9244a2a7b7..b2d97b4f7b 100644
--- a/src/include/utils/timeout.h
+++ b/src/include/utils/timeout.h
@@ -31,6 +31,7 @@ typedef enum TimeoutId
STANDBY_TIMEOUT,
STANDBY_LOCK_TIMEOUT,
IDLE_IN_TRANSACTION_SESSION_TIMEOUT,
+ CATCACHE_CLOCK_TIMEOUT,
/* First user-definable timeout reason */
USER_TIMEOUT,
/* Maximum number of timeout reasons */
--
2.16.3
----Next_Part(Wed_Feb_20_13_14_35_2019_032)----
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 1/4] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
---
doc/src/sgml/config.sgml | 38 ++++++
src/backend/access/transam/xact.c | 5 +
src/backend/utils/cache/catcache.c | 164 ++++++++++++++++++++++++--
src/backend/utils/misc/guc.c | 23 ++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/utils/catcache.h | 28 ++++-
6 files changed, 252 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9b7a7388d5..d0d2374944 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1662,6 +1662,44 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-syscache-memory-target" xreflabel="syscache_memory_target">
+ <term><varname>syscache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_memory_target</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which syscache is expanded
+ without pruning. The value defaults to 0, indicating that pruning is
+ always considered. After exceeding this size, syscache pruning is
+ considered according to
+ <xref linkend="guc-syscache-prune-min-age"/>. If you need to keep
+ certain amount of syscache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-syscache-prune-min-age" xreflabel="syscache_prune_min_age">
+ <term><varname>syscache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_prune_min_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ syscache entry is considered to be removed. -1 indicates that syscache
+ pruning is disabled at all. The value defaults to 600 seconds
+ (<literal>10 minutes</literal>). The syscache entries that are not
+ used for the duration can be removed to prevent syscache bloat. This
+ behavior is suppressed until the size of syscache exceeds
+ <xref linkend="guc-syscache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 92bda87804..ddc433c59e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -734,7 +734,12 @@ void
SetCurrentStatementStartTimestamp(void)
{
if (!IsParallelWorker())
+ {
stmtStartTimestamp = GetCurrentTimestamp();
+
+ /* Set this timestamp as aproximated current time */
+ SetCatCacheClock(stmtStartTimestamp);
+ }
else
Assert(stmtStartTimestamp != 0);
}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..769e173844 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -71,9 +71,24 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int cache_memory_target = 0;
+
+/* GUC variable to define the minimum age of entries that will be cosidered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int cache_prune_min_age = 600;
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Timestamp used for any operation on caches. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -490,6 +505,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_tupsize -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -841,6 +857,7 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_tupsize = 0;
/*
* new cache is initialized as far as we can go for now. print some
@@ -858,9 +875,127 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initilize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had no access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int i;
+ int nremoved = 0;
+ size_t hash_size;
+#ifdef CATCACHE_STATS
+ /* These variables are only for debugging purpose */
+ int ntotal = 0;
+ /*
+ * nth element in nentries stores the number of cache entries that have
+ * lived unaccessed for corresponding multiple in ageclass of
+ * cache_prune_min_age. The index of nremoved_entry is the value of the
+ * clock-sweep counter, which takes from 0 up to 2.
+ */
+ double ageclass[] = {0.05, 0.1, 1.0, 2.0, 3.0, 0.0};
+ int nentries[] = {0, 0, 0, 0, 0, 0};
+ int nremoved_entry[3] = {0, 0, 0};
+ int j;
+#endif
+
+ /* Return immediately if no pruning is wanted */
+ if (cache_prune_min_age < 0)
+ return false;
+
+ /*
+ * Return without pruning if the size of the hash is below the target.
+ */
+ hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+ if (hash_size + cp->cc_tupsize < (Size) cache_memory_target * 1024L)
+ return false;
+
+ /* Search the whole hash for entries to remove */
+ for (i = 0; i < cp->cc_nbuckets; i++)
+ {
+ dlist_mutable_iter iter;
+
+ dlist_foreach_modify(iter, &cp->cc_bucket[i])
+ {
+ CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+ long entry_age;
+ int us;
+
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+#ifdef CATCACHE_STATS
+ /* count catcache entries for each age class */
+ ntotal++;
+ for (j = 0 ;
+ ageclass[j] != 0.0 &&
+ entry_age > cache_prune_min_age * ageclass[j] ;
+ j++);
+ if (ageclass[j] == 0.0) j--;
+ nentries[j]++;
+#endif
+
+ /*
+ * Try to remove entries older than cache_prune_min_age seconds.
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (entry_age > cache_prune_min_age)
+ {
+#ifdef CATCACHE_STATS
+ Assert (ct->naccess >= 0 && ct->naccess <= 2);
+ nremoved_entry[ct->naccess]++;
+#endif
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else if (ct->refcount == 0 &&
+ (!ct->c_list || ct->c_list->refcount == 0))
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+ }
+ }
+
+#ifdef CATCACHE_STATS
+ ereport(DEBUG1,
+ (errmsg ("removed %d/%d, age(-%.0fs:%d, -%.0fs:%d, *-%.0fs:%d, -%.0fs:%d, -%.0fs:%d) naccessed(0:%d, 1:%d, 2:%d)",
+ nremoved, ntotal,
+ ageclass[0] * cache_prune_min_age, nentries[0],
+ ageclass[1] * cache_prune_min_age, nentries[1],
+ ageclass[2] * cache_prune_min_age, nentries[2],
+ ageclass[3] * cache_prune_min_age, nentries[3],
+ ageclass[4] * cache_prune_min_age, nentries[4],
+ nremoved_entry[0], nremoved_entry[1], nremoved_entry[2]),
+ errhidestmt(true)));
+#endif
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1274,6 +1409,11 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+ ct->lastaccess = catcacheclock;
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1819,11 +1959,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ int tupsize = 0;
/* negative entries have no tuple associated */
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,13 +1984,14 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
ct->tuple.t_data = (HeapTupleHeader)
MAXALIGN(((char *) ct) + sizeof(CatCTup));
+ ct->size = tupsize;
/* copy tuple contents */
memcpy((char *) ct->tuple.t_data,
(const char *) dtp->t_data,
@@ -1876,8 +2019,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
{
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup));
-
+ tupsize = sizeof(CatCTup);
+ ct = (CatCTup *) palloc(tupsize);
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,17 +2041,24 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ ct->size = tupsize;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ cache->cc_tupsize += tupsize;
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
return ct;
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8681ada33a..06c589f725 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2204,6 +2205,28 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Cache is not pruned before exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &cache_prune_min_age,
+ 600, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c7f53470df..108d332f2c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#cache_memory_target = 0kB # in kB
+#cache_prune_min_age = 600s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..5d24809900 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,7 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ int cc_tupsize; /* total amount of catcache tuples */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +121,9 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +193,28 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int cache_prune_min_age;
+extern int cache_memory_target;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Wed_Feb_06_15_17_59_2019_321)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v10-0002-Syscache-usage-tracking-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 2/4] Remove entries that haven't been used for a certain time
@ 2018-10-16 04:04 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2018-10-16 04:04 UTC (permalink / raw)
Catcache entries can be left alone for several reasons. It is not
desirable that they eat up memory. With this patch, This adds
consideration of removal of entries that haven't been used for a
certain time before enlarging the hash array.
This also can put a hard limit on the number of catcache entries.
---
doc/src/sgml/config.sgml | 38 ++++++
src/backend/access/transam/xact.c | 5 +
src/backend/utils/cache/catcache.c | 190 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 63 +++++++++
src/backend/utils/misc/postgresql.conf.sample | 2 +
src/include/utils/catcache.h | 32 ++++-
6 files changed, 322 insertions(+), 8 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 9b7a7388d5..d0d2374944 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1662,6 +1662,44 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-syscache-memory-target" xreflabel="syscache_memory_target">
+ <term><varname>syscache_memory_target</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_memory_target</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the maximum amount of memory to which syscache is expanded
+ without pruning. The value defaults to 0, indicating that pruning is
+ always considered. After exceeding this size, syscache pruning is
+ considered according to
+ <xref linkend="guc-syscache-prune-min-age"/>. If you need to keep
+ certain amount of syscache entries with intermittent usage, try
+ increase this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
+ <varlistentry id="guc-syscache-prune-min-age" xreflabel="syscache_prune_min_age">
+ <term><varname>syscache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>syscache_prune_min_age</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ syscache entry is considered to be removed. -1 indicates that syscache
+ pruning is disabled at all. The value defaults to 600 seconds
+ (<literal>10 minutes</literal>). The syscache entries that are not
+ used for the duration can be removed to prevent syscache bloat. This
+ behavior is suppressed until the size of syscache exceeds
+ <xref linkend="guc-syscache-memory-target"/>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/access/transam/xact.c b/src/backend/access/transam/xact.c
index 92bda87804..ddc433c59e 100644
--- a/src/backend/access/transam/xact.c
+++ b/src/backend/access/transam/xact.c
@@ -734,7 +734,12 @@ void
SetCurrentStatementStartTimestamp(void)
{
if (!IsParallelWorker())
+ {
stmtStartTimestamp = GetCurrentTimestamp();
+
+ /* Set this timestamp as aproximated current time */
+ SetCatCacheClock(stmtStartTimestamp);
+ }
else
Assert(stmtStartTimestamp != 0);
}
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 258a1d64cc..0a56390352 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -71,9 +71,32 @@
#define CACHE6_elog(a,b,c,d,e,f,g)
#endif
+/*
+ * GUC variable to define the minimum size of hash to cosider entry eviction.
+ * This variable is shared among various cache mechanisms.
+ */
+int cache_memory_target = 0;
+
+
+/*
+ * GUC for entry limit. Entries are removed when the number of them goes above
+ * cache_entry_limit by the ratio specified by cache_entry_limit_prune_ratio
+ */
+int cache_entry_limit = 0;
+double cache_entry_limit_prune_ratio = 0.8;
+
+/* GUC variable to define the minimum age of entries that will be cosidered to
+ * be evicted in seconds. This variable is shared among various cache
+ * mechanisms.
+ */
+int cache_prune_min_age = 600;
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Timestamp used for any operation on caches. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -481,6 +504,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -490,6 +514,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
CatCacheFreeKeys(cache->cc_tupdesc, cache->cc_nkeys,
cache->cc_keyno, ct->keys);
+ cache->cc_tupsize -= ct->size;
pfree(ct);
--cache->cc_ntup;
@@ -841,7 +866,9 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ cp->cc_tupsize = 0;
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
* debugging information, if appropriate.
@@ -858,9 +885,133 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initilize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries can be left alone for several reasons. We remove them if
+ * they are not accessed for a certain time to prevent catcache from
+ * bloating. The eviction is performed with the similar algorithm with buffer
+ * eviction using access counter. Entries that are accessed several times can
+ * live longer than those that have had no access in the same duration.
+ */
+#define PRUNE_BY_AGE 0x01
+#define PRUNE_BY_NUMBER 0x02
+
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ size_t hash_size;
+ int nelems_before = cp->cc_ntup;
+ int ndelelems = 0;
+ int action = 0;
+ dlist_mutable_iter iter;
+
+ if (cache_prune_min_age >= 0)
+ {
+ /* prune only if the size of the hash is above the target */
+
+ hash_size = cp->cc_nbuckets * sizeof(dlist_head);
+ if (hash_size + cp->cc_tupsize > (Size) cache_memory_target * 1024L)
+ action |= PRUNE_BY_AGE;
+ }
+
+ if (cache_entry_limit > 0 && nelems_before >= cache_entry_limit)
+ {
+ ndelelems = nelems_before -
+ (int) (cache_entry_limit * cache_entry_limit_prune_ratio);
+
+ if (ndelelems < 256)
+ ndelelems = 256;
+ if (ndelelems > nelems_before)
+ ndelelems = nelems_before;
+
+ action |= PRUNE_BY_NUMBER;
+ }
+
+ /* Return immediately if no pruning is wanted */
+ if (action == 0)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ bool remove_this = false;
+
+ /* We don't remove referenced entry */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /* check against age */
+ if (action & PRUNE_BY_AGE)
+ {
+ long entry_age;
+ int us;
+
+ /*
+ * Calculate the duration from the time of the last access to the
+ * "current" time. Since catcacheclock is not advanced within a
+ * transaction, the entries that are accessed within the current
+ * transaction won't be pruned.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < cache_prune_min_age)
+ {
+ /* no longer have a business with further entries, exit */
+ action &= ~PRUNE_BY_AGE;
+ break;
+ }
+
+ /*
+ * Entries that are not accessed after last pruning are removed in
+ * that seconds, and that has been accessed several times are
+ * removed after leaving alone for up to three times of the
+ * duration. We don't try shrink buckets since pruning effectively
+ * caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ remove_this = true;
+ }
+
+ /* check against entry number */
+ if (action & PRUNE_BY_NUMBER)
+ {
+ if (nremoved < ndelelems)
+ remove_this = true;
+ else
+ action &= ~PRUNE_BY_NUMBER; /* satisfied */
+ }
+
+ /* exit if finished */
+ if (action == 0)
+ break;
+
+ /* do the work */
+ if (remove_this)
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, nelems_before);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1274,6 +1425,12 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* Update access information for pruning */
+ if (ct->naccess < 2)
+ ct->naccess++;
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1819,11 +1976,13 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
CatCTup *ct;
HeapTuple dtp;
MemoryContext oldcxt;
+ int tupsize = 0;
/* negative entries have no tuple associated */
if (ntp)
{
int i;
+ int tupsize;
Assert(!negative);
@@ -1842,13 +2001,14 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
/* Allocate memory for CatCTup and the cached tuple in one go */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup) +
- MAXIMUM_ALIGNOF + dtp->t_len);
+ tupsize = sizeof(CatCTup) + MAXIMUM_ALIGNOF + dtp->t_len;
+ ct = (CatCTup *) palloc(tupsize);
ct->tuple.t_len = dtp->t_len;
ct->tuple.t_self = dtp->t_self;
ct->tuple.t_tableOid = dtp->t_tableOid;
ct->tuple.t_data = (HeapTupleHeader)
MAXALIGN(((char *) ct) + sizeof(CatCTup));
+ ct->size = tupsize;
/* copy tuple contents */
memcpy((char *) ct->tuple.t_data,
(const char *) dtp->t_data,
@@ -1876,8 +2036,8 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
{
Assert(negative);
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
- ct = (CatCTup *) palloc(sizeof(CatCTup));
-
+ tupsize = sizeof(CatCTup);
+ ct = (CatCTup *) palloc(tupsize);
/*
* Store keys - they'll point into separately allocated memory if not
* by-value.
@@ -1898,18 +2058,34 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
+ ct->size = tupsize;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ cache->cc_tupsize += tupsize;
+
+ /* increase refcount so that this survives pruning */
+ ct->refcount++;
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try cleanup by removing
+ * infrequently used entries to make a room for the new entry. If it
+ * failed, enlarge the bucket array instead. Quite arbitrarily, we try
+ * this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ /* we may still want to prune by entry number, check it */
+ else if (cache_entry_limit > 0 && cache->cc_ntup > cache_entry_limit)
+ CatCacheCleanupOldEntries(cache);
+
+ ct->refcount--;
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8681ada33a..d4df841982 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2204,6 +2205,58 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"cache_memory_target", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum syscache size to keep."),
+ gettext_noop("Cache is not pruned before exceeding this size."),
+ GUC_UNIT_KB
+ },
+ &cache_memory_target,
+ 0, 0, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the minimum unused duration of cache entries before removal."),
+ gettext_noop("Cache entries that live unused for longer than this seconds are considered to be removed."),
+ GUC_UNIT_S
+ },
+ &cache_prune_min_age,
+ 600, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
+ {
+ {"cache_entry_limit", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit,
+ 0, 0, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
@@ -3368,6 +3421,16 @@ static struct config_real ConfigureNamesReal[] =
NULL, NULL, NULL
},
+ {
+ {"cache_entry_limit_prune_ratio", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("Sets the maximum entries of catcache."),
+ NULL
+ },
+ &cache_entry_limit_prune_ratio,
+ 0.8, 0.0, 1.0,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0.0, 0.0, 0.0, NULL, NULL, NULL
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index c7f53470df..108d332f2c 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,8 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#cache_memory_target = 0kB # in kB
+#cache_prune_min_age = 600s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..973a87c2cf 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,8 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
+ int cc_tupsize; /* total amount of catcache tuples */
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,7 +122,10 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
-
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* approx. timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
+ int size; /* palloc'ed size off this tuple */
/*
* The tuple may also be a member of at most one CatCList. (If a single
* catcache is list-searched with varying numbers of keys, we may have to
@@ -189,6 +195,30 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int cache_prune_min_age;
+extern int cache_memory_target;
+extern int cache_entry_limit;
+extern double cache_entry_limit_prune_ratio;
+
+/* to use as access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/*
+ * SetCatCacheClock - set timestamp for catcache access record
+ */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
+static inline TimestampTz
+GetCatCacheClock(void)
+{
+ return catcacheclock;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Thu_Feb_07_15_24_18_2019_171)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v12-0003-Syscache-usage-tracking-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 2/6] Remove entries that haven't been used for a certain time
@ 2019-03-01 04:32 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2019-03-01 04:32 UTC (permalink / raw)
Catcache entries happen to be left alone for several reasons. It is
not desirable that such useless entries eat up memory. Catcache
pruning feature removes entries that haven't been accessed for a
certain time before enlarging hash array.
---
doc/src/sgml/config.sgml | 19 ++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/cache/catcache.c | 122 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/utils/catcache.h | 18 ++++
6 files changed, 171 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 6d42b7afe7..737a156bb4 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1661,6 +1661,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The entries that are not used for the duration
+ can be removed to prevent catalog cache from bloating with useless
+ entries.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 8b4d94c9a1..02b9ef98aa 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2584,6 +2585,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 78dd5714fa..4386957497 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -39,6 +39,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -61,9 +62,24 @@
#define CACHE_elog(...)
#endif
+/*
+ * GUC variable to define the minimum age of entries that will be considered
+ * to be evicted in seconds. -1 to disable the feature.
+ */
+int catalog_cache_prune_min_age = 300;
+
+/*
+ * Minimum interval between two successive moves of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define MIN_LRU_UPDATE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock for the last accessed time of a catcache entry. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -469,6 +485,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -829,6 +846,7 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
@@ -846,9 +864,83 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries happen to be left unused for a long time for several
+ * reasons. Remove such entries to prevent catcache from bloating. It is based
+ * on the similar algorithm with buffer eviction. Entries that are accessed
+ * several times in a certain period live longer than those that have had less
+ * access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ dlist_mutable_iter iter;
+
+ /* Return immediately if disabled */
+ if (catalog_cache_prune_min_age == 0)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ long entry_age;
+ int us;
+
+ /* Don't remove referenced entries */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /*
+ * Calculate the duration from the time from the last access to
+ * the "current" time. catcacheclock is updated per-statement
+ * basis.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ {
+ /*
+ * We don't have older entries, exit. At least one removal
+ * prevents rehashing this time.
+ */
+ break;
+ }
+
+ /*
+ * Entries that are not accessed after the last pruning are removed in
+ * that seconds, and their lives are prolonged according to how many
+ * times they are accessed up to three times of the duration. We don't
+ * try shrink buckets since pruning effectively caps catcache
+ * expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1260,6 +1352,20 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* prolong life of this entry */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * Don't update LRU too frequently. We need to maintain the LRU even
+ * if pruning is inactive since it can be turned on on-session.
+ */
+ if (catcacheclock - ct->lastaccess > MIN_LRU_UPDATE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1884,19 +1990,29 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ /* increase refcount so that the new entry survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try removing infrequently used
+ * entries to make a room for the new entry. If failed, enlarge the bucket
+ * array instead. Quite arbitrarily, we try this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 156d147c85..3acc86cd07 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -81,6 +81,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2205,6 +2206,17 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered for removal."),
+ gettext_noop("The value of -1 turns off pruning."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index bd6ea65d0c..e9e3acc903 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..a21c53644a 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,7 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,6 +121,9 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
/*
* The tuple may also be a member of at most one CatCList. (If a single
@@ -189,6 +194,19 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+
+/* source clock for access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/* SetCatCacheClock - set catcache timestamp source clodk */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Fri_Mar_01_17_32_45_2019_112)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v18-0003-Asynchronous-update-of-catcache-clock.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH] Remove entries that haven't been used for a certain time
@ 2019-03-01 04:32 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2019-03-01 04:32 UTC (permalink / raw)
Catcache entries happen to be left alone for several reasons. It is
not desirable that such useless entries eat up memory. Catcache
pruning feature removes entries that haven't been accessed for a
certain time before enlarging hash array.
---
doc/src/sgml/config.sgml | 19 ++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/cache/catcache.c | 121 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/utils/catcache.h | 17 ++++
6 files changed, 169 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d383de2512..4231235447 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1677,6 +1677,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The entries that are not used for the duration
+ can be removed to prevent catalog cache from bloating with useless
+ entries.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f9ce3d8f22..acab473d34 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2575,6 +2576,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index d05930bc4c..c4582fe5a3 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -38,6 +38,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -60,9 +61,18 @@
#define CACHE_elog(...)
#endif
+/*
+ * GUC variable to define the minimum age of entries that will be considered
+ * to be evicted in seconds. -1 to disable the feature.
+ */
+int catalog_cache_prune_min_age = 300;
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock for the last accessed time of a catcache entry. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -850,9 +860,99 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries happen to be left unused for a long time for several
+ * reasons. Remove such entries to prevent catcache from bloating. It is based
+ * on the similar algorithm with buffer eviction. Entries that are accessed
+ * several times in a certain period live longer than those that have had less
+ * access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ int i;
+ long oldest_ts = catcacheclock;
+ long age;
+ int us;
+
+ /* Return immediately if disabled */
+ if (catalog_cache_prune_min_age == 0)
+ return false;
+
+ /* Don't scan the hash when we know we don't have prunable entries */
+ TimestampDifference(cp->cc_oldest_ts, catcacheclock, &age, &us);
+ if (age < catalog_cache_prune_min_age)
+ return false;
+
+ /* Scan over the whole hash to find entries to remove */
+ for (i = 0 ; i < cp->cc_nbuckets ; i++)
+ {
+ dlist_mutable_iter iter;
+
+ dlist_foreach_modify(iter, &cp->cc_bucket[i])
+ {
+ CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+
+ /* Don't remove referenced entries */
+ if (ct->refcount == 0 &&
+ (ct->c_list == NULL || ct->c_list->refcount == 0))
+ {
+ /*
+ * Calculate the duration from the time from the last access
+ * to the "current" time. catcacheclock is updated
+ * per-statement basis and additionaly udpated periodically
+ * during a long running query.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &age, &us);
+
+ if (age >= catalog_cache_prune_min_age)
+ {
+ /*
+ * Entries that are not accessed after the last pruning
+ * are removed in that seconds, and their lives are
+ * prolonged according to how many times they are accessed
+ * up to three times of the duration. We don't try shrink
+ * buckets since pruning effectively caps catcache
+ * expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+
+ /* don't update oldest_ts by removed entry */
+ continue;
+ }
+ }
+ }
+
+ /* update oldest timestamp if the entry remains alive */
+ if (ct->lastaccess < oldest_ts)
+ oldest_ts = ct->lastaccess;
+ }
+ }
+
+ cp->cc_oldest_ts = oldest_ts;
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1264,6 +1364,12 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* prolong life of this entry */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ ct->lastaccess = catcacheclock;
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1888,19 +1994,28 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ /* increase refcount so that the new entry survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try removing infrequently used
+ * entries to make a room for the new entry. If failed, enlarge the bucket
+ * array instead. Quite arbitrarily, we try this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index aa564d153a..e624c74bf9 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -82,6 +82,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2202,6 +2203,17 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered for removal."),
+ gettext_noop("The value of -1 turns off pruning."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index cccb5f145a..fa117f0573 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..2f697d5ca4 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,7 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ long cc_oldest_ts;
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,6 +121,8 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* timestamp of the last usage */
/*
* The tuple may also be a member of at most one CatCList. (If a single
@@ -189,6 +193,19 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+
+/* source clock for access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/* SetCatCacheClock - set catcache timestamp source clodk */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Fri_Mar_29_17_24_40_2019_805)--
Content-Type: Application/Octet-Stream
Content-Transfer-Encoding: base64
Content-Disposition: attachment; filename="test_script.tar.gz"
H4sIAFPSnVwAA+1b63Paxhb3V+uvOAb5WtQG9AIaO7h1J2mvZ1y74yb3fkhTRkgLqBGSol01ztjc
v/2e3ZVAflCblIhJo+MZHrtnz/t3dpElRigbUDfxY9be+kykI/V6Hf5u9Dp68T2nLcOy7Z7VMQzd
2NINq9vTt6DzuQwqUkqZkwBsTaLEH6fMX8b32PwXSqyQ/zEJB2wYtOJgvTp4grtde2n+bduQ+dc7
lmXhuNHp4hDo6zXjYfrK81/fgXZKk/bQD9sxSQJFdaMg8CmDPtRqR8ooSojjTkD1QdNbLQPT1YBr
ZXvO1uoDjRM/ZCOtdgDurt7xAL/hZ9VvHCmzokCaDilLtHzkAEzkUORqqHlJFAN1J2TqgD8CcoUs
FHiBgutQ1/HI0W8hmpTzu2gZI/kKzifn75hsPnv2TJh8Zx1zhgERy1psV7c9mNvV4HK4/dz8TSfo
M1MR/0katuhk/ToewX9XtzL8W6ah9wT+dcus8F8G1XcE9IcOnShnFxe/nJ3++qqvapS8BwNUo6H8
cHp+eXHxqt+eRFPSzqPAFykvTl6dvDi9vDvlOcwRL4MPUfJuEJlcxs8nv756edlXM3HteEzfB0WG
s8vXd2ddhOlV/O3A4PM/vj47W8Zg5gw/X7xYxmMpZyc/vDz798uTF33x6VBR3vlB4AQBxBFl44RQ
hQaExGAp6txi2RjHA5cF0GzGY+5XX808B147TCHuJAL1ei5/NnUoI0nTGOi8GYGPDRHUPLpH4EWg
MX9KgDdccFyXUMo3XgN3XriBGM2emwTH0PbIn+0wDYIGmMf/Mvj6kCxVaq6o1FyHUmtFpci/itqn
pyOKFUWwYzl9UupwXcl5kxqN1TQO/r7OcgtFalzNS3MNXq5cmWvQuJqX1mpePrG4ORDUrCt+EhB+
RMUlIyFTWS4UhNJysZCpLBcMQmm5aMhUfnY4PKXGi3jAU8KnQ2IaeZuAhVS7AWhwxRuAh1S7AYhw
xRuAiVRbClSeUP6Ilk3/MKmoFCr+/r/bOdal47Hrf1bPlL//Datrmhb+/kduvfr9Xwbdu/4nL69B
jRJ+2Y05QYRtwnEnZBAnaUgGUz8cOGMCLAJdXm/bwCXDxy7xURIQl8E3MEqiaeEa31d0Ye+JtAT/
a/0fwKPX//P//ximoRv8+l/X6tkV/sug+/ivP70B7Fm6TveqLvAF08P4Nze3/1u23P+tCv9l0D9i
/+88Bfm80irk36Ul+N/M/m/2Oj27w/Fvm90K/2XQP2j/r7rAJ9DD+Jf/n1uXjkf3f3u+/2MLyPb/
ToX/Mug+/svH8ULHH1JHQ0EFy073Kxzvt2cVwh+hZfgv8/xvm/bi/G8b1fW/EunLPP9XHWNdtAT/
5Z7/rfn1P9voCPzbRnX/Xyn0d/C/Z2zy9F/1gHVQEf9xEvEGUPb9/7bVzZ//4I+A9Pj9vz2r2v9L
ofrObfinlMAZguzw8DXzA3j/oU3TaRvh5vkEwTOJ0sBD5BLXH30E7A0w8gMCoTMlNX7TvqbWTy5/
+g/s9EFHkKp+GKdsIHj6oPKpN/pbHB/5JPACEuLot/g1dP4kCe8qfejwZhJSZxoHhOL3poHzmCPG
J/Uj5Xs38sSEhvKjGEWcnh/A3vM9Dt6FtpsbEBaPHPzm8W4leNHiFrti2KE+TDib9vz0/Fg0CHcS
TWMuczskV0z40v5dbfNOsy0cy2zgjomGIjnkrcRa65sG8orxbVVgKnCGJOBOo/04KIyRY7GTYJCl
YYdQ4K5xRqmsIGLnf9D+nStoam/05rO3DW2QfWh8J+zjKnlQ+pkqsbivmosvA+ygPGIeGfkhRkO1
4Tv+cgg1vSYlfC9uMJFR5cvyiAuRM+yJQAJ6Ow5GMQ7YdYPf6L4+Fba13u436DwgcUonmlBwwG8p
FxJzcTvoG9ZcckM/0ka+oA4JcaMEC02UAYTpdEgSiEbAJhg5P8Ftgful5OGa18vzPDvoQqGI1Lpw
b1+6k2UjjHL5oygNvax+C6KkqfdUoLi8XnNd29mm+d+Ty/PT858OcSuj6Wjkuz7B4Xylxst9Lqgh
Nk6+eKYo4hVf8iBLe3cK7NDMA56Zr7ZuFQ/msjVfuM95W9K02ly/cBOGKStIJVeIZUa8Vm7NrFgN
NEoYXKsOOn0M6nAGYuJIkUlizjsiMpKjN08QvykcU8Q1UfmYTSfLIb2fsuP74VTlrj+VRfMG9/FF
h8AwvG1Ae7FIWs3riWTL0bIIczvlt/ZLLQfwR8prhhucLXtAERdbjOAiHmpCaBqw65p6zT2bDeQ7
x9WsNrsW8Jvx5DIZG+4hdZ3ASbRxQmK4BnUA5D0IRsAwiibWKHSTHCVi4kAyFgzYvovRWyDVBWNC
vAiKXUk2h5k80LhBhAE6PceldaUOsmClW/JRKBzGcTxjpdMQmM9wJD8z4d8VprAFtRt5tssc0XI/
+MmOxrKf4xJN0xYdvgn4OmYTTfrEg2w2uGWqyw+LQna2uCXltiAb1rLxLBliwfwweAA6xilXw+cz
c1U3M3SWn2XlEbUOlGDvdViUzD1rPsGzW7wLv3DNfu2+Dlnvw8j7KMUJJGgCSO/IRwq7Mua3JO/a
FG746VM8xCYOsQ/YIXEjhkPEv3z0opZvOlmPxBrMalUIy2tz3nNJ3mHlcjzih2HE+J2FqHQK/Izv
h2N5LgYvjQOfP0ADYRQ28zmpaQ7kh9QhqAt9eG5ZgDozvX9h520wP7h8DoxZ4WQPu92WNZKBzG2c
KdvF/Hw95/qKKqqooooqqqiiiiqqqKKKKqqooooq4vR/BXY1+wBQAAA=
----Next_Part(Fri_Mar_29_17_24_40_2019_805)----
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 2/2] Remove entries that haven't been used for a certain time
@ 2019-03-01 04:32 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2019-03-01 04:32 UTC (permalink / raw)
Catcache entries happen to be left alone for several reasons. It is
not desirable that such useless entries eat up memory. Catcache
pruning feature removes entries that haven't been accessed for a
certain time before enlarging hash array.
---
doc/src/sgml/config.sgml | 19 ++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/cache/catcache.c | 122 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/utils/catcache.h | 18 ++++
6 files changed, 171 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d383de2512..4231235447 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1677,6 +1677,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The entries that are not used for the duration
+ can be removed to prevent catalog cache from bloating with useless
+ entries.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f9ce3d8f22..acab473d34 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2575,6 +2576,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index d05930bc4c..c8ee0c98fb 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -38,6 +38,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -60,9 +61,24 @@
#define CACHE_elog(...)
#endif
+/*
+ * GUC variable to define the minimum age of entries that will be considered
+ * to be evicted in seconds. -1 to disable the feature.
+ */
+int catalog_cache_prune_min_age = 300;
+
+/*
+ * Minimum interval between two successive moves of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define MIN_LRU_UPDATE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock for the last accessed time of a catcache entry. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -473,6 +489,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -833,6 +850,7 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
@@ -850,9 +868,83 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries happen to be left unused for a long time for several
+ * reasons. Remove such entries to prevent catcache from bloating. It is based
+ * on the similar algorithm with buffer eviction. Entries that are accessed
+ * several times in a certain period live longer than those that have had less
+ * access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ dlist_mutable_iter iter;
+
+ /* Return immediately if disabled */
+ if (catalog_cache_prune_min_age == 0)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ long entry_age;
+ int us;
+
+ /* Don't remove referenced entries */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /*
+ * Calculate the duration from the time from the last access to
+ * the "current" time. catcacheclock is updated per-statement
+ * basis.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ {
+ /*
+ * We don't have older entries, exit. At least one removal
+ * prevents rehashing this time.
+ */
+ break;
+ }
+
+ /*
+ * Entries that are not accessed after the last pruning are removed in
+ * that seconds, and their lives are prolonged according to how many
+ * times they are accessed up to three times of the duration. We don't
+ * try shrink buckets since pruning effectively caps catcache
+ * expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1264,6 +1356,20 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* prolong life of this entry */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * Don't update LRU too frequently. We need to maintain the LRU even
+ * if pruning is inactive since it can be turned on on-session.
+ */
+ if (catcacheclock - ct->lastaccess > MIN_LRU_UPDATE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1888,19 +1994,29 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ /* increase refcount so that the new entry survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try removing infrequently used
+ * entries to make a room for the new entry. If failed, enlarge the bucket
+ * array instead. Quite arbitrarily, we try this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index aa564d153a..e624c74bf9 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -82,6 +82,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2202,6 +2203,17 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered for removal."),
+ gettext_noop("The value of -1 turns off pruning."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index cccb5f145a..fa117f0573 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..a21c53644a 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,7 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,6 +121,9 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
/*
* The tuple may also be a member of at most one CatCList. (If a single
@@ -189,6 +194,19 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+
+/* source clock for access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/* SetCatCacheClock - set catcache timestamp source clodk */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Fri_Mar_29_17_24_40_2019_805)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="FullScan-0001-Remove-entries-that-haven-t-been-used-for-a-certain-.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH 1/2] Remove entries that haven't been used for a certain time
@ 2019-03-01 04:32 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2019-03-01 04:32 UTC (permalink / raw)
Catcache entries happen to be left alone for several reasons. It is
not desirable that such useless entries eat up memory. Catcache
pruning feature removes entries that haven't been accessed for a
certain time before enlarging hash array.
---
doc/src/sgml/config.sgml | 19 ++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/cache/catcache.c | 124 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/utils/catcache.h | 18 ++++
6 files changed, 172 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index bc1d0f7bfa..819b252029 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1677,6 +1677,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The entries that are not used for the duration
+ can be removed to prevent catalog cache from bloating with useless
+ entries.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 44a59e1d4f..a0efac86bc 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2577,6 +2578,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index d05930bc4c..e85f2b038c 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -38,6 +38,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -60,9 +61,24 @@
#define CACHE_elog(...)
#endif
+/*
+ * GUC variable to define the minimum age of entries that will be considered
+ * to be evicted in seconds. -1 to disable the feature.
+ */
+int catalog_cache_prune_min_age = 300;
+
+/*
+ * Minimum interval between two successive moves of a cache entry in LRU list,
+ * in microseconds.
+ */
+#define MIN_LRU_UPDATE_INTERVAL 100000 /* 100ms */
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock for the last accessed time of a catcache entry. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -473,6 +489,7 @@ CatCacheRemoveCTup(CatCache *cache, CatCTup *ct)
/* delink from linked list */
dlist_delete(&ct->cache_elem);
+ dlist_delete(&ct->lru_node);
/*
* Free keys when we're dealing with a negative entry, normal entries just
@@ -833,6 +850,7 @@ InitCatCache(int id,
cp->cc_nkeys = nkeys;
for (i = 0; i < nkeys; ++i)
cp->cc_keyno[i] = key[i];
+ dlist_init(&cp->cc_lru_list);
/*
* new cache is initialized as far as we can go for now. print some
@@ -850,9 +868,83 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries happen to be left unused for a long time for several
+ * reasons. Remove such entries to prevent catcache from bloating. It is based
+ * on the similar algorithm with buffer eviction. Entries that are accessed
+ * several times in a certain period live longer than those that have had less
+ * access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ dlist_mutable_iter iter;
+
+ /* Return immediately if disabled */
+ if (catalog_cache_prune_min_age == 0)
+ return false;
+
+ /* Scan over LRU to find entries to remove */
+ dlist_foreach_modify(iter, &cp->cc_lru_list)
+ {
+ CatCTup *ct = dlist_container(CatCTup, lru_node, iter.cur);
+ long entry_age;
+ int us;
+
+ /* Don't remove referenced entries */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /*
+ * Calculate the duration from the time from the last access to
+ * the "current" time. catcacheclock is updated per-statement
+ * basis.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ {
+ /*
+ * We don't have older entries, exit. At least one removal
+ * prevents rehashing this time.
+ */
+ break;
+ }
+
+ /*
+ * Entries that are not accessed after the last pruning are removed in
+ * that seconds, and their lives are prolonged according to how many
+ * times they are accessed up to three times of the duration. We don't
+ * try shrink buckets since pruning effectively caps catcache
+ * expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1262,7 +1354,21 @@ SearchCatCacheInternal(CatCache *cache,
* most frequently accessed elements in any hashbucket will tend to be
* near the front of the hashbucket's list.)
*/
- dlist_move_head(bucket, &ct->cache_elem);
+ /* dlist_move_head(bucket, &ct->cache_elem);*/
+
+ /* prolong life of this entry */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ /*
+ * Don't update LRU too frequently. We need to maintain the LRU even
+ * if pruning is inactive since it can be turned on on-session.
+ */
+ if (catcacheclock - ct->lastaccess > MIN_LRU_UPDATE_INTERVAL)
+ {
+ ct->lastaccess = catcacheclock;
+ dlist_move_tail(&cache->cc_lru_list, &ct->lru_node);
+ }
/*
* If it's a positive entry, bump its refcount and return it. If it's
@@ -1888,19 +1994,29 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
+ dlist_push_tail(&cache->cc_lru_list, &ct->lru_node);
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ /* increase refcount so that the new entry survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try removing infrequently used
+ * entries to make a room for the new entry. If failed, enlarge the bucket
+ * array instead. Quite arbitrarily, we try this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 1766e46037..e671d4428e 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -82,6 +82,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2249,6 +2250,17 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered for removal."),
+ gettext_noop("The value of -1 turns off pruning."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index bbbeb4bb15..d88ec57382 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..a21c53644a 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -61,6 +62,7 @@ typedef struct catcache
slist_node cc_next; /* list link */
ScanKeyData cc_skey[CATCACHE_MAXKEYS]; /* precomputed key info for heap
* scans */
+ dlist_head cc_lru_list;
/*
* Keep these at the end, so that compiling catcache.c with CATCACHE_STATS
@@ -119,6 +121,9 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* timestamp of the last usage */
+ dlist_node lru_node; /* LRU node */
/*
* The tuple may also be a member of at most one CatCList. (If a single
@@ -189,6 +194,19 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+
+/* source clock for access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/* SetCatCacheClock - set catcache timestamp source clodk */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Thu_Apr_04_21_52_55_2019_099)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="0002-Benchmark-extension-for-catcache-pruning-feature.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH] Remove entries that haven't been used for a certain time
@ 2019-03-01 04:32 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2019-03-01 04:32 UTC (permalink / raw)
Catcache entries happen to be left alone for several reasons. It is
not desirable that such useless entries eat up memory. Catcache
pruning feature removes entries that haven't been accessed for a
certain time before enlarging hash array.
---
doc/src/sgml/config.sgml | 19 +++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/cache/catcache.c | 105 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/utils/catcache.h | 16 ++++
6 files changed, 152 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index d383de2512..4231235447 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1677,6 +1677,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The entries that are not used for the duration
+ can be removed to prevent catalog cache from bloating with useless
+ entries.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index f9ce3d8f22..acab473d34 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2575,6 +2576,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index d05930bc4c..52586bd415 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -38,6 +38,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -60,9 +61,18 @@
#define CACHE_elog(...)
#endif
+/*
+ * GUC variable to define the minimum age of entries that will be considered
+ * to be evicted in seconds. -1 to disable the feature.
+ */
+int catalog_cache_prune_min_age = 300;
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock for the last accessed time of a catcache entry. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -850,9 +860,83 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries happen to be left unused for a long time for several
+ * reasons. Remove such entries to prevent catcache from bloating. It is based
+ * on the similar algorithm with buffer eviction. Entries that are accessed
+ * several times in a certain period live longer than those that have had less
+ * access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ int i;
+
+ /* Return immediately if disabled */
+ if (catalog_cache_prune_min_age == 0)
+ return false;
+
+ /* Scan over the whole hash to find entries to remove */
+ for (i = 0 ; i < cp->cc_nbuckets ; i++)
+ {
+ dlist_mutable_iter iter;
+
+ dlist_foreach_modify(iter, &cp->cc_bucket[i])
+ {
+ CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+ long entry_age;
+ int us;
+
+ /* Don't remove referenced entries */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /*
+ * Calculate the duration from the time from the last access to
+ * the "current" time. catcacheclock is updated per-statement
+ * basis and additionaly udpated periodically during a long
+ * running query.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ continue;
+
+ /*
+ * Entries that are not accessed after the last pruning are
+ * removed in that seconds, and their lives are prolonged
+ * according to how many times they are accessed up to three times
+ * of the duration. We don't try shrink buckets since pruning
+ * effectively caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1264,6 +1348,12 @@ SearchCatCacheInternal(CatCache *cache,
*/
dlist_move_head(bucket, &ct->cache_elem);
+ /* prolong life of this entry */
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ ct->lastaccess = catcacheclock;
+
/*
* If it's a positive entry, bump its refcount and return it. If it's
* negative, we can report failure to the caller.
@@ -1888,19 +1978,28 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ /* increase refcount so that the new entry survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try removing infrequently used
+ * entries to make a room for the new entry. If failed, enlarge the bucket
+ * array instead. Quite arbitrarily, we try this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index aa564d153a..e624c74bf9 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -82,6 +82,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2202,6 +2203,17 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered for removal."),
+ gettext_noop("The value of -1 turns off pruning."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index cccb5f145a..fa117f0573 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..2134839ecf 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -119,6 +120,8 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* timestamp of the last usage */
/*
* The tuple may also be a member of at most one CatCList. (If a single
@@ -189,6 +192,19 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+
+/* source clock for access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/* SetCatCacheClock - set catcache timestamp source clodk */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Fri_Mar_29_17_24_40_2019_805)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="FullScan-mod-0001-Remove-entries-that-haven-t-been-used-for-a-certain-.patch"
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH] Remove entries that haven't been used for a certain time
@ 2019-03-01 04:32 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Kyotaro Horiguchi @ 2019-03-01 04:32 UTC (permalink / raw)
Catcache entries happen to be left alone for several reasons. It is
not desirable that such useless entries eat up memory. Catcache
pruning feature removes entries that haven't been accessed for a
certain time before enlarging hash array.
---
doc/src/sgml/config.sgml | 19 +++++
src/backend/tcop/postgres.c | 2 +
src/backend/utils/cache/catcache.c | 103 +++++++++++++++++++++++++-
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/utils/catcache.h | 16 ++++
6 files changed, 150 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index bc1d0f7bfa..819b252029 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1677,6 +1677,25 @@ include_dir 'conf.d'
</listitem>
</varlistentry>
+ <varlistentry id="guc-catalog-cache-prune-min-age" xreflabel="catalog_cache_prune_min_age">
+ <term><varname>catalog_cache_prune_min_age</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>catalog_cache_prune_min_age</varname> configuration
+ parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specifies the minimum amount of unused time in seconds at which a
+ system catalog cache entry is removed. -1 indicates that this feature
+ is disabled at all. The value defaults to 300 seconds (<literal>5
+ minutes</literal>). The entries that are not used for the duration
+ can be removed to prevent catalog cache from bloating with useless
+ entries.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-max-stack-depth" xreflabel="max_stack_depth">
<term><varname>max_stack_depth</varname> (<type>integer</type>)
<indexterm>
diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c
index 44a59e1d4f..a0efac86bc 100644
--- a/src/backend/tcop/postgres.c
+++ b/src/backend/tcop/postgres.c
@@ -71,6 +71,7 @@
#include "tcop/pquery.h"
#include "tcop/tcopprot.h"
#include "tcop/utility.h"
+#include "utils/catcache.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
@@ -2577,6 +2578,7 @@ start_xact_command(void)
* not desired, the timeout has to be disabled explicitly.
*/
enable_statement_timeout();
+ SetCatCacheClock(GetCurrentStatementStartTimestamp());
}
static void
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index d05930bc4c..03c2d8524c 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -38,6 +38,7 @@
#include "utils/rel.h"
#include "utils/resowner_private.h"
#include "utils/syscache.h"
+#include "utils/timeout.h"
/* #define CACHEDEBUG */ /* turns DEBUG elogs on */
@@ -60,9 +61,18 @@
#define CACHE_elog(...)
#endif
+/*
+ * GUC variable to define the minimum age of entries that will be considered
+ * to be evicted in seconds. -1 to disable the feature.
+ */
+int catalog_cache_prune_min_age = 300;
+
/* Cache management header --- pointer is NULL until created */
static CatCacheHeader *CacheHdr = NULL;
+/* Clock for the last accessed time of a catcache entry. */
+TimestampTz catcacheclock = 0;
+
static inline HeapTuple SearchCatCacheInternal(CatCache *cache,
int nkeys,
Datum v1, Datum v2,
@@ -850,9 +860,83 @@ InitCatCache(int id,
*/
MemoryContextSwitchTo(oldcxt);
+ /* initialize catcache reference clock if haven't done yet */
+ if (catcacheclock == 0)
+ catcacheclock = GetCurrentTimestamp();
+
return cp;
}
+/*
+ * CatCacheCleanupOldEntries - Remove infrequently-used entries
+ *
+ * Catcache entries happen to be left unused for a long time for several
+ * reasons. Remove such entries to prevent catcache from bloating. It is based
+ * on the similar algorithm with buffer eviction. Entries that are accessed
+ * several times in a certain period live longer than those that have had less
+ * access in the same duration.
+ */
+static bool
+CatCacheCleanupOldEntries(CatCache *cp)
+{
+ int nremoved = 0;
+ int i;
+
+ /* Return immediately if disabled */
+ if (catalog_cache_prune_min_age == 0)
+ return false;
+
+ /* Scan over the whole hash to find entries to remove */
+ for (i = 0 ; i < cp->cc_nbuckets ; i++)
+ {
+ dlist_mutable_iter iter;
+
+ dlist_foreach_modify(iter, &cp->cc_bucket[i])
+ {
+ CatCTup *ct = dlist_container(CatCTup, cache_elem, iter.cur);
+ long entry_age;
+ int us;
+
+ /* Don't remove referenced entries */
+ if (ct->refcount != 0 ||
+ (ct->c_list && ct->c_list->refcount != 0))
+ continue;
+
+ /*
+ * Calculate the duration from the time from the last access to
+ * the "current" time. catcacheclock is updated per-statement
+ * basis and additionaly udpated periodically during a long
+ * running query.
+ */
+ TimestampDifference(ct->lastaccess, catcacheclock, &entry_age, &us);
+
+ if (entry_age < catalog_cache_prune_min_age)
+ continue;
+
+ /*
+ * Entries that are not accessed after the last pruning are
+ * removed in that seconds, and their lives are prolonged
+ * according to how many times they are accessed up to three times
+ * of the duration. We don't try shrink buckets since pruning
+ * effectively caps catcache expansion in the long term.
+ */
+ if (ct->naccess > 0)
+ ct->naccess--;
+ else
+ {
+ CatCacheRemoveCTup(cp, ct);
+ nremoved++;
+ }
+ }
+ }
+
+ if (nremoved > 0)
+ elog(DEBUG1, "pruning catalog cache id=%d for %s: removed %d / %d",
+ cp->id, cp->cc_relname, nremoved, cp->cc_ntup + nremoved);
+
+ return nremoved > 0;
+}
+
/*
* Enlarge a catcache, doubling the number of buckets.
*/
@@ -1263,6 +1347,10 @@ SearchCatCacheInternal(CatCache *cache,
* near the front of the hashbucket's list.)
*/
dlist_move_head(bucket, &ct->cache_elem);
+ if (ct->naccess < 2)
+ ct->naccess++;
+
+ ct->lastaccess = catcacheclock;
/*
* If it's a positive entry, bump its refcount and return it. If it's
@@ -1888,19 +1976,28 @@ CatalogCacheCreateEntry(CatCache *cache, HeapTuple ntp, Datum *arguments,
ct->dead = false;
ct->negative = negative;
ct->hash_value = hashValue;
+ ct->naccess = 0;
+ ct->lastaccess = catcacheclock;
dlist_push_head(&cache->cc_bucket[hashIndex], &ct->cache_elem);
cache->cc_ntup++;
CacheHdr->ch_ntup++;
+ /* increase refcount so that the new entry survives pruning */
+ ct->refcount++;
+
/*
- * If the hash table has become too full, enlarge the buckets array. Quite
- * arbitrarily, we enlarge when fill factor > 2.
+ * If the hash table has become too full, try removing infrequently used
+ * entries to make a room for the new entry. If failed, enlarge the bucket
+ * array instead. Quite arbitrarily, we try this when fill factor > 2.
*/
- if (cache->cc_ntup > cache->cc_nbuckets * 2)
+ if (cache->cc_ntup > cache->cc_nbuckets * 2 &&
+ !CatCacheCleanupOldEntries(cache))
RehashCatCache(cache);
+ ct->refcount--;
+
return ct;
}
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 1766e46037..e671d4428e 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -82,6 +82,7 @@
#include "tsearch/ts_cache.h"
#include "utils/builtins.h"
#include "utils/bytea.h"
+#include "utils/catcache.h"
#include "utils/guc_tables.h"
#include "utils/float.h"
#include "utils/memutils.h"
@@ -2249,6 +2250,17 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"catalog_cache_prune_min_age", PGC_USERSET, RESOURCES_MEM,
+ gettext_noop("System catalog cache entries that live unused for longer than this seconds are considered for removal."),
+ gettext_noop("The value of -1 turns off pruning."),
+ GUC_UNIT_S
+ },
+ &catalog_cache_prune_min_age,
+ 300, -1, INT_MAX,
+ NULL, NULL, NULL
+ },
+
/*
* We use the hopefully-safely-small value of 100kB as the compiled-in
* default for max_stack_depth. InitializeGUCOptions will increase it if
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index bbbeb4bb15..d88ec57382 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -128,6 +128,7 @@
#work_mem = 4MB # min 64kB
#maintenance_work_mem = 64MB # min 1MB
#autovacuum_work_mem = -1 # min 1MB, or -1 to use maintenance_work_mem
+#catalog_cache_prune_min_age = 300s # -1 disables pruning
#max_stack_depth = 2MB # min 100kB
#shared_memory_type = mmap # the default is the first option
# supported by the operating system:
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index 65d816a583..2134839ecf 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -22,6 +22,7 @@
#include "access/htup.h"
#include "access/skey.h"
+#include "datatype/timestamp.h"
#include "lib/ilist.h"
#include "utils/relcache.h"
@@ -119,6 +120,8 @@ typedef struct catctup
bool dead; /* dead but not yet removed? */
bool negative; /* negative cache entry? */
HeapTupleData tuple; /* tuple management header */
+ int naccess; /* # of access to this entry, up to 2 */
+ TimestampTz lastaccess; /* timestamp of the last usage */
/*
* The tuple may also be a member of at most one CatCList. (If a single
@@ -189,6 +192,19 @@ typedef struct catcacheheader
/* this extern duplicates utils/memutils.h... */
extern PGDLLIMPORT MemoryContext CacheMemoryContext;
+/* for guc.c, not PGDLLPMPORT'ed */
+extern int catalog_cache_prune_min_age;
+
+/* source clock for access timestamp of catcache entries */
+extern TimestampTz catcacheclock;
+
+/* SetCatCacheClock - set catcache timestamp source clodk */
+static inline void
+SetCatCacheClock(TimestampTz ts)
+{
+ catcacheclock = ts;
+}
+
extern void CreateCacheMemoryContext(void);
extern CatCache *InitCatCache(int id, Oid reloid, Oid indexoid,
--
2.16.3
----Next_Part(Fri_Apr_05_09_44_07_2019_159)----
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
@ 2022-01-22 21:20 Stephen Frost <[email protected]>
2022-01-24 19:19 ` Re: CREATEROLE and role ownership hierarchies Andrew Dunstan <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-25 23:17 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
0 siblings, 3 replies; 28+ messages in thread
From: Stephen Frost @ 2022-01-22 21:20 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; +Cc: Joshua Brindle <[email protected]>; Andrew Dunstan <[email protected]>; Shinya Kato <[email protected]>; Bossart, Nathan <[email protected]>; pgsql-hackers; Jeff Davis <[email protected]>
Greetings,
* Mark Dilger ([email protected]) wrote:
> > On Jan 4, 2022, at 12:47 PM, Joshua Brindle <[email protected]> wrote:
> >
> >> I was able to reproduce that using REASSIGN OWNED BY to cause a user to own itself. Is that how you did it, or is there yet another way to get into that state?
> >
> > I did:
> > ALTER ROLE brindle OWNER TO brindle;
>
> Ok, thanks. I have rebased, fixed both REASSIGN OWNED BY and ALTER ROLE .. OWNER TO cases, and added regression coverage for them.
>
> The last patch set to contain significant changes was v2, with v3 just being a rebase. Relative to those sets:
>
> 0001 -- rebased.
> 0002 -- rebased; extend AlterRoleOwner_internal to disallow making a role its own immediate owner.
> 0003 -- rebased; extend AlterRoleOwner_internal to disallow cycles in the role ownership graph.
> 0004 -- rebased.
> 0005 -- new; removes the broken pg_auth_members.grantor field.
> Subject: [PATCH v4 1/5] Add tests of the CREATEROLE attribute.
No particular issue with this one.
> Subject: [PATCH v4 2/5] Add owners to roles
>
> All roles now have owners. By default, roles belong to the role
> that created them, and initdb-time roles are owned by POSTGRES.
... database superuser, not 'POSTGRES'.
> +++ b/src/backend/catalog/aclchk.c
> @@ -5430,6 +5434,57 @@ pg_statistics_object_ownercheck(Oid stat_oid, Oid roleid)
> return has_privs_of_role(roleid, ownerId);
> }
>
> +/*
> + * Ownership check for a role (specified by OID)
> + */
> +bool
> +pg_role_ownercheck(Oid role_oid, Oid roleid)
> +{
> + HeapTuple tuple;
> + Form_pg_authid authform;
> + Oid owner_oid;
> +
> + /* Superusers bypass all permission checking. */
> + if (superuser_arg(roleid))
> + return true;
> +
> + /* Otherwise, look up the owner of the role */
> + tuple = SearchSysCache1(AUTHOID, ObjectIdGetDatum(role_oid));
> + if (!HeapTupleIsValid(tuple))
> + ereport(ERROR,
> + (errcode(ERRCODE_UNDEFINED_OBJECT),
> + errmsg("role with OID %u does not exist",
> + role_oid)));
> + authform = (Form_pg_authid) GETSTRUCT(tuple);
> + owner_oid = authform->rolowner;
> +
> + /*
> + * Roles must necessarily have owners. Even the bootstrap user has an
> + * owner. (It owns itself). Other roles must form a proper tree.
> + */
> + if (!OidIsValid(owner_oid))
> + ereport(ERROR,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("role \"%s\" with OID %u has invalid owner",
> + authform->rolname.data, authform->oid)));
> + if (authform->oid != BOOTSTRAP_SUPERUSERID &&
> + authform->rolowner == authform->oid)
> + ereport(ERROR,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("role \"%s\" with OID %u owns itself",
> + authform->rolname.data, authform->oid)));
> + if (authform->oid == BOOTSTRAP_SUPERUSERID &&
> + authform->rolowner != BOOTSTRAP_SUPERUSERID)
> + ereport(ERROR,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("role \"%s\" with OID %u owned by role with OID %u",
> + authform->rolname.data, authform->oid,
> + authform->rolowner)));
> + ReleaseSysCache(tuple);
> +
> + return (owner_oid == roleid);
> +}
Do we really need all of these checks on every call of this function..?
Also, there isn't much point in including the role OID twice in the last
error message, is there? Unless things have gotten quite odd, it's
goint to be the same value both times as we just proved to ourselves
that it is, in fact, the same value (and that it's not the
BOOTSTRAP_SUPERUSERID).
This function also doesn't actually do any kind of checking to see if
the role ownership forms a proper tree, so it seems a bit odd to have
the comment talking about that here where it's doing other checks.
> +++ b/src/backend/commands/user.c
> @@ -77,6 +79,9 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
> Datum new_record[Natts_pg_authid];
> bool new_record_nulls[Natts_pg_authid];
> Oid roleid;
> + Oid owner_uid;
> + Oid saved_uid;
> + int save_sec_context;
Seems a bit odd to introduce 'uid' into this file, which hasn't got any
such anywhere in it, and I'm not entirely sure that any of these are
actually needed..?
> @@ -108,6 +113,16 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
> DefElem *dvalidUntil = NULL;
> DefElem *dbypassRLS = NULL;
>
> + GetUserIdAndSecContext(&saved_uid, &save_sec_context);
> +
> + /*
> + * Who is supposed to own the new role?
> + */
> + if (stmt->authrole)
> + owner_uid = get_rolespec_oid(stmt->authrole, false);
> + else
> + owner_uid = saved_uid;
> +
> /* The defaults can vary depending on the original statement type */
> switch (stmt->stmt_type)
> {
> @@ -254,6 +269,10 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
> ereport(ERROR,
> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> errmsg("must be superuser to create superusers")));
> + if (!superuser_arg(owner_uid))
> + ereport(ERROR,
> + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> + errmsg("must be superuser to own superusers")));
> }
> else if (isreplication)
> {
So, we're telling a superuser (which is the only way you could get to
this point...) that they aren't allowed to create a superuser role which
is owned by a non-superuser... Why?
> @@ -310,6 +329,19 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
> errmsg("role \"%s\" already exists",
> stmt->role)));
>
> + /*
> + * If the requested authorization is different from the current user,
> + * temporarily set the current user so that the object(s) will be created
> + * with the correct ownership.
> + *
> + * (The setting will be restored at the end of this routine, or in case of
> + * error, transaction abort will clean things up.)
> + */
> + if (saved_uid != owner_uid)
> + SetUserIdAndSecContext(owner_uid,
> + save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
Err, why is this needed? This looks copied from the CreateSchemaCommand
but, unlike with the create schema command, CreateRole doesn't actually
allow sub-commands to be run to create other objects in the way that
CreateSchemaCommand does.
> @@ -478,6 +513,9 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
> */
> table_close(pg_authid_rel, NoLock);
>
> + /* Reset current user and security context */
> + SetUserIdAndSecContext(saved_uid, save_sec_context);
> +
> return roleid;
> }
... ditto with this.
> @@ -1675,3 +1714,110 @@ DelRoleMems(const char *rolename, Oid roleid,
> +static void
> +AlterRoleOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
> +{
> + Form_pg_authid authForm;
> +
> + Assert(tup->t_tableOid == AuthIdRelationId);
> + Assert(RelationGetRelid(rel) == AuthIdRelationId);
> +
> + authForm = (Form_pg_authid) GETSTRUCT(tup);
> +
> + /*
> + * If the new owner is the same as the existing owner, consider the
> + * command to have succeeded. This is for dump restoration purposes.
> + */
> + if (authForm->rolowner != newOwnerId)
> + {
> + /* Otherwise, must be owner of the existing object */
> + if (!pg_role_ownercheck(authForm->oid, GetUserId()))
> + aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_ROLE,
> + NameStr(authForm->rolname));
> +
> + /* Must be able to become new owner */
> + check_is_member_of_role(GetUserId(), newOwnerId);
Feels like we should be saying a bit more about why we check for role
membership vs. has_privs_of_role() here. I'm generally of the opinion
that membership is the right thing to check here, just feel like we
should try to explain more why that's the right thing.
> + /*
> + * must have CREATEROLE rights
> + *
> + * NOTE: This is different from most other alter-owner checks in that
> + * the current user is checked for create privileges instead of the
> + * destination owner. This is consistent with the CREATE case for
> + * roles. Because superusers will always have this right, we need no
> + * special case for them.
> + */
> + if (!have_createrole_privilege())
> + aclcheck_error(ACLCHECK_NO_PRIV, OBJECT_ROLE,
> + NameStr(authForm->rolname));
> +
I would think we'd be trying to get away from the role attribute stuff.
> diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
> + CREATE ROLE RoleId AUTHORIZATION RoleSpec opt_with OptRoleList
> + {
> + CreateRoleStmt *n = makeNode(CreateRoleStmt);
> + n->stmt_type = ROLESTMT_ROLE;
> + n->role = $3;
> + n->authrole = $5;
> + n->options = $7;
> + $$ = (Node *)n;
> + }
> ;
...
> @@ -1218,6 +1229,10 @@ CreateOptRoleElem:
> {
> $$ = makeDefElem("addroleto", (Node *)$3, @1);
> }
> + | OWNER RoleSpec
> + {
> + $$ = makeDefElem("owner", (Node *)$2, @1);
> + }
> ;
Not sure why we'd have both AUTHORIZATION and OWNER for CREATE ROLE..?
We don't do that for other objects.
> diff --git a/src/test/regress/sql/create_role.sql b/src/test/regress/sql/create_role.sql
> @@ -1,6 +1,7 @@
> -- ok, superuser can create users with any set of privileges
> CREATE ROLE regress_role_super SUPERUSER;
> CREATE ROLE regress_role_1 CREATEDB CREATEROLE REPLICATION BYPASSRLS;
> +GRANT CREATE ON DATABASE regression TO regress_role_1;
Seems odd to add this as part of this patch, or am I missing something?
> From 1784a5b51d4dbebf99798b5832d92b0f585feb08 Mon Sep 17 00:00:00 2001
> From: Mark Dilger <[email protected]>
> Date: Tue, 4 Jan 2022 11:42:27 -0800
> Subject: [PATCH v4 3/5] Give role owners control over owned roles
>
> Create a role ownership hierarchy. The previous commit added owners
> to roles. This goes further, making role ownership transitive. If
> role A owns role B, and role B owns role C, then role A can act as
> the owner of role C. Also, roles A and B can perform any action on
> objects belonging to role C that role C could itself perform.
>
> This is a preparatory patch for changing how CREATEROLE works.
This feels odd to have be an independent commit.
> diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c
> index ddd205d656..ef36fad700 100644
> --- a/src/backend/catalog/aclchk.c
> +++ b/src/backend/catalog/aclchk.c
> @@ -5440,61 +5440,20 @@ pg_statistics_object_ownercheck(Oid stat_oid, Oid roleid)
> bool
> pg_role_ownercheck(Oid role_oid, Oid roleid)
> {
> - HeapTuple tuple;
> - Form_pg_authid authform;
> - Oid owner_oid;
> -
> /* Superusers bypass all permission checking. */
> if (superuser_arg(roleid))
> return true;
>
> - /* Otherwise, look up the owner of the role */
> - tuple = SearchSysCache1(AUTHOID, ObjectIdGetDatum(role_oid));
> - if (!HeapTupleIsValid(tuple))
> - ereport(ERROR,
> - (errcode(ERRCODE_UNDEFINED_OBJECT),
> - errmsg("role with OID %u does not exist",
> - role_oid)));
> - authform = (Form_pg_authid) GETSTRUCT(tuple);
> - owner_oid = authform->rolowner;
> -
> - /*
> - * Roles must necessarily have owners. Even the bootstrap user has an
> - * owner. (It owns itself). Other roles must form a proper tree.
> - */
> - if (!OidIsValid(owner_oid))
> - ereport(ERROR,
> - (errcode(ERRCODE_DATA_CORRUPTED),
> - errmsg("role \"%s\" with OID %u has invalid owner",
> - authform->rolname.data, authform->oid)));
> - if (authform->oid != BOOTSTRAP_SUPERUSERID &&
> - authform->rolowner == authform->oid)
> - ereport(ERROR,
> - (errcode(ERRCODE_DATA_CORRUPTED),
> - errmsg("role \"%s\" with OID %u owns itself",
> - authform->rolname.data, authform->oid)));
> - if (authform->oid == BOOTSTRAP_SUPERUSERID &&
> - authform->rolowner != BOOTSTRAP_SUPERUSERID)
> - ereport(ERROR,
> - (errcode(ERRCODE_DATA_CORRUPTED),
> - errmsg("role \"%s\" with OID %u owned by role with OID %u",
> - authform->rolname.data, authform->oid,
> - authform->rolowner)));
> - ReleaseSysCache(tuple);
> -
> - return (owner_oid == roleid);
> + /* Otherwise, check the role ownership hierarchy */
> + return is_owner_of_role_nosuper(roleid, role_oid);
> }
The function being basically entirely rewritten in this patch would be
one reason why it seems an odd split.
> /*
> * Check whether specified role has CREATEROLE privilege (or is a superuser)
> *
> - * Note: roles do not have owners per se; instead we use this test in
> - * places where an ownership-like permissions test is needed for a role.
> - * Be sure to apply it to the role trying to do the operation, not the
> - * role being operated on! Also note that this generally should not be
> - * considered enough privilege if the target role is a superuser.
> - * (We don't handle that consideration here because we want to give a
> - * separate error message for such cases, so the caller has to deal with it.)
> + * Note: In versions prior to PostgreSQL version 15, roles did not have owners
> + * per se; instead we used this test in places where an ownership-like
> + * permissions test was needed for a role.
> */
> bool
> has_createrole_privilege(Oid roleid)
Surely this should be in the prior commit, if the split is kept..
> diff --git a/src/backend/commands/schemacmds.c b/src/backend/commands/schemacmds.c
> @@ -363,7 +363,7 @@ AlterSchemaOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
> /*
> * must have create-schema rights
> *
> - * NOTE: This is different from other alter-owner checks in that the
> + * NOTE: This is different from most other alter-owner checks in that the
> * current user is checked for create privileges instead of the
> * destination owner. This is consistent with the CREATE case for
> * schemas. Because superusers will always have this right, we need
Not a fan of just dropping 'most' in here, doesn't really help someone
understand what is being talked about. I'd suggest adjusting the
comment to talk about alter-owner checks for objects which exist in
schemas, as that's really what is being referred to.
> diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c
> index 14820744bf..11d5dffc90 100644
> --- a/src/backend/commands/user.c
> +++ b/src/backend/commands/user.c
> @@ -724,7 +724,7 @@ AlterRole(ParseState *pstate, AlterRoleStmt *stmt)
> !rolemembers &&
> !validUntil &&
> dpassword &&
> - roleid == GetUserId()))
> + !pg_role_ownercheck(roleid, GetUserId())))
> ereport(ERROR,
> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> errmsg("permission denied")));
> @@ -925,7 +925,8 @@ AlterRoleSet(AlterRoleSetStmt *stmt)
> }
> else
> {
> - if (!have_createrole_privilege() && roleid != GetUserId())
> + if (!have_createrole_privilege() &&
> + !pg_role_ownercheck(roleid, GetUserId()))
> ereport(ERROR,
> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> errmsg("permission denied")));
> @@ -977,11 +978,6 @@ DropRole(DropRoleStmt *stmt)
> pg_auth_members_rel;
> ListCell *item;
>
> - if (!have_createrole_privilege())
> - ereport(ERROR,
> - (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> - errmsg("permission denied to drop role")));
> -
> /*
> * Scan the pg_authid relation to find the Oid of the role(s) to be
> * deleted.
> @@ -1053,6 +1049,12 @@ DropRole(DropRoleStmt *stmt)
> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> errmsg("must be superuser to drop superusers")));
>
> + if (!have_createrole_privilege() &&
> + !pg_role_ownercheck(roleid, GetUserId()))
> + ereport(ERROR,
> + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> + errmsg("permission denied to drop role")));
> +
> /* DROP hook for the role being removed */
> InvokeObjectDropHook(AuthIdRelationId, roleid, 0);
>
> @@ -1811,6 +1813,18 @@ AlterRoleOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
> (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> errmsg("role may not own itself")));
>
> + /*
> + * Must not create cycles in the role ownership hierarchy. If this
> + * role owns (directly or indirectly) the proposed new owner, disallow
> + * the ownership transfer.
> + */
> + if (is_owner_of_role_nosuper(authForm->oid, newOwnerId))
> + ereport(ERROR,
> + (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
> + errmsg("role \"%s\" may not both own and be owned by role \"%s\"",
> + NameStr(authForm->rolname),
> + GetUserNameFromId(newOwnerId, false))));
> +
> authForm->rolowner = newOwnerId;
> CatalogTupleUpdate(rel, &tup->t_self, tup);
> diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c
> +/*
> + * Get a list of roles which own the given role, directly or indirectly.
> + *
> + * Each role has only one direct owner. The returned list contains the given
> + * role's owner, that role's owner, etc., up to the top of the ownership
> + * hierarchy, which is always the bootstrap superuser.
> + *
> + * Raises an error if any role ownership invariant is violated. Returns NIL if
> + * the given roleid is invalid.
> + */
> +static List *
> +roles_is_owned_by(Oid roleid)
> +{
> + List *owners_list = NIL;
> + Oid role_oid = roleid;
> +
> + /*
> + * Start with the current role and follow the ownership chain upwards until
> + * we reach the bootstrap superuser. To defend against getting into an
> + * infinite loop, we must check for ownership cycles. We choose to perform
> + * other corruption checks on the ownership structure while iterating, too.
> + */
> + while (OidIsValid(role_oid))
> + {
> + HeapTuple tuple;
> + Form_pg_authid authform;
> + Oid owner_oid;
> +
> + /* Find the owner of the current iteration's role */
> + tuple = SearchSysCache1(AUTHOID, ObjectIdGetDatum(role_oid));
> + if (!HeapTupleIsValid(tuple))
> + ereport(ERROR,
> + (errcode(ERRCODE_UNDEFINED_OBJECT),
> + errmsg("role with OID %u does not exist", role_oid)));
> +
> + authform = (Form_pg_authid) GETSTRUCT(tuple);
> + owner_oid = authform->rolowner;
> +
> + /*
> + * Roles must necessarily have owners. Even the bootstrap user has an
> + * owner. (It owns itself).
> + */
> + if (!OidIsValid(owner_oid))
> + ereport(ERROR,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("role \"%s\" with OID %u has invalid owner",
> + NameStr(authform->rolname), authform->oid)));
> +
> + /* The bootstrap user must own itself */
> + if (authform->oid == BOOTSTRAP_SUPERUSERID &&
> + owner_oid != BOOTSTRAP_SUPERUSERID)
> + ereport(ERROR,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("role \"%s\" with OID %u owned by role with OID %u",
> + NameStr(authform->rolname), authform->oid,
> + authform->rolowner)));
> +
> + /*
> + * Roles other than the bootstrap user must not be their own direct
> + * owners.
> + */
> + if (authform->oid != BOOTSTRAP_SUPERUSERID &&
> + authform->oid == owner_oid)
> + ereport(ERROR,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("role \"%s\" with OID %u owns itself",
> + NameStr(authform->rolname), authform->oid)));
> +
> + ReleaseSysCache(tuple);
> +
> + /* If we have reached the bootstrap user, we're done. */
> + if (role_oid == BOOTSTRAP_SUPERUSERID)
> + {
> + if (!owners_list)
> + owners_list = lappend_oid(owners_list, owner_oid);
> + break;
> + }
> +
> + /*
> + * For all other users, check they do not own themselves indirectly
> + * through an ownership cycle.
> + *
> + * Scanning the list each time through this loop results in overall
> + * quadratic work in the depth of the ownership chain, but we're
> + * not on a critical performance path, nor do we expect ownership
> + * hierarchies to be deep.
> + */
> + if (owners_list && list_member_oid(owners_list,
> + ObjectIdGetDatum(owner_oid)))
> + ereport(ERROR,
> + (errcode(ERRCODE_DATA_CORRUPTED),
> + errmsg("role \"%s\" with OID %u indirectly owns itself",
> + GetUserNameFromId(owner_oid, false),
> + owner_oid)));
> +
> + /* Done with sanity checks. Add this owner to the list. */
> + owners_list = lappend_oid(owners_list, owner_oid);
> +
> + /* Otherwise, iterate on this iteration's owner_oid. */
> + role_oid = owner_oid;
> + }
> +
> + return owners_list;
> +}
> @@ -4850,6 +4955,10 @@ has_privs_of_role(Oid member, Oid role)
> + /* Owners of roles have every privilge the owned role has */
> + if (pg_role_ownercheck(role, member))
> + return true;
Whoah, really? No, I don't agree with this, it's throwing away the
entire concept around inheritance of role rights and how you can have
roles which you can get the privileges of by doing a SET ROLE to them
but you don't automatically have those rights.
> +/*
> + * Is owner a direct or indirect owner of the role, not considering
> + * superuserness?
> + */
> +bool
> +is_owner_of_role_nosuper(Oid owner, Oid role)
> +{
> + return list_member_oid(roles_is_owned_by(role), owner);
> +}
Surely if you're a member of a role which owns another role, you should
be considered to be an owner of that role too..? Just checking if the
current role is a member of the roles which directly own the specified
role misses that case.
That is:
CREATE ROLE r1;
CREATE ROLE r2;
GRANT r2 to r1;
CREATE ROLE r3 AUTHORIZATION r2;
Surely, r1 is to be considered an owner of r3 in this case, but the
above check wouldn't consider that to be the case- it would only return
true if the current role is r2.
We do need some kind of direct membership check in the list of owners to
avoid creating loops, so maybe this function is kept as that and the
pg_role_ownership() check is changed to address the above case, but I
don't think we should just ignore role membership when it comes to role
ownership- we don't do that for any other kind of ownership check.
> Subject: [PATCH v4 4/5] Restrict power granted via CREATEROLE.
I would think this would be done independently of the other patches and
probably be first.
> diff --git a/doc/src/sgml/ref/alter_role.sgml b/doc/src/sgml/ref/alter_role.sgml
> @@ -70,18 +70,18 @@ ALTER ROLE { <replaceable class="parameter">role_specification</replaceable> | A
> <link linkend="sql-revoke"><command>REVOKE</command></link> for that.)
> Attributes not mentioned in the command retain their previous settings.
> Database superusers can change any of these settings for any role.
> - Roles having <literal>CREATEROLE</literal> privilege can change any of these
> - settings except <literal>SUPERUSER</literal>, <literal>REPLICATION</literal>,
> - and <literal>BYPASSRLS</literal>; but only for non-superuser and
> - non-replication roles.
> - Ordinary roles can only change their own password.
> + Role owners can change any of these settings on roles they own except
> + <literal>SUPERUSER</literal>, <literal>REPLICATION</literal>, and
> + <literal>BYPASSRLS</literal>; but only for non-superuser and non-replication
> + roles, and only if the role owner does not alter the target role to have a
> + privilege which the role owner itself lacks. Ordinary roles can only change
> + their own password.
> </para>
Having contemplated this a bit more, I don't like it, and it's not how
things work when it comes to regular privileges.
Consider that I can currently GRANT someone UPDATE privileges on an
object, but they can't GRANT that privilege to someone else unless I
explicitly allow it. The same could certainly be said for roles-
perhaps I want to allow someone the privilege to create non-login roles,
but I don't want them to be able to create new login roles, even if they
themselves have LOGIN.
As another point, I might want to have an 'admin' role that I want
admins to SET ROLE to before they go creating other roles, because I
don't want them to be creating roles as their regular user and so that
those other roles are owned by the 'admin' role, but I don't want that
role to have the 'login' attribute.
In other words, we should really consider what role attributes a given
role has to be independent of what role attributes that role is allowed
to set on roles they create. I appreciate that "just whatever the
current role has" is simpler and less work but also will be difficult to
walk back from once it's in the wild.
> @@ -1457,7 +1449,7 @@ AddRoleMems(const char *rolename, Oid roleid,
> /*
> - * Check permissions: must have createrole or admin option on the role to
> + * Check permissions: must be owner or have admin option on the role to
> * be changed. To mess with a superuser role, you gotta be superuser.
> */
> if (superuser_arg(roleid))
...
> @@ -1467,9 +1459,9 @@ AddRoleMems(const char *rolename, Oid roleid,
> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
> errmsg("must be superuser to alter superusers")));
> }
> - else
> + else if (!superuser())
> {
> - if (!have_createrole_privilege() &&
> + if (!pg_role_ownercheck(roleid, grantorId) &&
> !is_admin_of_role(grantorId, roleid))
> ereport(ERROR,
> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
I'm not entirely sure about including owners here though I'm not
completely against it either. This conflation of what the 'admin'
privileges on a role means vs. the 'ownership' of a role is part of what
I dislike about having two distinct systems for saying who is allowed to
GRANT one role to another.
Also, if we're going to always consider owners to be admins of roles
they own, why not push that into is_admin_of_role()?
> Subject: [PATCH v4 5/5] Remove grantor field from pg_auth_members
While I do think we should fix the issue with dangling references, I
dislike just getting rid of this entirely. While I don't really agree
with the spec about running around DROP'ing objects when a user's
privilege to create those objects has been revoked, I do think we should
be REVOKE'ing rights when a user's right to GRANT has been revoked, and
tracking the information about who GRANT'd what role to what other role
is needed for that. Further, we track who GRANT'd access to what in the
regular ACL system and I don't like the idea of removing that for roles.
> We could fix the bug, but there is no clear solution to the problem
> that existing installations may have broken data. Since the field
> is not used for any purpose, removing it seems the best option.
Existing broken systems will have to eventually be upgraded and the
admin will have to deal with such cases then, so I don't really consider
this to be that big of an issue or reason to entirely remove this.
If we're going to do this, it should also be done independently of the
role ownership stuff too.
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (819B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
@ 2022-01-24 19:19 ` Andrew Dunstan <[email protected]>
2 siblings, 0 replies; 28+ messages in thread
From: Andrew Dunstan @ 2022-01-24 19:19 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; Mark Dilger <[email protected]>; +Cc: Joshua Brindle <[email protected]>; Shinya Kato <[email protected]>; Bossart, Nathan <[email protected]>; pgsql-hackers; Jeff Davis <[email protected]>
On 1/22/22 16:20, Stephen Frost wrote:
>> Subject: [PATCH v4 1/5] Add tests of the CREATEROLE attribute.
> No particular issue with this one.
>
>
I'm going to commit this piece forthwith so we get it out of the way.
That will presumably make the cfbot unhappy until Mark submits a new
patch set.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
@ 2022-01-24 20:33 ` Robert Haas <[email protected]>
2022-01-24 21:00 ` Re: CREATEROLE and role ownership hierarchies Andrew Dunstan <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2 siblings, 2 replies; 28+ messages in thread
From: Robert Haas @ 2022-01-24 20:33 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Mark Dilger <[email protected]>; Joshua Brindle <[email protected]>; Andrew Dunstan <[email protected]>; Shinya Kato <[email protected]>; Bossart, Nathan <[email protected]>; pgsql-hackers; Jeff Davis <[email protected]>
On Sat, Jan 22, 2022 at 4:20 PM Stephen Frost <[email protected]> wrote:
> Whoah, really? No, I don't agree with this, it's throwing away the
> entire concept around inheritance of role rights and how you can have
> roles which you can get the privileges of by doing a SET ROLE to them
> but you don't automatically have those rights.
I see it differently. In my opinion, what that does is make the patch
actually useful instead of largely a waste of time. If you are a
service provider, you want to give your customers a super-user-like
experience without actually making them superuser. You don't want to
actually make them superuser, because then they could do things like
change archive_command or install plperlu and shell out to the OS
account, which you don't want. But you do want them to be able to
administer objects within the database just as a superuser could. And
a superuser has privileges over objects they own and objects belonging
to other users automatically, without needing to SET ROLE.
Imagine what happens if we adopt your proposal here. Everybody now has
to understand the behavior of a regular account, the behavior of a
superuser account, and the behavior of this third type of account
which is sort of like a superuser but requires a lot more SET ROLE
commands. And also every tool. So for example pg_dump and restore
isn't going to work, not even on the set of objects this
elevated-privilege user can access. pgAdmin isn't going to understand
that it needs to insert a bunch of extra SET ROLE commands to
administer objects. Ditto literally every other tool anyone has ever
written to administer PostgreSQL. And for all of that pain, we get
exactly zero extra security.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
@ 2022-01-24 21:00 ` Andrew Dunstan <[email protected]>
1 sibling, 0 replies; 28+ messages in thread
From: Andrew Dunstan @ 2022-01-24 21:00 UTC (permalink / raw)
To: Robert Haas <[email protected]>; Stephen Frost <[email protected]>; +Cc: Mark Dilger <[email protected]>; Joshua Brindle <[email protected]>; Shinya Kato <[email protected]>; Bossart, Nathan <[email protected]>; pgsql-hackers; Jeff Davis <[email protected]>
On 1/24/22 15:33, Robert Haas wrote:
> On Sat, Jan 22, 2022 at 4:20 PM Stephen Frost <[email protected]> wrote:
>> Whoah, really? No, I don't agree with this, it's throwing away the
>> entire concept around inheritance of role rights and how you can have
>> roles which you can get the privileges of by doing a SET ROLE to them
>> but you don't automatically have those rights.
> I see it differently. In my opinion, what that does is make the patch
> actually useful instead of largely a waste of time. If you are a
> service provider, you want to give your customers a super-user-like
> experience without actually making them superuser. You don't want to
> actually make them superuser, because then they could do things like
> change archive_command or install plperlu and shell out to the OS
> account, which you don't want. But you do want them to be able to
> administer objects within the database just as a superuser could. And
> a superuser has privileges over objects they own and objects belonging
> to other users automatically, without needing to SET ROLE.
>
+many
I encountered such issues on a cloud provider several years ago, and
blogged about the difficulties, which would have been solved very nicely
and cleanly by this proposal. It was when I understood properly how this
proposal worked, precisely as Robert states, that I became more
enthusiastic about it.
cheers
andrew
--
Andrew Dunstan
EDB: https://www.enterprisedb.com
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
@ 2022-01-24 21:23 ` Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
1 sibling, 1 reply; 28+ messages in thread
From: Stephen Frost @ 2022-01-24 21:23 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; Mark Dilger <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
Greetings,
On Mon, Jan 24, 2022 at 15:33 Robert Haas <[email protected]> wrote:
> On Sat, Jan 22, 2022 at 4:20 PM Stephen Frost <[email protected]> wrote:
> > Whoah, really? No, I don't agree with this, it's throwing away the
> > entire concept around inheritance of role rights and how you can have
> > roles which you can get the privileges of by doing a SET ROLE to them
> > but you don't automatically have those rights.
>
> I see it differently. In my opinion, what that does is make the patch
> actually useful instead of largely a waste of time.
The idea behind this patch is to enable creation and dropping of roles,
which isn’t possible now without being effectively a superuser.
Forcing owners to also implicitly have all rights of the roles they create
is orthogonal to that and an unnecessary change.
If you are a
> service provider, you want to give your customers a super-user-like
> experience without actually making them superuser. You don't want to
> actually make them superuser, because then they could do things like
> change archive_command or install plperlu and shell out to the OS
> account, which you don't want. But you do want them to be able to
> administer objects within the database just as a superuser could. And
> a superuser has privileges over objects they own and objects belonging
> to other users automatically, without needing to SET ROLE.
I am not saying that we would explicitly set all cases to be noninherit or
that we would even change the default away from what it is today, only that
we should use the existing role system and it’s concept of
inherit-vs-noninherit rather than throwing all of that away.
Everybody now has
> to understand the behavior of a regular account, the behavior of a
> superuser account, and the behavior of this third type of account
> which is sort of like a superuser but requires a lot more SET ROLE
> commands.
Inherit vs. noninherit roles is not a new concept, it has existed since the
role system was implemented. Further, that system does not require a lot
of SET ROLE commands unless and until an admin sets up a non-inherit role.
At that time, however, it’s expected that the rights of a role which has
inherit set to false are not automatically allowed for the role to which it
was GRANT’d. That’s how roles have always worked since they were
introduced.
And also every tool. So for example pg_dump and restore
> isn't going to work, not even on the set of objects this
> elevated-privilege user can access. pgAdmin isn't going to understand
> that it needs to insert a bunch of extra SET ROLE commands to
> administer objects. Ditto literally every other tool anyone has ever
> written to administer PostgreSQL. And for all of that pain, we get
> exactly zero extra security.
We have an inherit system today and pg_dump works just fine, as far as I’m
aware, and it does, indeed, issue SET ROLE at various points. Perhaps you
could explain with PG today what the issue is that is caused? Or what
issue pgAdmin has with PG’s existing role inherit system?
Further, being able to require a SET ROLE before running a given operation
is certainly a benefit in much the same way that having a user have to sudo
before running an operation is.
Thanks,
Stephen
>
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
@ 2022-01-24 21:41 ` Robert Haas <[email protected]>
2022-01-24 22:21 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Robert Haas @ 2022-01-24 21:41 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; Mark Dilger <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
On Mon, Jan 24, 2022 at 4:23 PM Stephen Frost <[email protected]> wrote:
> The idea behind this patch is to enable creation and dropping of roles, which isn’t possible now without being effectively a superuser.
>
> Forcing owners to also implicitly have all rights of the roles they create is orthogonal to that and an unnecessary change.
I just took a look at the first email on this thread and it says this:
>>> These patches have been split off the now deprecated monolithic "Delegating superuser tasks to new security roles" thread at [1].
Therefore I think it is pretty clear that the goals of this patch set
include being able to delegate superuser tasks to new security roles.
And having those tasks be delegated but *work randomly differently* is
much less useful.
> I am not saying that we would explicitly set all cases to be noninherit or that we would even change the default away from what it is today, only that we should use the existing role system and it’s concept of inherit-vs-noninherit rather than throwing all of that away.
INHERIT vs. NOINHERIT is documented to control the behavior of role
*membership*. This patch is introducing a new concept of role
*ownership*. It's not self-evident that what applies to one case
should apply to the other.
> Further, being able to require a SET ROLE before running a given operation is certainly a benefit in much the same way that having a user have to sudo before running an operation is.
That's a reasonable point of view, but having things work similarly to
what happens for a superuser is ALSO a very big benefit. In my
opinion, in fact, it is a far larger benefit.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
@ 2022-01-24 22:21 ` Stephen Frost <[email protected]>
2022-01-24 23:18 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
2022-01-25 16:42 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
2022-01-25 19:04 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
0 siblings, 3 replies; 28+ messages in thread
From: Stephen Frost @ 2022-01-24 22:21 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; Mark Dilger <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
Greetings,
On Mon, Jan 24, 2022 at 16:42 Robert Haas <[email protected]> wrote:
> On Mon, Jan 24, 2022 at 4:23 PM Stephen Frost <[email protected]> wrote:
> > The idea behind this patch is to enable creation and dropping of roles,
> which isn’t possible now without being effectively a superuser.
> >
> > Forcing owners to also implicitly have all rights of the roles they
> create is orthogonal to that and an unnecessary change.
>
> I just took a look at the first email on this thread and it says this:
>
> >>> These patches have been split off the now deprecated monolithic
> "Delegating superuser tasks to new security roles" thread at [1].
>
> Therefore I think it is pretty clear that the goals of this patch set
> include being able to delegate superuser tasks to new security roles.
> And having those tasks be delegated but *work randomly differently* is
> much less useful.
Being able to create and drop users is, in fact, effectively a
superuser-only task today. We could throw out the entire idea of role
ownership, in fact, as being entirely unnecessary when talking about that
specific task.
> I am not saying that we would explicitly set all cases to be noninherit
> or that we would even change the default away from what it is today, only
> that we should use the existing role system and it’s concept of
> inherit-vs-noninherit rather than throwing all of that away.
>
> INHERIT vs. NOINHERIT is documented to control the behavior of role
> *membership*. This patch is introducing a new concept of role
> *ownership*. It's not self-evident that what applies to one case
> should apply to the other.
This is an argument to drop the role ownership concept, as I view it.
Privileges are driven by membership today and inventing some new
independent way to do that is increasing confusion, not improving things.
I disagree that adding role ownership should necessarily change how the
regular GRANT privilege system works or throw away basic concepts of that
system which have been in place for decades. Increasing the number of
independent ways to answer the question of “what users have what rights on
object X” is an active bad thing. Anything that cares about object access
will now also have to address role ownership to answer that question, while
if we don’t include this one change then they don’t need to directly have
any concern for ownership because regular object privileges still work the
same way they did before.
> Further, being able to require a SET ROLE before running a given
> operation is certainly a benefit in much the same way that having a user
> have to sudo before running an operation is.
>
> That's a reasonable point of view, but having things work similarly to
> what happens for a superuser is ALSO a very big benefit. In my
> opinion, in fact, it is a far larger benefit.
Superuser is a problem specifically because it gives people access to do
absolutely anything, both for security and safety concerns. Disallowing a
way to curtail that same risk when it comes to role ownership invites
exactly those same problems.
I appreciate that there’s an edge between the ownership system being
proposed and the existing role membership system, but we’d be much better
off trying to minimize the amount that they end up overlapping- role
ownership should be about managing roles.
To push back on the original “tenant” argument, consider that one of the
bigger issues in cloud computing today is exactly the problem that the
cloud managers can potentially gain access to the sensitive data of their
tenants and that’s not generally viewed as a positive thing. This change
would make it so that every landlord can go and SELECT from the tables of
their tenants without so much as a by-your-leave. The tenants likely don’t
like that idea, and almost as likely the landlords in many cases aren’t
thrilled with it either. Should the landlords be able to DROP the tenant
due to the tenant not paying their bill? Of course, and that should then
eliminate the tenant’s tables and other objects which take up resources,
but that’s not the same thing as saying that a landlord should be able to
unlock a tenant’s old phone that they left behind (and yeah, maybe the
analogy falls apart a bit there, but the point I’m trying to get at is that
it’s not as simple as it’s being made out to be here and we should think
about these things and not just implicitly grant all access to the owner
because that’s an easy thing to do- and is exactly what viewing owners as
“mini superusers” does and leads to many of the same issues we already have
with superusers).
Thanks,
Stephen
>
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 22:21 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
@ 2022-01-24 23:18 ` Mark Dilger <[email protected]>
2022-01-25 06:55 ` Re: CREATEROLE and role ownership hierarchies Fujii Masao <[email protected]>
2 siblings, 1 reply; 28+ messages in thread
From: Mark Dilger @ 2022-01-24 23:18 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
> On Jan 24, 2022, at 2:21 PM, Stephen Frost <[email protected]> wrote:
>
> Superuser is a problem specifically because it gives people access to do absolutely anything, both for security and safety concerns. Disallowing a way to curtail that same risk when it comes to role ownership invites exactly those same problems.
Before the patch, users with CREATEROLE can do mischief. After the patch, users with CREATEROLE can do mischief. The difference is that the mischief that can be done after the patch is a proper subset of the mischief that can be done before the patch. (Counter-examples highly welcome.)
Specifically, I claim that before the patch, non-superuser "bob" with CREATEROLE can interfere with *any* non-superuser. After the patch, non-superuser "bob" with CREATEROLE can interfere with *some* non-superusers; specifically, with non-superusers he created himself, or which have had ownership transferred to him.
Restricting the scope of bob's mischief is a huge win, in my view.
The argument about whether owners should always implicitly inherit privileges from roles they own is a bit orthogonal to my point about mischief-making. Do we at least agree on the mischief-abatement aspect of this patch set?
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 22:21 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 23:18 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
@ 2022-01-25 06:55 ` Fujii Masao <[email protected]>
2022-01-25 16:21 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
0 siblings, 1 reply; 28+ messages in thread
From: Fujii Masao @ 2022-01-25 06:55 UTC (permalink / raw)
To: Mark Dilger <[email protected]>; Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
On 2022/01/25 8:18, Mark Dilger wrote:
>
>
>> On Jan 24, 2022, at 2:21 PM, Stephen Frost <[email protected]> wrote:
>>
>> Superuser is a problem specifically because it gives people access to do absolutely anything, both for security and safety concerns. Disallowing a way to curtail that same risk when it comes to role ownership invites exactly those same problems.
>
> Before the patch, users with CREATEROLE can do mischief. After the patch, users with CREATEROLE can do mischief. The difference is that the mischief that can be done after the patch is a proper subset of the mischief that can be done before the patch. (Counter-examples highly welcome.)
>
> Specifically, I claim that before the patch, non-superuser "bob" with CREATEROLE can interfere with *any* non-superuser. After the patch, non-superuser "bob" with CREATEROLE can interfere with *some* non-superusers; specifically, with non-superusers he created himself, or which have had ownership transferred to him.
>
> Restricting the scope of bob's mischief is a huge win, in my view.
+1
One of "mischiefs" I'm thinking problematic is that users with CREATEROLE can give any predefined role that they don't have, to other users including themselves. For example, users with CREATEROLE can give pg_execute_server_program to themselves and run any OS commands by COPY PROGRAM. This would be an issue when providing something like PostgreSQL cloud service that wants to prevent end users from running OS commands but allow them to create/drop roles. Does the proposed patch fix also this issue?
Regards,
--
Fujii Masao
Advanced Computing Technology Center
Research and Development Headquarters
NTT DATA CORPORATION
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 22:21 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 23:18 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
2022-01-25 06:55 ` Re: CREATEROLE and role ownership hierarchies Fujii Masao <[email protected]>
@ 2022-01-25 16:21 ` Mark Dilger <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Mark Dilger @ 2022-01-25 16:21 UTC (permalink / raw)
To: Fujii Masao <[email protected]>; +Cc: Stephen Frost <[email protected]>; Robert Haas <[email protected]>; Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
> On Jan 24, 2022, at 10:55 PM, Fujii Masao <[email protected]> wrote:
>
> +1
>
> One of "mischiefs" I'm thinking problematic is that users with CREATEROLE can give any predefined role that they don't have, to other users including themselves. For example, users with CREATEROLE can give pg_execute_server_program to themselves and run any OS commands by COPY PROGRAM. This would be an issue when providing something like PostgreSQL cloud service that wants to prevent end users from running OS commands but allow them to create/drop roles. Does the proposed patch fix also this issue?
Yes, the patch restricts CREATEROLE privilege from granting any privilege they themselves lack. There is a regression test in the patch set which demonstrates this. See src/test/regress/expected/create_role.out. The diffs from v6-0004-Restrict-power-granted-via-CREATEROLE.patch are quoted here for ease of viewing:
--- ok, having CREATEROLE is enough to create roles in privileged roles
+-- fail, having CREATEROLE is not enough to create roles in privileged roles
CREATE ROLE regress_read_all_data IN ROLE pg_read_all_data;
+ERROR: must have admin option on role "pg_read_all_data"
CREATE ROLE regress_write_all_data IN ROLE pg_write_all_data;
+ERROR: must have admin option on role "pg_write_all_data"
CREATE ROLE regress_monitor IN ROLE pg_monitor;
+ERROR: must have admin option on role "pg_monitor"
CREATE ROLE regress_read_all_settings IN ROLE pg_read_all_settings;
+ERROR: must have admin option on role "pg_read_all_settings"
CREATE ROLE regress_read_all_stats IN ROLE pg_read_all_stats;
+ERROR: must have admin option on role "pg_read_all_stats"
CREATE ROLE regress_stat_scan_tables IN ROLE pg_stat_scan_tables;
+ERROR: must have admin option on role "pg_stat_scan_tables"
CREATE ROLE regress_read_server_files IN ROLE pg_read_server_files;
+ERROR: must have admin option on role "pg_read_server_files"
CREATE ROLE regress_write_server_files IN ROLE pg_write_server_files;
+ERROR: must have admin option on role "pg_write_server_files"
CREATE ROLE regress_execute_server_program IN ROLE pg_execute_server_program;
+ERROR: must have admin option on role "pg_execute_server_program"
CREATE ROLE regress_signal_backend IN ROLE pg_signal_backend;
+ERROR: must have admin option on role "pg_signal_backend"
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 22:21 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
@ 2022-01-25 16:42 ` Mark Dilger <[email protected]>
2 siblings, 0 replies; 28+ messages in thread
From: Mark Dilger @ 2022-01-25 16:42 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Robert Haas <[email protected]>; Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
> On Jan 24, 2022, at 2:21 PM, Stephen Frost <[email protected]> wrote:
>
> To push back on the original “tenant” argument, consider that one of the bigger issues in cloud computing today is exactly the problem that the cloud managers can potentially gain access to the sensitive data of their tenants and that’s not generally viewed as a positive thing.
+1. This is a real problem. I have been viewing this problem as separate from the one which role ownership is intended to fix. Do you have a suggestion about how to tackle the problems together with less work than tackling them separately?
> This change would make it so that every landlord can go and SELECT from the tables of their tenants without so much as a by-your-leave.
I would expect that is already true. A user with CREATEROLE can do almost everything. This patch closes some CREATEROLE related security problems, but not this one you mention.
> The tenants likely don’t like that idea
+1
> , and almost as likely the landlords in many cases aren’t thrilled with it either.
+1
> Should the landlords be able to DROP the tenant due to the tenant not paying their bill? Of course, and that should then eliminate the tenant’s tables and other objects which take up resources, but that’s not the same thing as saying that a landlord should be able to unlock a tenant’s old phone that they left behind (and yeah, maybe the analogy falls apart a bit there, but the point I’m trying to get at is that it’s not as simple as it’s being made out to be here and we should think about these things and not just implicitly grant all access to the owner because that’s an easy thing to do- and is exactly what viewing owners as “mini superusers” does and leads to many of the same issues we already have with superusers).
This is a pretty interesting argument. I don't believe it will work to do as you say unconditionally, as there is still a need to have CREATEROLE users who have privileges on their created roles' objects, even if for no other purpose than to be able to REASSIGN OWNED BY those objects before dropping roles. But maybe there is also a need to have CREATEROLE users who lack that privilege? Would that be a privilege bit akin to (but not the same as!) the INHERIT privilege? Should I redesign for something like that?
I like that the current patch restricts CREATEROLE users from granting privileges they themselves lack. Would such a new privilege bit work the same way? Imagine that you, "stephen", have CREATEROLE but not this new bit, and you create me, "mark" as a tenant with CREATEROLE. Can you give me the bit? Or does the fact that you lack the bit mean you can't give it to me, either?
Other suggestions?
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 22:21 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
@ 2022-01-25 19:04 ` Robert Haas <[email protected]>
2 siblings, 0 replies; 28+ messages in thread
From: Robert Haas @ 2022-01-25 19:04 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Andrew Dunstan <[email protected]>; Bossart, Nathan <[email protected]>; Jeff Davis <[email protected]>; Joshua Brindle <[email protected]>; Mark Dilger <[email protected]>; pgsql-hackers; Shinya Kato <[email protected]>
On Mon, Jan 24, 2022 at 5:21 PM Stephen Frost <[email protected]> wrote:
> This is an argument to drop the role ownership concept, as I view it. Privileges are driven by membership today and inventing some new independent way to do that is increasing confusion, not improving things. I disagree that adding role ownership should necessarily change how the regular GRANT privilege system works or throw away basic concepts of that system which have been in place for decades. Increasing the number of independent ways to answer the question of “what users have what rights on object X” is an active bad thing. Anything that cares about object access will now also have to address role ownership to answer that question, while if we don’t include this one change then they don’t need to directly have any concern for ownership because regular object privileges still work the same way they did before.
It really feels to me like you just keep moving the goalposts. We
started out with a conversation where Mark said he'd like to be able
to grant permissions on GUCs to non-superusers.[1] You argued
repeatedly that we really needed to do something about CREATEROLE
[2,3,4]. Mark argued that this was an unrelated problem[5] but you
argued that unless it were addressed, users would still be able to
break out of the sandbox[6] which must mean either the OS user, or at
least PostgreSQL users other than the ones they were supposed to be
able to control.
That led *directly* to the patch at hand, which solves the problem by
inventing the notion of role ownership, so that you can distinguish
the roles you can administer from the ones you drop. You are now
proposing that we get rid of that concept, a concept that was added
four months ago[7] as a direct response to your previous feedback.
It's completely unfair to make an argument that results in the
addition of a complex piece of machinery to a body of work that was
initially on an only marginally related topic and then turn around and
argue, quite close to the end of the release cycle, for the removal of
that exact same mechanism.
And your argument about whether the privileges should be able to be
exercised without SET ROLE is also just completely baffling to me
given the previous conversation. It seems 100% clear from the previous
discussion that we were talking about service provider environments
and trying to deliver a good user experience to "lead tenants" in such
environments. Regardless of the technical details of how INHERIT or
anything else work, an actual superuser would not be subject to a
restriction similar to the one you're talking about, so arguing that
it ought to be present here for some technical reason is placing
technicalities ahead of what seemed at the time to be a shared goal.
There's a perfectly good argument to be made that the superuser role
should not work the way it does, but it's too late to relitigate that.
And I can't imagine why any service provider would find any value in a
new role that requires all of the extra push-ups you're trying to
impose on it.
I just can't shake the feeling that you're trying to redesign this
patch out of (a) getting committed and (b) solving any of the problems
it intends to solve, problems with which you largely seemed to agree.
I assume that is not actually your intention, but I can't think of
anything you'd be doing differently here if it were.
[1] https://www.postgresql.org/message-id/F9408A5A-B20B-42D2-9E7F-49CD3D1547BC%40enterprisedb.com
[2] https://www.postgresql.org/message-id/20210726200542.GX20766%40tamriel.snowman.net
[3] https://www.postgresql.org/message-id/20210726205433.GA20766%40tamriel.snowman.net
[4] https://www.postgresql.org/message-id/20210823181351.GB17906%40tamriel.snowman.net
[5] https://www.postgresql.org/message-id/92AA9A52-A644-42FE-B699-8ECAEE12E635%40enterprisedb.com
[6] https://www.postgresql.org/message-id/20210823195130.GF17906%40tamriel.snowman.net
[7] https://www.postgresql.org/message-id/67BB2F92-704B-415C-8D47-149327CA8F4B%40enterprisedb.com
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 28+ messages in thread
* Re: CREATEROLE and role ownership hierarchies
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
@ 2022-01-25 23:17 ` Mark Dilger <[email protected]>
2 siblings, 0 replies; 28+ messages in thread
From: Mark Dilger @ 2022-01-25 23:17 UTC (permalink / raw)
To: Stephen Frost <[email protected]>; +Cc: Joshua Brindle <[email protected]>; Andrew Dunstan <[email protected]>; Shinya Kato <[email protected]>; Bossart, Nathan <[email protected]>; pgsql-hackers; Jeff Davis <[email protected]>
> On Jan 22, 2022, at 1:20 PM, Stephen Frost <[email protected]> wrote:
>
>> Subject: [PATCH v4 1/5] Add tests of the CREATEROLE attribute.
>
> No particular issue with this one.
Andrew already committed this, forcing the remaining patches to be renumbered. Per your comments below, I have combined what was 0002+0003 into 0001, renumbered 0004 as 0002, and abandoned 0005. (It may come back as an independent patch.) Also owing to the fact that 0001 has been committed, I really need to post another patch set right away, to make the cfbot happy. I'm fixing non-controversial deficits you call out in your review, but leaving other things unchanged, in the interest of getting a patch posted sooner rather than later.
>> Subject: [PATCH v4 2/5] Add owners to roles
>>
>> All roles now have owners. By default, roles belong to the role
>> that created them, and initdb-time roles are owned by POSTGRES.
>
> ... database superuser, not 'POSTGRES'.
I rephrased this as "bootstrap superuser" in the commit message.
>> +++ b/src/backend/catalog/aclchk.c
>> @@ -5430,6 +5434,57 @@ pg_statistics_object_ownercheck(Oid stat_oid, Oid roleid)
>> return has_privs_of_role(roleid, ownerId);
>> }
>>
>> +/*
>> + * Ownership check for a role (specified by OID)
>> + */
>> +bool
>> +pg_role_ownercheck(Oid role_oid, Oid roleid)
>> +{
>> + HeapTuple tuple;
>> + Form_pg_authid authform;
>> + Oid owner_oid;
>> +
>> + /* Superusers bypass all permission checking. */
>> + if (superuser_arg(roleid))
>> + return true;
>> +
>> + /* Otherwise, look up the owner of the role */
>> + tuple = SearchSysCache1(AUTHOID, ObjectIdGetDatum(role_oid));
>> + if (!HeapTupleIsValid(tuple))
>> + ereport(ERROR,
>> + (errcode(ERRCODE_UNDEFINED_OBJECT),
>> + errmsg("role with OID %u does not exist",
>> + role_oid)));
>> + authform = (Form_pg_authid) GETSTRUCT(tuple);
>> + owner_oid = authform->rolowner;
>> +
>> + /*
>> + * Roles must necessarily have owners. Even the bootstrap user has an
>> + * owner. (It owns itself). Other roles must form a proper tree.
>> + */
>> + if (!OidIsValid(owner_oid))
>> + ereport(ERROR,
>> + (errcode(ERRCODE_DATA_CORRUPTED),
>> + errmsg("role \"%s\" with OID %u has invalid owner",
>> + authform->rolname.data, authform->oid)));
>> + if (authform->oid != BOOTSTRAP_SUPERUSERID &&
>> + authform->rolowner == authform->oid)
>> + ereport(ERROR,
>> + (errcode(ERRCODE_DATA_CORRUPTED),
>> + errmsg("role \"%s\" with OID %u owns itself",
>> + authform->rolname.data, authform->oid)));
>> + if (authform->oid == BOOTSTRAP_SUPERUSERID &&
>> + authform->rolowner != BOOTSTRAP_SUPERUSERID)
>> + ereport(ERROR,
>> + (errcode(ERRCODE_DATA_CORRUPTED),
>> + errmsg("role \"%s\" with OID %u owned by role with OID %u",
>> + authform->rolname.data, authform->oid,
>> + authform->rolowner)));
>> + ReleaseSysCache(tuple);
>> +
>> + return (owner_oid == roleid);
>> +}
>
> Do we really need all of these checks on every call of this function..?
Since the function is following the ownership chain upwards, it seems necessary to check that the chain is wellformed, else we might get into an infinite loop or return the wrong answer. These would only happen under corrupt conditions, but it seems sensible to check for those, since they are cheap to check. (Actually, the check for nontrivial cycles included in the patch is not as efficient as it could be, but I'm punting the work of improving that algorithm from quadratic to linear until a later patch version, in the interest of posting the patch soon.)
> Also, there isn't much point in including the role OID twice in the last
> error message, is there? Unless things have gotten quite odd, it's
> goint to be the same value both times as we just proved to ourselves
> that it is, in fact, the same value (and that it's not the
> BOOTSTRAP_SUPERUSERID).
It is comparing the authform->oid against the authform->rolowner, which are not the same. The first is the owned role, the second is the owning role. We could hardcode the message to say something like "bootstrap superuser owned by role with Oid %u", but that hardcodes "bootstrap superuser" into the message, rather than something like "stephen". I don't feel strongly about the wording. Let me know if you still want me to change it.
> This function also doesn't actually do any kind of checking to see if
> the role ownership forms a proper tree, so it seems a bit odd to have
> the comment talking about that here where it's doing other checks.
Right. The comment simply explains the structure we expect, not the structure we are fully validating. The point is that each link in the hierarchy must be compatible with the expected structure. It would be overkill to validate the whole tree in this one function. I don't mind rewording the code comment, if you have a less confusing suggestion.
>> +++ b/src/backend/commands/user.c
>> @@ -77,6 +79,9 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
>> Datum new_record[Natts_pg_authid];
>> bool new_record_nulls[Natts_pg_authid];
>> Oid roleid;
>> + Oid owner_uid;
>> + Oid saved_uid;
>> + int save_sec_context;
>
> Seems a bit odd to introduce 'uid' into this file, which hasn't got any
> such anywhere in it, and I'm not entirely sure that any of these are
> actually needed..?
Good catch! The implementation in v6 was wrong. It didn't enforce that the creating role was a member of the target owner, something this next patch set does.
>> @@ -108,6 +113,16 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
>> DefElem *dvalidUntil = NULL;
>> DefElem *dbypassRLS = NULL;
>>
>> + GetUserIdAndSecContext(&saved_uid, &save_sec_context);
>> +
>> + /*
>> + * Who is supposed to own the new role?
>> + */
>> + if (stmt->authrole)
>> + owner_uid = get_rolespec_oid(stmt->authrole, false);
>> + else
>> + owner_uid = saved_uid;
>> +
>> /* The defaults can vary depending on the original statement type */
>> switch (stmt->stmt_type)
>> {
>> @@ -254,6 +269,10 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
>> ereport(ERROR,
>> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
>> errmsg("must be superuser to create superusers")));
>> + if (!superuser_arg(owner_uid))
>> + ereport(ERROR,
>> + (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
>> + errmsg("must be superuser to own superusers")));
>> }
>> else if (isreplication)
>> {
>
> So, we're telling a superuser (which is the only way you could get to
> this point...) that they aren't allowed to create a superuser role which
> is owned by a non-superuser... Why?
The reason is one you won't like very much. Given that roles have the privileges of roles they own (which you don't like), allowing a non-superuser to own a superuser effectively promotes that owner to superuser status. That's a pretty obscure way of making someone a superuser, probably not what was intended, and quite a high-caliber foot-gun.
Even if roles didn't inherit privileges from roles they own, I think it would be odd for a non-superuser to own a superuser. The definition of "ownership" would have to be extremely restricted to prevent the owner from using their ownership to obtain superuser.
>> @@ -310,6 +329,19 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
>> errmsg("role \"%s\" already exists",
>> stmt->role)));
>>
>> + /*
>> + * If the requested authorization is different from the current user,
>> + * temporarily set the current user so that the object(s) will be created
>> + * with the correct ownership.
>> + *
>> + * (The setting will be restored at the end of this routine, or in case of
>> + * error, transaction abort will clean things up.)
>> + */
>> + if (saved_uid != owner_uid)
>> + SetUserIdAndSecContext(owner_uid,
>> + save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
>
> Err, why is this needed? This looks copied from the CreateSchemaCommand
> but, unlike with the create schema command, CreateRole doesn't actually
> allow sub-commands to be run to create other objects in the way that
> CreateSchemaCommand does.
Not quite. There are still the check_password_hook and RunObjectPostCreateHook() to consider. The check_password_hook might want to validate the validuntil_time parameter against the owner's validuntil time, or some other property of the owner. And the RunObjectPostCreateHook (called via InvokeObjectPostCreateHook(AuthIdRelationId, roleid, 0)) may want the information, too.
I'm not saying these are super strong arguments. If people generally feel that CREATE ROLE ... AUTHORIZATION shouldn't call SetUserIdAndSecContext, feel free to argue that.
>> @@ -1675,3 +1714,110 @@ DelRoleMems(const char *rolename, Oid roleid,
>> +static void
>> +AlterRoleOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
>> +{
>> + Form_pg_authid authForm;
>> +
>> + Assert(tup->t_tableOid == AuthIdRelationId);
>> + Assert(RelationGetRelid(rel) == AuthIdRelationId);
>> +
>> + authForm = (Form_pg_authid) GETSTRUCT(tup);
>> +
>> + /*
>> + * If the new owner is the same as the existing owner, consider the
>> + * command to have succeeded. This is for dump restoration purposes.
>> + */
>> + if (authForm->rolowner != newOwnerId)
>> + {
>> + /* Otherwise, must be owner of the existing object */
>> + if (!pg_role_ownercheck(authForm->oid, GetUserId()))
>> + aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_ROLE,
>> + NameStr(authForm->rolname));
>> +
>> + /* Must be able to become new owner */
>> + check_is_member_of_role(GetUserId(), newOwnerId);
>
> Feels like we should be saying a bit more about why we check for role
> membership vs. has_privs_of_role() here. I'm generally of the opinion
> that membership is the right thing to check here, just feel like we
> should try to explain more why that's the right thing.
For orthogonality with how ALTER .. OWNER TO works for everything else? AlterEventTriggerOwner_internal doesn't check this explicitly, but that's because it has already checked that the new owner is superuser, so the check must necessarily succeed. I'm not aware of any ALTER .. OWNER TO commands that don't require this, at least implicitly.
We could explain this in AlterRoleOwner_internal, as you suggest, but if we need it there, do we need to put the same explanation in functions which handle other object types? I don't see why this one function would require the explanation if other equivalent functions do not.
>> + /*
>> + * must have CREATEROLE rights
>> + *
>> + * NOTE: This is different from most other alter-owner checks in that
>> + * the current user is checked for create privileges instead of the
>> + * destination owner. This is consistent with the CREATE case for
>> + * roles. Because superusers will always have this right, we need no
>> + * special case for them.
>> + */
>> + if (!have_createrole_privilege())
>> + aclcheck_error(ACLCHECK_NO_PRIV, OBJECT_ROLE,
>> + NameStr(authForm->rolname));
>> +
>
> I would think we'd be trying to get away from the role attribute stuff.
That's not a bad idea, but I thought it was discussed months ago. The two options were (1) keep using CREATEROLE but change it to be less powerful, and (2) add a new built-in role, say "pg_create_role", and have membership in that role be what we use. Option (2) was generally viewed less favorably, or that was my sense of people's opinions, on the theory that we'd be better off fixing how CREATEROLE works than having two different ways of doing roughly the same thing.
>> diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
>
>> + CREATE ROLE RoleId AUTHORIZATION RoleSpec opt_with OptRoleList
>> + {
>> + CreateRoleStmt *n = makeNode(CreateRoleStmt);
>> + n->stmt_type = ROLESTMT_ROLE;
>> + n->role = $3;
>> + n->authrole = $5;
>> + n->options = $7;
>> + $$ = (Node *)n;
>> + }
>> ;
>
> ...
>
>> @@ -1218,6 +1229,10 @@ CreateOptRoleElem:
>> {
>> $$ = makeDefElem("addroleto", (Node *)$3, @1);
>> }
>> + | OWNER RoleSpec
>> + {
>> + $$ = makeDefElem("owner", (Node *)$2, @1);
>> + }
>> ;
>
> Not sure why we'd have both AUTHORIZATION and OWNER for CREATE ROLE..?
> We don't do that for other objects.
Good catch! The "OWNER RoleSpec" here was unused. I have removed it from the new patch set.
>> diff --git a/src/test/regress/sql/create_role.sql b/src/test/regress/sql/create_role.sql
>
>> @@ -1,6 +1,7 @@
>> -- ok, superuser can create users with any set of privileges
>> CREATE ROLE regress_role_super SUPERUSER;
>> CREATE ROLE regress_role_1 CREATEDB CREATEROLE REPLICATION BYPASSRLS;
>> +GRANT CREATE ON DATABASE regression TO regress_role_1;
>
> Seems odd to add this as part of this patch, or am I missing something?
It's not used much in patch 0001 where it gets introduced, but gets used more in patch 0002. I put it here to reduce the number of diffs the next patch creates.
>> From 1784a5b51d4dbebf99798b5832d92b0f585feb08 Mon Sep 17 00:00:00 2001
>> From: Mark Dilger <[email protected]>
>> Date: Tue, 4 Jan 2022 11:42:27 -0800
>> Subject: [PATCH v4 3/5] Give role owners control over owned roles
>>
>> Create a role ownership hierarchy. The previous commit added owners
>> to roles. This goes further, making role ownership transitive. If
>> role A owns role B, and role B owns role C, then role A can act as
>> the owner of role C. Also, roles A and B can perform any action on
>> objects belonging to role C that role C could itself perform.
>>
>> This is a preparatory patch for changing how CREATEROLE works.
>
> This feels odd to have be an independent commit.
Reworked the v6-0002 and v6-0003 patches into just one, as discussed at the top of this email.
>> diff --git a/src/backend/commands/schemacmds.c b/src/backend/commands/schemacmds.c
>
>> @@ -363,7 +363,7 @@ AlterSchemaOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
>> /*
>> * must have create-schema rights
>> *
>> - * NOTE: This is different from other alter-owner checks in that the
>> + * NOTE: This is different from most other alter-owner checks in that the
>> * current user is checked for create privileges instead of the
>> * destination owner. This is consistent with the CREATE case for
>> * schemas. Because superusers will always have this right, we need
>
> Not a fan of just dropping 'most' in here, doesn't really help someone
> understand what is being talked about. I'd suggest adjusting the
> comment to talk about alter-owner checks for objects which exist in
> schemas, as that's really what is being referred to.
Yeah, that's a better approach. This next patch set changes the comment in both AlterSchemaOwner_internal and AlterRoleOwner_internal to make that clear.
...<snip>...
> Whoah, really? No, I don't agree with this, it's throwing away the
> entire concept around inheritance of role rights and how you can have
> roles which you can get the privileges of by doing a SET ROLE to them
> but you don't automatically have those rights.
I didn't change any of this for the next patch set, not because I'm ignoring you, but because we're still arguing out what the right behavior should be. Whatever we come up with, I think it should allow the use case that Robert has been talking about. Doing that and also doing what you are talking about might be hard, but I'm still hoping to find some solution.
Recall that upthread, months ago, we discussed that it is abnormal for any role to be a member of a login role. You can think of "login role" as a synonym for "user", and "non-login role" as a synonym for "group", and that language makes it easier to think about how weird it is for users to be members of other users.
It's perfectly sensible to have users own users, but not for users to be members of users. If not for that, I'd be in favor of what you suggest, excepting that I'd accommodate Robert's requirements by having the owner of a role have ADMIN on that role by default, with grammar for requesting the alternative. Maybe there is something I'm forgetting to consider just now, but I'd think that would handle Robert's "tenant" type argument while also making it easy to operate the way that you want. But, again, it does require having users be members of users, something which was rejected in the discussion months ago.
>> +/*
>> + * Is owner a direct or indirect owner of the role, not considering
>> + * superuserness?
>> + */
>> +bool
>> +is_owner_of_role_nosuper(Oid owner, Oid role)
>> +{
>> + return list_member_oid(roles_is_owned_by(role), owner);
>> +}
>
>
> Surely if you're a member of a role which owns another role, you should
> be considered to be an owner of that role too..? Just checking if the
> current role is a member of the roles which directly own the specified
> role misses that case.
>
> That is:
>
> CREATE ROLE r1;
> CREATE ROLE r2;
>
> GRANT r2 to r1;
>
> CREATE ROLE r3 AUTHORIZATION r2;
>
> Surely, r1 is to be considered an owner of r3 in this case, but the
> above check wouldn't consider that to be the case- it would only return
> true if the current role is r2.
>
> We do need some kind of direct membership check in the list of owners to
> avoid creating loops, so maybe this function is kept as that and the
> pg_role_ownership() check is changed to address the above case, but I
> don't think we should just ignore role membership when it comes to role
> ownership- we don't do that for any other kind of ownership check.
I like this line of reasoning, and it appears to be an argument in your favor where the larger question is concerned. If role ownership is transitive, and role membership is transitive, it gets weird trying to work out larger relationship chains.
This deserves more attention.
>> Subject: [PATCH v4 4/5] Restrict power granted via CREATEROLE.
>
> I would think this would be done independently of the other patches and
> probably be first.
The way I'm trying to fix CREATEROLE is first by introducing the concept of role owners, then second by restricting what roles can do based on whether they own a target role. I don't see how I can reverse the order.
>> diff --git a/doc/src/sgml/ref/alter_role.sgml b/doc/src/sgml/ref/alter_role.sgml
>
>> @@ -70,18 +70,18 @@ ALTER ROLE { <replaceable class="parameter">role_specification</replaceable> | A
>> <link linkend="sql-revoke"><command>REVOKE</command></link> for that.)
>> Attributes not mentioned in the command retain their previous settings.
>> Database superusers can change any of these settings for any role.
>> - Roles having <literal>CREATEROLE</literal> privilege can change any of these
>> - settings except <literal>SUPERUSER</literal>, <literal>REPLICATION</literal>,
>> - and <literal>BYPASSRLS</literal>; but only for non-superuser and
>> - non-replication roles.
>> - Ordinary roles can only change their own password.
>> + Role owners can change any of these settings on roles they own except
>> + <literal>SUPERUSER</literal>, <literal>REPLICATION</literal>, and
>> + <literal>BYPASSRLS</literal>; but only for non-superuser and non-replication
>> + roles, and only if the role owner does not alter the target role to have a
>> + privilege which the role owner itself lacks. Ordinary roles can only change
>> + their own password.
>> </para>
>
> Having contemplated this a bit more, I don't like it, and it's not how
> things work when it comes to regular privileges.
>
> Consider that I can currently GRANT someone UPDATE privileges on an
> object, but they can't GRANT that privilege to someone else unless I
> explicitly allow it. The same could certainly be said for roles-
> perhaps I want to allow someone the privilege to create non-login roles,
> but I don't want them to be able to create new login roles, even if they
> themselves have LOGIN.
This comment conflates privileges like LOGIN for which there isn't any "with grant option" logic with privileges that do. Granting someone UPDATE privileges on a relation will be tracked in an aclitem including whether the "with grant option" bit is set. Nothing like that will exist for LOGIN. I'm not dead-set against having that functionality for the privileges that currently lack it, but we'd have to do so in a way that doesn't gratuitously break backward compatibility, and how to do so has not been discussed.
> As another point, I might want to have an 'admin' role that I want
> admins to SET ROLE to before they go creating other roles, because I
> don't want them to be creating roles as their regular user and so that
> those other roles are owned by the 'admin' role, but I don't want that
> role to have the 'login' attribute.
Same problem. We don't have aclitem bits for this.
> In other words, we should really consider what role attributes a given
> role has to be independent of what role attributes that role is allowed
> to set on roles they create. I appreciate that "just whatever the
> current role has" is simpler and less work but also will be difficult to
> walk back from once it's in the wild.
I don't feel there is any fundamental disagreement here, except perhaps whether it needs to be done as part of this patch, vs. implemented in a future development cycle. We don't currently have any syntax for "CREATE ROLE bob LOGIN WITH GRANT OPTION". I can see some advantages in doing it all in one go, but also some advantage in being incremental. More discussion is needed here.
>> @@ -1457,7 +1449,7 @@ AddRoleMems(const char *rolename, Oid roleid,
>
>> /*
>> - * Check permissions: must have createrole or admin option on the role to
>> + * Check permissions: must be owner or have admin option on the role to
>> * be changed. To mess with a superuser role, you gotta be superuser.
>> */
>> if (superuser_arg(roleid))
>
> ...
>
>> @@ -1467,9 +1459,9 @@ AddRoleMems(const char *rolename, Oid roleid,
>> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
>> errmsg("must be superuser to alter superusers")));
>> }
>> - else
>> + else if (!superuser())
>> {
>> - if (!have_createrole_privilege() &&
>> + if (!pg_role_ownercheck(roleid, grantorId) &&
>> !is_admin_of_role(grantorId, roleid))
>> ereport(ERROR,
>> (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
>
> I'm not entirely sure about including owners here though I'm not
> completely against it either. This conflation of what the 'admin'
> privileges on a role means vs. the 'ownership' of a role is part of what
> I dislike about having two distinct systems for saying who is allowed to
> GRANT one role to another.
>
> Also, if we're going to always consider owners to be admins of roles
> they own, why not push that into is_admin_of_role()?
Unchanged in this patch set, but worth further discussion and evaluation.
>> Subject: [PATCH v4 5/5] Remove grantor field from pg_auth_members
...<snip>...
> If we're going to do this, it should also be done independently of the
> role ownership stuff too.
I've withdrawn 0005 from this patch set, and we can come back to it separately.
> Thanks,
>
> Stephen
Thanks for the review! I hope we can keep pushing this forward. Again, no offense is intended in having not addressed all your concerns in the v7 patch set:
—
Mark Dilger
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company
Attachments:
[application/octet-stream] v7-0001-Add-owners-to-roles.patch (59.8K, ../../[email protected]/2-v7-0001-Add-owners-to-roles.patch)
download | inline diff:
From 08b59a68ed3a333bfb92de20f6ea74ccbc42b652 Mon Sep 17 00:00:00 2001
From: Mark Dilger <[email protected]>
Date: Tue, 25 Jan 2022 11:15:19 -0800
Subject: [PATCH v7 1/2] Add owners to roles
All roles now have owners. By default, created roles belong to the
role that created them, and initdb-time roles are owned by the
bootstrap superuser. Role ownership is transitive; if role A owns
role B, and role B owns role C, then role A can act as the owner of
role C. Roles may drop or alter roles that they own, and have all
privileges that any role they own has.
---
doc/src/sgml/ref/create_role.sgml | 23 ++-
src/backend/catalog/aclchk.c | 30 ++-
src/backend/catalog/objectaddress.c | 22 +--
src/backend/catalog/pg_shdepend.c | 5 +
src/backend/catalog/system_views.sql | 1 +
src/backend/commands/alter.c | 3 +
src/backend/commands/schemacmds.c | 10 +-
src/backend/commands/user.c | 180 +++++++++++++++++-
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/parser/gram.y | 19 ++
src/backend/utils/adt/acl.c | 118 ++++++++++++
src/bin/psql/describe.c | 12 ++
src/include/catalog/pg_authid.h | 1 +
src/include/commands/user.h | 2 +
src/include/nodes/parsenodes.h | 1 +
src/include/utils/acl.h | 2 +
.../expected/dummy_seclabel.out | 12 +-
.../dummy_seclabel/sql/dummy_seclabel.sql | 12 +-
.../unsafe_tests/expected/rolenames.out | 6 +-
.../modules/unsafe_tests/sql/rolenames.sql | 3 +-
src/test/regress/expected/create_role.out | 177 ++++++++++++++---
src/test/regress/expected/oidjoins.out | 1 +
src/test/regress/expected/privileges.out | 9 +-
src/test/regress/expected/rules.out | 1 +
src/test/regress/sql/create_role.sql | 100 ++++++++--
src/test/regress/sql/privileges.sql | 10 +-
27 files changed, 657 insertions(+), 105 deletions(-)
diff --git a/doc/src/sgml/ref/create_role.sgml b/doc/src/sgml/ref/create_role.sgml
index b6a4ea1f72..1e9347b2ce 100644
--- a/doc/src/sgml/ref/create_role.sgml
+++ b/doc/src/sgml/ref/create_role.sgml
@@ -21,9 +21,16 @@ PostgreSQL documentation
<refsynopsisdiv>
<synopsis>
-CREATE ROLE <replaceable class="parameter">name</replaceable> [ [ WITH ] <replaceable class="parameter">option</replaceable> [ ... ] ]
+CREATE ROLE <replaceable class="parameter">name</replaceable> [ AUTHORIZATION <replaceable class="parameter">role_specification</replaceable> ] [ [ WITH ] <replaceable class="parameter">option</replaceable> [ ... ] ]
-<phrase>where <replaceable class="parameter">option</replaceable> can be:</phrase>
+<phrase>where <replaceable class="parameter">role_specification</replaceable> can be:</phrase>
+
+ <replaceable class="parameter">user_name</replaceable>
+ | CURRENT_ROLE
+ | CURRENT_USER
+ | SESSION_USER
+
+<phrase>and <replaceable class="parameter">option</replaceable> can be:</phrase>
SUPERUSER | NOSUPERUSER
| CREATEDB | NOCREATEDB
@@ -83,6 +90,18 @@ in sync when changing the above synopsis!
</listitem>
</varlistentry>
+ <varlistentry>
+ <term><replaceable class="parameter">user_name</replaceable></term>
+ <listitem>
+ <para>
+ The role name of the user who will own the new role. If omitted,
+ defaults to the user executing the command. To create a role
+ owned by another role, you must be a direct or indirect member of
+ that role, directly or indirectly own that role, or be a superuser.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry>
<term><literal>SUPERUSER</literal></term>
<term><literal>NOSUPERUSER</literal></term>
diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c
index 1dd03a8e51..1e0ee503e4 100644
--- a/src/backend/catalog/aclchk.c
+++ b/src/backend/catalog/aclchk.c
@@ -3385,6 +3385,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype,
case OBJECT_PUBLICATION:
msg = gettext_noop("permission denied for publication %s");
break;
+ case OBJECT_ROLE:
+ msg = gettext_noop("permission denied for role %s");
+ break;
case OBJECT_ROUTINE:
msg = gettext_noop("permission denied for routine %s");
break;
@@ -3429,7 +3432,6 @@ aclcheck_error(AclResult aclerr, ObjectType objtype,
case OBJECT_DOMCONSTRAINT:
case OBJECT_PUBLICATION_NAMESPACE:
case OBJECT_PUBLICATION_REL:
- case OBJECT_ROLE:
case OBJECT_RULE:
case OBJECT_TABCONSTRAINT:
case OBJECT_TRANSFORM:
@@ -3511,6 +3513,9 @@ aclcheck_error(AclResult aclerr, ObjectType objtype,
case OBJECT_PUBLICATION:
msg = gettext_noop("must be owner of publication %s");
break;
+ case OBJECT_ROLE:
+ msg = gettext_noop("must be owner of role %s");
+ break;
case OBJECT_ROUTINE:
msg = gettext_noop("must be owner of routine %s");
break;
@@ -3569,7 +3574,6 @@ aclcheck_error(AclResult aclerr, ObjectType objtype,
case OBJECT_DOMCONSTRAINT:
case OBJECT_PUBLICATION_NAMESPACE:
case OBJECT_PUBLICATION_REL:
- case OBJECT_ROLE:
case OBJECT_TRANSFORM:
case OBJECT_TSPARSER:
case OBJECT_TSTEMPLATE:
@@ -5430,16 +5434,22 @@ pg_statistics_object_ownercheck(Oid stat_oid, Oid roleid)
return has_privs_of_role(roleid, ownerId);
}
+/*
+ * Ownership check for a role (specified by OID)
+ */
+bool
+pg_role_ownercheck(Oid owned_role_oid, Oid owner_roleid)
+{
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(owner_roleid))
+ return true;
+
+ /* Otherwise, check the role ownership hierarchy */
+ return is_owner_of_role_nosuper(owner_roleid, owned_role_oid);
+}
+
/*
* Check whether specified role has CREATEROLE privilege (or is a superuser)
- *
- * Note: roles do not have owners per se; instead we use this test in
- * places where an ownership-like permissions test is needed for a role.
- * Be sure to apply it to the role trying to do the operation, not the
- * role being operated on! Also note that this generally should not be
- * considered enough privilege if the target role is a superuser.
- * (We don't handle that consideration here because we want to give a
- * separate error message for such cases, so the caller has to deal with it.)
*/
bool
has_createrole_privilege(Oid roleid)
diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c
index f30c742d48..1a47f28829 100644
--- a/src/backend/catalog/objectaddress.c
+++ b/src/backend/catalog/objectaddress.c
@@ -2596,25 +2596,9 @@ check_object_ownership(Oid roleid, ObjectType objtype, ObjectAddress address,
NameListToString(castNode(List, object)));
break;
case OBJECT_ROLE:
-
- /*
- * We treat roles as being "owned" by those with CREATEROLE priv,
- * except that superusers are only owned by superusers.
- */
- if (superuser_arg(address.objectId))
- {
- if (!superuser_arg(roleid))
- ereport(ERROR,
- (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser")));
- }
- else
- {
- if (!has_createrole_privilege(roleid))
- ereport(ERROR,
- (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must have CREATEROLE privilege")));
- }
+ if (!pg_role_ownercheck(address.objectId, roleid))
+ aclcheck_error(ACLCHECK_NOT_OWNER, objtype,
+ strVal(object));
break;
case OBJECT_TSPARSER:
case OBJECT_TSTEMPLATE:
diff --git a/src/backend/catalog/pg_shdepend.c b/src/backend/catalog/pg_shdepend.c
index 3e8fa008b9..52f69ad76e 100644
--- a/src/backend/catalog/pg_shdepend.c
+++ b/src/backend/catalog/pg_shdepend.c
@@ -61,6 +61,7 @@
#include "commands/tablecmds.h"
#include "commands/tablespace.h"
#include "commands/typecmds.h"
+#include "commands/user.h"
#include "miscadmin.h"
#include "storage/lmgr.h"
#include "utils/acl.h"
@@ -1578,6 +1579,10 @@ shdepReassignOwned(List *roleids, Oid newrole)
AlterSubscriptionOwner_oid(sdepForm->objid, newrole);
break;
+ case AuthIdRelationId:
+ AlterRoleOwner_oid(sdepForm->objid, newrole);
+ break;
+
/* Generic alter owner cases */
case CollationRelationId:
case ConversionRelationId:
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 3cb69b1f87..057d17460b 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -17,6 +17,7 @@
CREATE VIEW pg_roles AS
SELECT
rolname,
+ pg_get_userbyid(rolowner) AS rolowner,
rolsuper,
rolinherit,
rolcreaterole,
diff --git a/src/backend/commands/alter.c b/src/backend/commands/alter.c
index 1f64c8aa51..e6c7c84c87 100644
--- a/src/backend/commands/alter.c
+++ b/src/backend/commands/alter.c
@@ -840,6 +840,9 @@ ExecAlterOwnerStmt(AlterOwnerStmt *stmt)
case OBJECT_DATABASE:
return AlterDatabaseOwner(strVal(stmt->object), newowner);
+ case OBJECT_ROLE:
+ return AlterRoleOwner(strVal(stmt->object), newowner);
+
case OBJECT_SCHEMA:
return AlterSchemaOwner(strVal(stmt->object), newowner);
diff --git a/src/backend/commands/schemacmds.c b/src/backend/commands/schemacmds.c
index 984000a5bc..320e8378c4 100644
--- a/src/backend/commands/schemacmds.c
+++ b/src/backend/commands/schemacmds.c
@@ -363,11 +363,11 @@ AlterSchemaOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
/*
* must have create-schema rights
*
- * NOTE: This is different from other alter-owner checks in that the
- * current user is checked for create privileges instead of the
- * destination owner. This is consistent with the CREATE case for
- * schemas. Because superusers will always have this right, we need
- * no special case for them.
+ * NOTE: alter-schema and alter-role are different from other
+ * alter-owner checks in that the current user is checked for create
+ * privileges instead of the destination owner. Alter-schema is
+ * consistent with the CREATE case for schemas. Because superusers
+ * will always have this right, we need no special case for them.
*/
aclresult = pg_database_aclcheck(MyDatabaseId, GetUserId(),
ACL_CREATE);
diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c
index f9d3c1246b..3726e40f36 100644
--- a/src/backend/commands/user.c
+++ b/src/backend/commands/user.c
@@ -55,6 +55,8 @@ static void AddRoleMems(const char *rolename, Oid roleid,
static void DelRoleMems(const char *rolename, Oid roleid,
List *memberSpecs, List *memberIds,
bool admin_opt);
+static void AlterRoleOwner_internal(HeapTuple tup, Relation rel,
+ Oid newOwnerId);
/* Check if current user has createrole privileges */
@@ -77,6 +79,9 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
Datum new_record[Natts_pg_authid];
bool new_record_nulls[Natts_pg_authid];
Oid roleid;
+ Oid owner_uid;
+ Oid saved_uid;
+ int save_sec_context;
ListCell *item;
ListCell *option;
char *password = NULL; /* user password */
@@ -108,6 +113,19 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
DefElem *dvalidUntil = NULL;
DefElem *dbypassRLS = NULL;
+ GetUserIdAndSecContext(&saved_uid, &save_sec_context);
+
+ /*
+ * Who is supposed to own the new role?
+ */
+ if (stmt->authrole)
+ {
+ owner_uid = get_rolespec_oid(stmt->authrole, false);
+ check_is_member_of_role(saved_uid, owner_uid);
+ }
+ else
+ owner_uid = saved_uid;
+
/* The defaults can vary depending on the original statement type */
switch (stmt->stmt_type)
{
@@ -254,6 +272,10 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to create superusers")));
+ if (!superuser_arg(owner_uid))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must be superuser to own superusers")));
}
else if (isreplication)
{
@@ -310,6 +332,19 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
errmsg("role \"%s\" already exists",
stmt->role)));
+ /*
+ * If the requested authorization is different from the current user,
+ * temporarily set the current user so that the object(s) will be created
+ * with the correct ownership.
+ *
+ * (The setting will be restored at the end of this routine, or in case of
+ * error, transaction abort will clean things up.)
+ */
+ if (saved_uid != owner_uid)
+ SetUserIdAndSecContext(owner_uid,
+ save_sec_context | SECURITY_LOCAL_USERID_CHANGE);
+
+
/* Convert validuntil to internal form */
if (validUntil)
{
@@ -345,6 +380,7 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
DirectFunctionCall1(namein, CStringGetDatum(stmt->role));
new_record[Anum_pg_authid_rolsuper - 1] = BoolGetDatum(issuper);
+ new_record[Anum_pg_authid_rolowner - 1] = ObjectIdGetDatum(owner_uid);
new_record[Anum_pg_authid_rolinherit - 1] = BoolGetDatum(inherit);
new_record[Anum_pg_authid_rolcreaterole - 1] = BoolGetDatum(createrole);
new_record[Anum_pg_authid_rolcreatedb - 1] = BoolGetDatum(createdb);
@@ -422,6 +458,8 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
*/
CatalogTupleInsert(pg_authid_rel, tuple);
+ recordDependencyOnOwner(AuthIdRelationId, roleid, owner_uid);
+
/*
* Advance command counter so we can see new record; else tests in
* AddRoleMems may fail.
@@ -478,6 +516,9 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
*/
table_close(pg_authid_rel, NoLock);
+ /* Reset current user and security context */
+ SetUserIdAndSecContext(saved_uid, save_sec_context);
+
return roleid;
}
@@ -655,7 +696,8 @@ AlterRole(ParseState *pstate, AlterRoleStmt *stmt)
{
/* check the rest */
if (dinherit || dcreaterole || dcreatedb || dcanlogin || dconnlimit ||
- drolemembers || dvalidUntil || !dpassword || roleid != GetUserId())
+ drolemembers || dvalidUntil || !dpassword ||
+ !pg_role_ownercheck(roleid, GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("permission denied")));
@@ -860,7 +902,8 @@ AlterRoleSet(AlterRoleSetStmt *stmt)
}
else
{
- if (!have_createrole_privilege() && roleid != GetUserId())
+ if (!have_createrole_privilege() &&
+ !pg_role_ownercheck(roleid, GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("permission denied")));
@@ -912,11 +955,6 @@ DropRole(DropRoleStmt *stmt)
pg_auth_members_rel;
ListCell *item;
- if (!have_createrole_privilege())
- ereport(ERROR,
- (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("permission denied to drop role")));
-
/*
* Scan the pg_authid relation to find the Oid of the role(s) to be
* deleted.
@@ -988,6 +1026,12 @@ DropRole(DropRoleStmt *stmt)
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to drop superusers")));
+ if (!have_createrole_privilege() &&
+ !pg_role_ownercheck(roleid, GetUserId()))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("permission denied to drop role")));
+
/* DROP hook for the role being removed */
InvokeObjectDropHook(AuthIdRelationId, roleid, 0);
@@ -1051,8 +1095,9 @@ DropRole(DropRoleStmt *stmt)
systable_endscan(sscan);
/*
- * Remove any comments or security labels on this role.
+ * Remove any dependencies, comments or security labels on this role.
*/
+ deleteSharedDependencyRecordsFor(AuthIdRelationId, roleid, 0);
DeleteSharedComments(roleid, AuthIdRelationId);
DeleteSharedSecurityLabel(roleid, AuthIdRelationId);
@@ -1648,3 +1693,122 @@ DelRoleMems(const char *rolename, Oid roleid,
*/
table_close(pg_authmem_rel, NoLock);
}
+
+/*
+ * Change role owner
+ */
+ObjectAddress
+AlterRoleOwner(const char *name, Oid newOwnerId)
+{
+ Oid roleid;
+ HeapTuple tup;
+ Relation rel;
+ ObjectAddress address;
+ Form_pg_authid authform;
+
+ rel = table_open(AuthIdRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(AUTHNAME, CStringGetDatum(name));
+ if (!HeapTupleIsValid(tup))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("role \"%s\" does not exist", name)));
+
+ authform = (Form_pg_authid) GETSTRUCT(tup);
+ roleid = authform->oid;
+
+ AlterRoleOwner_internal(tup, rel, newOwnerId);
+
+ ObjectAddressSet(address, AuthIdRelationId, roleid);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+
+ return address;
+}
+
+void
+AlterRoleOwner_oid(Oid roleOid, Oid newOwnerId)
+{
+ HeapTuple tup;
+ Relation rel;
+
+ rel = table_open(AuthIdRelationId, RowExclusiveLock);
+
+ tup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(roleOid));
+ if (!HeapTupleIsValid(tup))
+ elog(ERROR, "cache lookup failed for role %u", roleOid);
+
+ AlterRoleOwner_internal(tup, rel, newOwnerId);
+
+ ReleaseSysCache(tup);
+
+ table_close(rel, RowExclusiveLock);
+}
+
+static void
+AlterRoleOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
+{
+ Form_pg_authid authForm;
+
+ Assert(tup->t_tableOid == AuthIdRelationId);
+ Assert(RelationGetRelid(rel) == AuthIdRelationId);
+
+ authForm = (Form_pg_authid) GETSTRUCT(tup);
+
+ /*
+ * If the new owner is the same as the existing owner, consider the
+ * command to have succeeded. This is for dump restoration purposes.
+ */
+ if (authForm->rolowner != newOwnerId)
+ {
+ /* Otherwise, must be owner of the existing object */
+ if (!pg_role_ownercheck(authForm->oid, GetUserId()))
+ aclcheck_error(ACLCHECK_NOT_OWNER, OBJECT_ROLE,
+ NameStr(authForm->rolname));
+
+ /* Must be able to become new owner */
+ check_is_member_of_role(GetUserId(), newOwnerId);
+
+ /*
+ * must have CREATEROLE rights
+ *
+ * NOTE: Alter-role and alter-schema are different from other
+ * alter-owner checks in that the current user is checked for create
+ * privileges instead of the destination owner. Alter-role is
+ * consistent with the CREATE case for roles. Because superusers will
+ * always have this right, we need no special case for them.
+ */
+ if (!have_createrole_privilege())
+ aclcheck_error(ACLCHECK_NO_PRIV, OBJECT_ROLE,
+ NameStr(authForm->rolname));
+
+ /* Only the bootstrap superuser is allowed to own itself. */
+ if (newOwnerId != BOOTSTRAP_SUPERUSERID && authForm->oid == newOwnerId)
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("role may not own itself")));
+
+ /*
+ * Must not create cycles in the role ownership hierarchy. If this
+ * role owns (directly or indirectly) the proposed new owner, disallow
+ * the ownership transfer.
+ */
+ if (is_owner_of_role_nosuper(authForm->oid, newOwnerId))
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("role \"%s\" may not both own and be owned by role \"%s\"",
+ NameStr(authForm->rolname),
+ GetUserNameFromId(newOwnerId, false))));
+
+ authForm->rolowner = newOwnerId;
+ CatalogTupleUpdate(rel, &tup->t_self, tup);
+
+ /* Update owner dependency reference */
+ changeDependencyOnOwner(AuthIdRelationId, authForm->oid, newOwnerId);
+ }
+
+ InvokeObjectPostAlterHook(AuthIdRelationId,
+ authForm->oid, 0);
+}
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 90b5da51c9..f43c92a70f 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -4522,6 +4522,7 @@ _copyCreateRoleStmt(const CreateRoleStmt *from)
COPY_SCALAR_FIELD(stmt_type);
COPY_STRING_FIELD(role);
+ COPY_NODE_FIELD(authrole);
COPY_NODE_FIELD(options);
return newnode;
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 06345da3ba..328f155208 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -2128,6 +2128,7 @@ _equalCreateRoleStmt(const CreateRoleStmt *a, const CreateRoleStmt *b)
{
COMPARE_SCALAR_FIELD(stmt_type);
COMPARE_STRING_FIELD(role);
+ COMPARE_NODE_FIELD(authrole);
COMPARE_NODE_FIELD(options);
return true;
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b5966712ce..024d96470f 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -1077,9 +1077,20 @@ CreateRoleStmt:
CreateRoleStmt *n = makeNode(CreateRoleStmt);
n->stmt_type = ROLESTMT_ROLE;
n->role = $3;
+ n->authrole = NULL;
n->options = $5;
$$ = (Node *)n;
}
+ |
+ CREATE ROLE RoleId AUTHORIZATION RoleSpec opt_with OptRoleList
+ {
+ CreateRoleStmt *n = makeNode(CreateRoleStmt);
+ n->stmt_type = ROLESTMT_ROLE;
+ n->role = $3;
+ n->authrole = $5;
+ n->options = $7;
+ $$ = (Node *)n;
+ }
;
@@ -9585,6 +9596,14 @@ AlterOwnerStmt: ALTER AGGREGATE aggregate_with_argtypes OWNER TO RoleSpec
n->newowner = $6;
$$ = (Node *)n;
}
+ | ALTER ROLE name OWNER TO RoleSpec
+ {
+ AlterOwnerStmt *n = makeNode(AlterOwnerStmt);
+ n->objectType = OBJECT_ROLE;
+ n->object = (Node *) makeString($3);
+ n->newowner = $6;
+ $$ = (Node *)n;
+ }
| ALTER ROUTINE function_with_argtypes OWNER TO RoleSpec
{
AlterOwnerStmt *n = makeNode(AlterOwnerStmt);
diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c
index 0a16f8156c..97336db058 100644
--- a/src/backend/utils/adt/acl.c
+++ b/src/backend/utils/adt/acl.c
@@ -4832,6 +4832,111 @@ roles_is_member_of(Oid roleid, enum RoleRecurseType type,
}
+/*
+ * Get a list of roles which own the given role, directly or indirectly.
+ *
+ * Each role has only one direct owner. The returned list contains the given
+ * role's owner, that role's owner, etc., up to the top of the ownership
+ * hierarchy, which is always the bootstrap superuser.
+ *
+ * Raises an error if any role ownership invariant is violated. Returns NIL if
+ * the given roleid is invalid.
+ */
+static List *
+roles_is_owned_by(Oid roleid)
+{
+ List *owners_list = NIL;
+ Oid role_oid = roleid;
+
+ /*
+ * Start with the current role and follow the ownership chain upwards until
+ * we reach the bootstrap superuser. To defend against getting into an
+ * infinite loop, we must check for ownership cycles. We choose to perform
+ * other corruption checks on the ownership structure while iterating, too.
+ */
+ while (OidIsValid(role_oid))
+ {
+ HeapTuple tuple;
+ Form_pg_authid authform;
+ Oid owner_oid;
+
+ /* Find the owner of the current iteration's role */
+ tuple = SearchSysCache1(AUTHOID, ObjectIdGetDatum(role_oid));
+ if (!HeapTupleIsValid(tuple))
+ ereport(ERROR,
+ (errcode(ERRCODE_UNDEFINED_OBJECT),
+ errmsg("role with OID %u does not exist", role_oid)));
+
+ authform = (Form_pg_authid) GETSTRUCT(tuple);
+ owner_oid = authform->rolowner;
+
+ /*
+ * Roles must necessarily have owners. Even the bootstrap user has an
+ * owner. (It owns itself).
+ */
+ if (!OidIsValid(owner_oid))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("role \"%s\" with OID %u has invalid owner",
+ NameStr(authform->rolname), authform->oid)));
+
+ /* The bootstrap user must own itself */
+ if (authform->oid == BOOTSTRAP_SUPERUSERID &&
+ owner_oid != BOOTSTRAP_SUPERUSERID)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("role \"%s\" with OID %u owned by role with OID %u",
+ NameStr(authform->rolname), authform->oid,
+ authform->rolowner)));
+
+ /*
+ * Roles other than the bootstrap user must not be their own direct
+ * owners.
+ */
+ if (authform->oid != BOOTSTRAP_SUPERUSERID &&
+ authform->oid == owner_oid)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("role \"%s\" with OID %u owns itself",
+ NameStr(authform->rolname), authform->oid)));
+
+ ReleaseSysCache(tuple);
+
+ /* If we have reached the bootstrap user, we're done. */
+ if (role_oid == BOOTSTRAP_SUPERUSERID)
+ {
+ if (!owners_list)
+ owners_list = lappend_oid(owners_list, owner_oid);
+ break;
+ }
+
+ /*
+ * For all other users, check they do not own themselves indirectly
+ * through an ownership cycle.
+ *
+ * Scanning the list each time through this loop results in overall
+ * quadratic work in the depth of the ownership chain, but we're
+ * not on a critical performance path, nor do we expect ownership
+ * hierarchies to be deep.
+ */
+ if (owners_list && list_member_oid(owners_list,
+ ObjectIdGetDatum(owner_oid)))
+ ereport(ERROR,
+ (errcode(ERRCODE_DATA_CORRUPTED),
+ errmsg("role \"%s\" with OID %u indirectly owns itself",
+ GetUserNameFromId(owner_oid, false),
+ owner_oid)));
+
+ /* Done with sanity checks. Add this owner to the list. */
+ owners_list = lappend_oid(owners_list, owner_oid);
+
+ /* Otherwise, iterate on this iteration's owner_oid. */
+ role_oid = owner_oid;
+ }
+
+ return owners_list;
+}
+
/*
* Does member have the privileges of role (directly or indirectly)?
*
@@ -4850,6 +4955,10 @@ has_privs_of_role(Oid member, Oid role)
if (superuser_arg(member))
return true;
+ /* Owners of roles have every privilege the owned role has */
+ if (pg_role_ownercheck(role, member))
+ return true;
+
/*
* Find all the roles that member has the privileges of, including
* multi-level recursion, then see if target role is any one of them.
@@ -4921,6 +5030,15 @@ is_member_of_role_nosuper(Oid member, Oid role)
role);
}
+/*
+ * Is owner a direct or indirect owner of the role, not considering
+ * superuserness?
+ */
+bool
+is_owner_of_role_nosuper(Oid owner, Oid role)
+{
+ return list_member_oid(roles_is_owned_by(role), owner);
+}
/*
* Is member an admin of role? That is, is member the role itself (subject to
diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c
index 346cd92793..406138609d 100644
--- a/src/bin/psql/describe.c
+++ b/src/bin/psql/describe.c
@@ -3518,6 +3518,12 @@ describeRoles(const char *pattern, bool verbose, bool showSystem)
appendPQExpBufferStr(&buf, "\n, r.rolbypassrls");
}
+ if (pset.sversion >= 150000)
+ {
+ appendPQExpBufferStr(&buf, "\n, r.rolowner");
+ ncols++;
+ }
+
appendPQExpBufferStr(&buf, "\nFROM pg_catalog.pg_roles r\n");
if (!showSystem && !pattern)
@@ -3538,6 +3544,8 @@ describeRoles(const char *pattern, bool verbose, bool showSystem)
printTableInit(&cont, &myopt, _("List of roles"), ncols, nrows);
printTableAddHeader(&cont, gettext_noop("Role name"), true, align);
+ if (pset.sversion >= 150000)
+ printTableAddHeader(&cont, gettext_noop("Owner"), true, align);
printTableAddHeader(&cont, gettext_noop("Attributes"), true, align);
/* ignores implicit memberships from superuser & pg_database_owner */
printTableAddHeader(&cont, gettext_noop("Member of"), true, align);
@@ -3549,6 +3557,10 @@ describeRoles(const char *pattern, bool verbose, bool showSystem)
{
printTableAddCell(&cont, PQgetvalue(res, i, 0), false, false);
+ if (pset.sversion >= 150000)
+ printTableAddCell(&cont, PQgetvalue(res, i, (verbose ? 12 : 11)),
+ false, false);
+
resetPQExpBuffer(&buf);
if (strcmp(PQgetvalue(res, i, 1), "t") == 0)
add_role_attribute(&buf, _("Superuser"));
diff --git a/src/include/catalog/pg_authid.h b/src/include/catalog/pg_authid.h
index 4b65e39a1f..3af0f3908c 100644
--- a/src/include/catalog/pg_authid.h
+++ b/src/include/catalog/pg_authid.h
@@ -32,6 +32,7 @@ CATALOG(pg_authid,1260,AuthIdRelationId) BKI_SHARED_RELATION BKI_ROWTYPE_OID(284
{
Oid oid; /* oid */
NameData rolname; /* name of role */
+ Oid rolowner BKI_DEFAULT(POSTGRES) BKI_LOOKUP(pg_authid); /* owner of this role */
bool rolsuper; /* read this field via superuser() only! */
bool rolinherit; /* inherit privileges from other roles? */
bool rolcreaterole; /* allowed to create more roles? */
diff --git a/src/include/commands/user.h b/src/include/commands/user.h
index 0b7a3cd65f..c32127e41e 100644
--- a/src/include/commands/user.h
+++ b/src/include/commands/user.h
@@ -33,5 +33,7 @@ extern ObjectAddress RenameRole(const char *oldname, const char *newname);
extern void DropOwnedObjects(DropOwnedStmt *stmt);
extern void ReassignOwnedObjects(ReassignOwnedStmt *stmt);
extern List *roleSpecsToIds(List *memberNames);
+extern ObjectAddress AlterRoleOwner(const char *name, Oid newOwnerId);
+extern void AlterRoleOwner_oid(Oid roleOid, Oid newOwnerId);
#endif /* USER_H */
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index 3e9bdc781f..b9d124d38f 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -2623,6 +2623,7 @@ typedef struct CreateRoleStmt
NodeTag type;
RoleStmtType stmt_type; /* ROLE/USER/GROUP */
char *role; /* role name */
+ RoleSpec *authrole; /* the owner of the created role */
List *options; /* List of DefElem nodes */
} CreateRoleStmt;
diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h
index 1ce4c5556e..572cae0f27 100644
--- a/src/include/utils/acl.h
+++ b/src/include/utils/acl.h
@@ -209,6 +209,7 @@ extern bool has_privs_of_role(Oid member, Oid role);
extern bool is_member_of_role(Oid member, Oid role);
extern bool is_member_of_role_nosuper(Oid member, Oid role);
extern bool is_admin_of_role(Oid member, Oid role);
+extern bool is_owner_of_role_nosuper(Oid owner, Oid role);
extern void check_is_member_of_role(Oid member, Oid role);
extern Oid get_role_oid(const char *rolename, bool missing_ok);
extern Oid get_role_oid_or_public(const char *rolename);
@@ -316,6 +317,7 @@ extern bool pg_extension_ownercheck(Oid ext_oid, Oid roleid);
extern bool pg_publication_ownercheck(Oid pub_oid, Oid roleid);
extern bool pg_subscription_ownercheck(Oid sub_oid, Oid roleid);
extern bool pg_statistics_object_ownercheck(Oid stat_oid, Oid roleid);
+extern bool pg_role_ownercheck(Oid owned_role_oid, Oid owner_roleid);
extern bool has_createrole_privilege(Oid roleid);
extern bool has_bypassrls_privilege(Oid roleid);
diff --git a/src/test/modules/dummy_seclabel/expected/dummy_seclabel.out b/src/test/modules/dummy_seclabel/expected/dummy_seclabel.out
index b2d898a7d1..93cf82b750 100644
--- a/src/test/modules/dummy_seclabel/expected/dummy_seclabel.out
+++ b/src/test/modules/dummy_seclabel/expected/dummy_seclabel.out
@@ -7,8 +7,11 @@ SET client_min_messages TO 'warning';
DROP ROLE IF EXISTS regress_dummy_seclabel_user1;
DROP ROLE IF EXISTS regress_dummy_seclabel_user2;
RESET client_min_messages;
-CREATE USER regress_dummy_seclabel_user1 WITH CREATEROLE;
+CREATE USER regress_dummy_seclabel_user0 WITH CREATEROLE;
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user0;
+CREATE USER regress_dummy_seclabel_user1;
CREATE USER regress_dummy_seclabel_user2;
+RESET SESSION AUTHORIZATION;
CREATE TABLE dummy_seclabel_tbl1 (a int, b text);
CREATE TABLE dummy_seclabel_tbl2 (x int, y text);
CREATE VIEW dummy_seclabel_view1 AS SELECT * FROM dummy_seclabel_tbl2;
@@ -19,7 +22,7 @@ ALTER TABLE dummy_seclabel_tbl2 OWNER TO regress_dummy_seclabel_user2;
--
-- Test of SECURITY LABEL statement with a plugin
--
-SET SESSION AUTHORIZATION regress_dummy_seclabel_user1;
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user0;
SECURITY LABEL ON TABLE dummy_seclabel_tbl1 IS 'classified'; -- OK
SECURITY LABEL ON COLUMN dummy_seclabel_tbl1.a IS 'unclassified'; -- OK
SECURITY LABEL ON COLUMN dummy_seclabel_tbl1 IS 'unclassified'; -- fail
@@ -29,6 +32,7 @@ ERROR: '...invalid label...' is not a valid security label
SECURITY LABEL FOR 'dummy' ON TABLE dummy_seclabel_tbl1 IS 'unclassified'; -- OK
SECURITY LABEL FOR 'unknown_seclabel' ON TABLE dummy_seclabel_tbl1 IS 'classified'; -- fail
ERROR: security label provider "unknown_seclabel" is not loaded
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user1;
SECURITY LABEL ON TABLE dummy_seclabel_tbl2 IS 'unclassified'; -- fail (not owner)
ERROR: must be owner of table dummy_seclabel_tbl2
SECURITY LABEL ON TABLE dummy_seclabel_tbl1 IS 'secret'; -- fail (not superuser)
@@ -42,7 +46,7 @@ SECURITY LABEL ON TABLE dummy_seclabel_tbl2 IS 'classified'; -- OK
--
-- Test for shared database object
--
-SET SESSION AUTHORIZATION regress_dummy_seclabel_user1;
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user0;
SECURITY LABEL ON ROLE regress_dummy_seclabel_user1 IS 'classified'; -- OK
SECURITY LABEL ON ROLE regress_dummy_seclabel_user1 IS '...invalid label...'; -- fail
ERROR: '...invalid label...' is not a valid security label
@@ -55,7 +59,7 @@ SECURITY LABEL ON ROLE regress_dummy_seclabel_user3 IS 'unclassified'; -- fail (
ERROR: role "regress_dummy_seclabel_user3" does not exist
SET SESSION AUTHORIZATION regress_dummy_seclabel_user2;
SECURITY LABEL ON ROLE regress_dummy_seclabel_user2 IS 'unclassified'; -- fail (not privileged)
-ERROR: must have CREATEROLE privilege
+ERROR: must be owner of role regress_dummy_seclabel_user2
RESET SESSION AUTHORIZATION;
--
-- Test for various types of object
diff --git a/src/test/modules/dummy_seclabel/sql/dummy_seclabel.sql b/src/test/modules/dummy_seclabel/sql/dummy_seclabel.sql
index 8c347b6a68..bf575343cf 100644
--- a/src/test/modules/dummy_seclabel/sql/dummy_seclabel.sql
+++ b/src/test/modules/dummy_seclabel/sql/dummy_seclabel.sql
@@ -11,8 +11,12 @@ DROP ROLE IF EXISTS regress_dummy_seclabel_user2;
RESET client_min_messages;
-CREATE USER regress_dummy_seclabel_user1 WITH CREATEROLE;
+CREATE USER regress_dummy_seclabel_user0 WITH CREATEROLE;
+
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user0;
+CREATE USER regress_dummy_seclabel_user1;
CREATE USER regress_dummy_seclabel_user2;
+RESET SESSION AUTHORIZATION;
CREATE TABLE dummy_seclabel_tbl1 (a int, b text);
CREATE TABLE dummy_seclabel_tbl2 (x int, y text);
@@ -26,7 +30,7 @@ ALTER TABLE dummy_seclabel_tbl2 OWNER TO regress_dummy_seclabel_user2;
--
-- Test of SECURITY LABEL statement with a plugin
--
-SET SESSION AUTHORIZATION regress_dummy_seclabel_user1;
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user0;
SECURITY LABEL ON TABLE dummy_seclabel_tbl1 IS 'classified'; -- OK
SECURITY LABEL ON COLUMN dummy_seclabel_tbl1.a IS 'unclassified'; -- OK
@@ -34,6 +38,8 @@ SECURITY LABEL ON COLUMN dummy_seclabel_tbl1 IS 'unclassified'; -- fail
SECURITY LABEL ON TABLE dummy_seclabel_tbl1 IS '...invalid label...'; -- fail
SECURITY LABEL FOR 'dummy' ON TABLE dummy_seclabel_tbl1 IS 'unclassified'; -- OK
SECURITY LABEL FOR 'unknown_seclabel' ON TABLE dummy_seclabel_tbl1 IS 'classified'; -- fail
+
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user1;
SECURITY LABEL ON TABLE dummy_seclabel_tbl2 IS 'unclassified'; -- fail (not owner)
SECURITY LABEL ON TABLE dummy_seclabel_tbl1 IS 'secret'; -- fail (not superuser)
SECURITY LABEL ON TABLE dummy_seclabel_tbl3 IS 'unclassified'; -- fail (not found)
@@ -45,7 +51,7 @@ SECURITY LABEL ON TABLE dummy_seclabel_tbl2 IS 'classified'; -- OK
--
-- Test for shared database object
--
-SET SESSION AUTHORIZATION regress_dummy_seclabel_user1;
+SET SESSION AUTHORIZATION regress_dummy_seclabel_user0;
SECURITY LABEL ON ROLE regress_dummy_seclabel_user1 IS 'classified'; -- OK
SECURITY LABEL ON ROLE regress_dummy_seclabel_user1 IS '...invalid label...'; -- fail
diff --git a/src/test/modules/unsafe_tests/expected/rolenames.out b/src/test/modules/unsafe_tests/expected/rolenames.out
index eb608fdc2e..8b79a63b80 100644
--- a/src/test/modules/unsafe_tests/expected/rolenames.out
+++ b/src/test/modules/unsafe_tests/expected/rolenames.out
@@ -1086,6 +1086,10 @@ REVOKE pg_read_all_settings FROM regress_role_haspriv;
\c
DROP SCHEMA test_roles_schema;
DROP OWNED BY regress_testrol0, "Public", "current_role", "current_user", regress_testrol1, regress_testrol2, regress_testrolx CASCADE;
-DROP ROLE regress_testrol0, regress_testrol1, regress_testrol2, regress_testrolx;
+DROP ROLE regress_testrol0, regress_testrol1, regress_testrol2, regress_testrolx; -- fails with owner of role regress_role_haspriv
+ERROR: role "regress_testrol2" cannot be dropped because some objects depend on it
+DETAIL: owner of role regress_role_haspriv
+owner of role regress_role_nopriv
DROP ROLE "Public", "None", "current_role", "current_user", "session_user", "user";
DROP ROLE regress_role_haspriv, regress_role_nopriv;
+DROP ROLE regress_testrol0, regress_testrol1, regress_testrol2, regress_testrolx; -- ok now
diff --git a/src/test/modules/unsafe_tests/sql/rolenames.sql b/src/test/modules/unsafe_tests/sql/rolenames.sql
index adac36536d..95a54ce70d 100644
--- a/src/test/modules/unsafe_tests/sql/rolenames.sql
+++ b/src/test/modules/unsafe_tests/sql/rolenames.sql
@@ -499,6 +499,7 @@ REVOKE pg_read_all_settings FROM regress_role_haspriv;
DROP SCHEMA test_roles_schema;
DROP OWNED BY regress_testrol0, "Public", "current_role", "current_user", regress_testrol1, regress_testrol2, regress_testrolx CASCADE;
-DROP ROLE regress_testrol0, regress_testrol1, regress_testrol2, regress_testrolx;
+DROP ROLE regress_testrol0, regress_testrol1, regress_testrol2, regress_testrolx; -- fails with owner of role regress_role_haspriv
DROP ROLE "Public", "None", "current_role", "current_user", "session_user", "user";
DROP ROLE regress_role_haspriv, regress_role_nopriv;
+DROP ROLE regress_testrol0, regress_testrol1, regress_testrol2, regress_testrolx; -- ok now
diff --git a/src/test/regress/expected/create_role.out b/src/test/regress/expected/create_role.out
index 4e67d72760..4ad6fd4161 100644
--- a/src/test/regress/expected/create_role.out
+++ b/src/test/regress/expected/create_role.out
@@ -1,6 +1,8 @@
-- ok, superuser can create users with any set of privileges
CREATE ROLE regress_role_super SUPERUSER;
+CREATE ROLE regress_role_bystander;
CREATE ROLE regress_role_admin CREATEDB CREATEROLE REPLICATION BYPASSRLS;
+GRANT CREATE ON DATABASE regression TO regress_role_admin;
-- fail, only superusers can create users with these privileges
SET SESSION AUTHORIZATION regress_role_admin;
CREATE ROLE regress_nosuch_superuser SUPERUSER;
@@ -11,14 +13,108 @@ CREATE ROLE regress_nosuch_replication REPLICATION;
ERROR: must be superuser to create replication users
CREATE ROLE regress_nosuch_bypassrls BYPASSRLS;
ERROR: must be superuser to create bypassrls users
+-- fail, only superusers can own superusers
+RESET SESSION AUTHORIZATION;
+CREATE ROLE regress_nosuch_superuser AUTHORIZATION regress_role_admin SUPERUSER;
+ERROR: must be superuser to own superusers
+-- ok, superuser can create superusers belonging to other superusers
+CREATE ROLE regress_superuser AUTHORIZATION regress_role_super SUPERUSER;
+-- fail, can only create roles belonging to other roles that we belong to
+SET SESSION AUTHORIZATION regress_role_admin;
+CREATE ROLE regress_nosuch_alice AUTHORIZATION regress_role_super;
+ERROR: must be member of role "regress_role_super"
+CREATE ROLE regress_nosuch_bob AUTHORIZATION regress_superuser;
+ERROR: must be member of role "regress_superuser"
+CREATE ROLE regress_nosuch_charlie AUTHORIZATION regress_role_bystander;
+ERROR: must be member of role "regress_role_bystander"
+-- ok, superuser can create users with these privileges for normal role
+RESET SESSION AUTHORIZATION;
+CREATE ROLE regress_replication_bypassrls AUTHORIZATION regress_role_admin REPLICATION BYPASSRLS;
+CREATE ROLE regress_replication AUTHORIZATION regress_role_admin REPLICATION;
+CREATE ROLE regress_bypassrls AUTHORIZATION regress_role_admin BYPASSRLS;
+\du+ regress_superuser
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+-------------------+--------------------+-------------------------+-----------+-------------
+ regress_superuser | regress_role_super | Superuser, Cannot login | {} |
+
+\du+ regress_replication_bypassrls
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+-------------------------------+--------------------+---------------------------------------+-----------+-------------
+ regress_replication_bypassrls | regress_role_admin | Cannot login, Replication, Bypass RLS | {} |
+
+\du+ regress_replication
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+---------------------+--------------------+---------------------------+-----------+-------------
+ regress_replication | regress_role_admin | Cannot login, Replication | {} |
+
+\du+ regress_bypassrls
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+-------------------+--------------------+--------------------------+-----------+-------------
+ regress_bypassrls | regress_role_admin | Cannot login, Bypass RLS | {} |
+
+-- fail, roles are not allowed to own themselves
+ALTER ROLE regress_bypassrls OWNER TO regress_bypassrls;
+ERROR: role may not own itself
-- ok, having CREATEROLE is enough to create users with these privileges
+SET SESSION AUTHORIZATION regress_role_admin;
CREATE ROLE regress_createdb CREATEDB;
CREATE ROLE regress_createrole CREATEROLE;
CREATE ROLE regress_login LOGIN;
CREATE ROLE regress_inherit INHERIT;
CREATE ROLE regress_connection_limit CONNECTION LIMIT 5;
-CREATE ROLE regress_encrypted_password ENCRYPTED PASSWORD 'foo';
-CREATE ROLE regress_password_null PASSWORD NULL;
+CREATE ROLE regress_encrypted_password PASSWORD NULL;
+CREATE ROLE regress_password_null
+ CREATEDB CREATEROLE INHERIT CONNECTION LIMIT 2 ENCRYPTED PASSWORD 'foo'
+ IN ROLE regress_createdb, regress_createrole;
+COMMENT ON ROLE regress_password_null IS 'no login test role';
+\du+ regress_createdb
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+------------------+--------------------+-------------------------+-----------+-------------
+ regress_createdb | regress_role_admin | Create DB, Cannot login | {} |
+
+\du+ regress_createrole
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+--------------------+--------------------+---------------------------+-----------+-------------
+ regress_createrole | regress_role_admin | Create role, Cannot login | {} |
+
+\du+ regress_login
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+---------------+--------------------+------------+-----------+-------------
+ regress_login | regress_role_admin | | {} |
+
+\du+ regress_inherit
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+-----------------+--------------------+--------------+-----------+-------------
+ regress_inherit | regress_role_admin | Cannot login | {} |
+
+\du+ regress_connection_limit
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+--------------------------+--------------------+---------------+-----------+-------------
+ regress_connection_limit | regress_role_admin | Cannot login +| {} |
+ | | 5 connections | |
+
+\du+ regress_encrypted_password
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+----------------------------+--------------------+--------------+-----------+-------------
+ regress_encrypted_password | regress_role_admin | Cannot login | {} |
+
+\du+ regress_password_null
+ List of roles
+ Role name | Owner | Attributes | Member of | Description
+-----------------------+--------------------+--------------------------------------+---------------------------------------+--------------------
+ regress_password_null | regress_role_admin | Create role, Create DB, Cannot login+| {regress_createdb,regress_createrole} | no login test role
+ | | 2 connections | |
+
-- ok, backwards compatible noise words should be ignored
CREATE ROLE regress_noiseword SYSID 12345;
NOTICE: SYSID can no longer be specified
@@ -58,21 +154,18 @@ CREATE TABLE tenant_table (i integer);
CREATE INDEX tenant_idx ON tenant_table(i);
CREATE VIEW tenant_view AS SELECT * FROM pg_catalog.pg_class;
REVOKE ALL PRIVILEGES ON tenant_table FROM PUBLIC;
--- fail, these objects belonging to regress_tenant
+-- ok, owning role can manage owned role's objects
SET SESSION AUTHORIZATION regress_createrole;
DROP INDEX tenant_idx;
-ERROR: must be owner of index tenant_idx
ALTER TABLE tenant_table ADD COLUMN t text;
-ERROR: must be owner of table tenant_table
DROP TABLE tenant_table;
-ERROR: must be owner of table tenant_table
+-- fail, not a member of target role
ALTER VIEW tenant_view OWNER TO regress_role_admin;
-ERROR: must be owner of view tenant_view
+ERROR: must be member of role "regress_role_admin"
+-- ok
DROP VIEW tenant_view;
-ERROR: must be owner of view tenant_view
--- fail, cannot take ownership of these objects from regress_tenant
+-- ok, can take ownership objects from owned roles
REASSIGN OWNED BY regress_tenant TO regress_createrole;
-ERROR: permission denied to reassign objects
-- ok, having CREATEROLE is enough to create roles in privileged roles
CREATE ROLE regress_read_all_data IN ROLE pg_read_all_data;
CREATE ROLE regress_write_all_data IN ROLE pg_write_all_data;
@@ -84,16 +177,24 @@ CREATE ROLE regress_read_server_files IN ROLE pg_read_server_files;
CREATE ROLE regress_write_server_files IN ROLE pg_write_server_files;
CREATE ROLE regress_execute_server_program IN ROLE pg_execute_server_program;
CREATE ROLE regress_signal_backend IN ROLE pg_signal_backend;
--- fail, creation of these roles failed above so they do not now exist
+-- fail, cannot create ownership cycles
+RESET SESSION AUTHORIZATION;
+REASSIGN OWNED BY regress_role_admin TO regress_tenant;
+ERROR: role "regress_createrole" may not both own and be owned by role "regress_tenant"
+ALTER ROLE regress_role_admin OWNER TO regress_tenant;
+ERROR: role "regress_role_admin" may not both own and be owned by role "regress_tenant"
+-- ok, can take ownership from owned roles
+SET SESSION AUTHORIZATION regress_role_admin;
+ALTER ROLE regress_plainrole OWNER TO regress_role_admin;
+REASSIGN OWNED BY regress_plainrole TO regress_role_admin;
+-- ok, superuser roles can drop superuser roles they own
+SET SESSION AUTHORIZATION regress_role_super;
+DROP ROLE regress_superuser;
+-- ok, non-superuser roles can drop non-superuser roles they own
SET SESSION AUTHORIZATION regress_role_admin;
-DROP ROLE regress_nosuch_superuser;
-ERROR: role "regress_nosuch_superuser" does not exist
-DROP ROLE regress_nosuch_replication_bypassrls;
-ERROR: role "regress_nosuch_replication_bypassrls" does not exist
-DROP ROLE regress_nosuch_replication;
-ERROR: role "regress_nosuch_replication" does not exist
-DROP ROLE regress_nosuch_bypassrls;
-ERROR: role "regress_nosuch_bypassrls" does not exist
+DROP ROLE regress_replication_bypassrls;
+DROP ROLE regress_replication;
+DROP ROLE regress_bypassrls;
DROP ROLE regress_nosuch_super;
ERROR: role "regress_nosuch_super" does not exist
DROP ROLE regress_nosuch_dbowner;
@@ -103,9 +204,23 @@ ERROR: role "regress_nosuch_recursive" does not exist
DROP ROLE regress_nosuch_admin_recursive;
ERROR: role "regress_nosuch_admin_recursive" does not exist
DROP ROLE regress_plainrole;
--- ok, should be able to drop non-superuser roles we created
-DROP ROLE regress_createdb;
+-- fail, cannot drop roles that own other roles
DROP ROLE regress_createrole;
+ERROR: role "regress_createrole" cannot be dropped because some objects depend on it
+DETAIL: owner of role regress_rolecreator
+owner of role regress_tenant
+owner of role regress_read_all_data
+owner of role regress_write_all_data
+owner of role regress_monitor
+owner of role regress_read_all_settings
+owner of role regress_read_all_stats
+owner of role regress_stat_scan_tables
+owner of role regress_read_server_files
+owner of role regress_write_server_files
+owner of role regress_execute_server_program
+owner of role regress_signal_backend
+-- ok, should be able to drop these non-superuser roles
+DROP ROLE regress_createdb;
DROP ROLE regress_login;
DROP ROLE regress_inherit;
DROP ROLE regress_connection_limit;
@@ -115,6 +230,7 @@ DROP ROLE regress_noiseword;
DROP ROLE regress_inroles;
DROP ROLE regress_adminroles;
DROP ROLE regress_rolecreator;
+DROP ROLE regress_tenant;
DROP ROLE regress_read_all_data;
DROP ROLE regress_write_all_data;
DROP ROLE regress_monitor;
@@ -125,21 +241,20 @@ DROP ROLE regress_read_server_files;
DROP ROLE regress_write_server_files;
DROP ROLE regress_execute_server_program;
DROP ROLE regress_signal_backend;
--- fail, role still owns database objects
-DROP ROLE regress_tenant;
-ERROR: role "regress_tenant" cannot be dropped because some objects depend on it
-DETAIL: owner of table tenant_table
-owner of view tenant_view
-- fail, cannot drop ourself nor superusers
DROP ROLE regress_role_super;
ERROR: must be superuser to drop superusers
DROP ROLE regress_role_admin;
ERROR: current user cannot be dropped
--- ok
+-- ok, no more owned roles remain
+DROP ROLE regress_createrole;
+-- fail, cannot drop role with remaining privileges
RESET SESSION AUTHORIZATION;
-DROP INDEX tenant_idx;
-DROP TABLE tenant_table;
-DROP VIEW tenant_view;
-DROP ROLE regress_tenant;
DROP ROLE regress_role_admin;
+ERROR: role "regress_role_admin" cannot be dropped because some objects depend on it
+DETAIL: privileges for database regression
+-- ok, can drop role if we revoke privileges first
+REVOKE CREATE ON DATABASE regression FROM regress_role_admin;
+DROP ROLE regress_role_admin;
+DROP ROLE regress_role_bystander;
DROP ROLE regress_role_super;
diff --git a/src/test/regress/expected/oidjoins.out b/src/test/regress/expected/oidjoins.out
index 215eb899be..266a30a85b 100644
--- a/src/test/regress/expected/oidjoins.out
+++ b/src/test/regress/expected/oidjoins.out
@@ -194,6 +194,7 @@ NOTICE: checking pg_database {dattablespace} => pg_tablespace {oid}
NOTICE: checking pg_db_role_setting {setdatabase} => pg_database {oid}
NOTICE: checking pg_db_role_setting {setrole} => pg_authid {oid}
NOTICE: checking pg_tablespace {spcowner} => pg_authid {oid}
+NOTICE: checking pg_authid {rolowner} => pg_authid {oid}
NOTICE: checking pg_auth_members {roleid} => pg_authid {oid}
NOTICE: checking pg_auth_members {member} => pg_authid {oid}
NOTICE: checking pg_auth_members {grantor} => pg_authid {oid}
diff --git a/src/test/regress/expected/privileges.out b/src/test/regress/expected/privileges.out
index 291e21d7a6..9ce619fd5f 100644
--- a/src/test/regress/expected/privileges.out
+++ b/src/test/regress/expected/privileges.out
@@ -27,8 +27,10 @@ CREATE USER regress_priv_user4;
CREATE USER regress_priv_user5;
CREATE USER regress_priv_user5; -- duplicate
ERROR: role "regress_priv_user5" already exists
-CREATE USER regress_priv_user6;
+CREATE USER regress_priv_user6 CREATEROLE;
+SET SESSION AUTHORIZATION regress_priv_user6;
CREATE USER regress_priv_user7;
+RESET SESSION AUTHORIZATION;
CREATE USER regress_priv_user8;
CREATE USER regress_priv_user9;
CREATE USER regress_priv_user10;
@@ -2356,7 +2358,12 @@ DROP USER regress_priv_user3;
DROP USER regress_priv_user4;
DROP USER regress_priv_user5;
DROP USER regress_priv_user6;
+ERROR: role "regress_priv_user6" cannot be dropped because some objects depend on it
+DETAIL: owner of role regress_priv_user7
+SET SESSION AUTHORIZATION regress_priv_user6;
DROP USER regress_priv_user7;
+RESET SESSION AUTHORIZATION;
+DROP USER regress_priv_user6;
DROP USER regress_priv_user8; -- does not exist
ERROR: role "regress_priv_user8" does not exist
-- permissions with LOCK TABLE
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index d652f7b5fb..b6e3c508ba 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1482,6 +1482,7 @@ pg_replication_slots| SELECT l.slot_name,
FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, safe_wal_size, two_phase)
LEFT JOIN pg_database d ON ((l.datoid = d.oid)));
pg_roles| SELECT pg_authid.rolname,
+ pg_get_userbyid(pg_authid.rolowner) AS rolowner,
pg_authid.rolsuper,
pg_authid.rolinherit,
pg_authid.rolcreaterole,
diff --git a/src/test/regress/sql/create_role.sql b/src/test/regress/sql/create_role.sql
index 292dc08797..6266586b8b 100644
--- a/src/test/regress/sql/create_role.sql
+++ b/src/test/regress/sql/create_role.sql
@@ -1,6 +1,8 @@
-- ok, superuser can create users with any set of privileges
CREATE ROLE regress_role_super SUPERUSER;
+CREATE ROLE regress_role_bystander;
CREATE ROLE regress_role_admin CREATEDB CREATEROLE REPLICATION BYPASSRLS;
+GRANT CREATE ON DATABASE regression TO regress_role_admin;
-- fail, only superusers can create users with these privileges
SET SESSION AUTHORIZATION regress_role_admin;
@@ -9,14 +11,53 @@ CREATE ROLE regress_nosuch_replication_bypassrls REPLICATION BYPASSRLS;
CREATE ROLE regress_nosuch_replication REPLICATION;
CREATE ROLE regress_nosuch_bypassrls BYPASSRLS;
+-- fail, only superusers can own superusers
+RESET SESSION AUTHORIZATION;
+CREATE ROLE regress_nosuch_superuser AUTHORIZATION regress_role_admin SUPERUSER;
+
+-- ok, superuser can create superusers belonging to other superusers
+CREATE ROLE regress_superuser AUTHORIZATION regress_role_super SUPERUSER;
+
+-- fail, can only create roles belonging to other roles that we belong to
+SET SESSION AUTHORIZATION regress_role_admin;
+CREATE ROLE regress_nosuch_alice AUTHORIZATION regress_role_super;
+CREATE ROLE regress_nosuch_bob AUTHORIZATION regress_superuser;
+CREATE ROLE regress_nosuch_charlie AUTHORIZATION regress_role_bystander;
+
+-- ok, superuser can create users with these privileges for normal role
+RESET SESSION AUTHORIZATION;
+CREATE ROLE regress_replication_bypassrls AUTHORIZATION regress_role_admin REPLICATION BYPASSRLS;
+CREATE ROLE regress_replication AUTHORIZATION regress_role_admin REPLICATION;
+CREATE ROLE regress_bypassrls AUTHORIZATION regress_role_admin BYPASSRLS;
+
+\du+ regress_superuser
+\du+ regress_replication_bypassrls
+\du+ regress_replication
+\du+ regress_bypassrls
+
+-- fail, roles are not allowed to own themselves
+ALTER ROLE regress_bypassrls OWNER TO regress_bypassrls;
+
-- ok, having CREATEROLE is enough to create users with these privileges
+SET SESSION AUTHORIZATION regress_role_admin;
CREATE ROLE regress_createdb CREATEDB;
CREATE ROLE regress_createrole CREATEROLE;
CREATE ROLE regress_login LOGIN;
CREATE ROLE regress_inherit INHERIT;
CREATE ROLE regress_connection_limit CONNECTION LIMIT 5;
-CREATE ROLE regress_encrypted_password ENCRYPTED PASSWORD 'foo';
-CREATE ROLE regress_password_null PASSWORD NULL;
+CREATE ROLE regress_encrypted_password PASSWORD NULL;
+CREATE ROLE regress_password_null
+ CREATEDB CREATEROLE INHERIT CONNECTION LIMIT 2 ENCRYPTED PASSWORD 'foo'
+ IN ROLE regress_createdb, regress_createrole;
+COMMENT ON ROLE regress_password_null IS 'no login test role';
+
+\du+ regress_createdb
+\du+ regress_createrole
+\du+ regress_login
+\du+ regress_inherit
+\du+ regress_connection_limit
+\du+ regress_encrypted_password
+\du+ regress_password_null
-- ok, backwards compatible noise words should be ignored
CREATE ROLE regress_noiseword SYSID 12345;
@@ -63,15 +104,19 @@ CREATE INDEX tenant_idx ON tenant_table(i);
CREATE VIEW tenant_view AS SELECT * FROM pg_catalog.pg_class;
REVOKE ALL PRIVILEGES ON tenant_table FROM PUBLIC;
--- fail, these objects belonging to regress_tenant
+-- ok, owning role can manage owned role's objects
SET SESSION AUTHORIZATION regress_createrole;
DROP INDEX tenant_idx;
ALTER TABLE tenant_table ADD COLUMN t text;
DROP TABLE tenant_table;
+
+-- fail, not a member of target role
ALTER VIEW tenant_view OWNER TO regress_role_admin;
+
+-- ok
DROP VIEW tenant_view;
--- fail, cannot take ownership of these objects from regress_tenant
+-- ok, can take ownership objects from owned roles
REASSIGN OWNED BY regress_tenant TO regress_createrole;
-- ok, having CREATEROLE is enough to create roles in privileged roles
@@ -86,21 +131,36 @@ CREATE ROLE regress_write_server_files IN ROLE pg_write_server_files;
CREATE ROLE regress_execute_server_program IN ROLE pg_execute_server_program;
CREATE ROLE regress_signal_backend IN ROLE pg_signal_backend;
--- fail, creation of these roles failed above so they do not now exist
+-- fail, cannot create ownership cycles
+RESET SESSION AUTHORIZATION;
+REASSIGN OWNED BY regress_role_admin TO regress_tenant;
+ALTER ROLE regress_role_admin OWNER TO regress_tenant;
+
+-- ok, can take ownership from owned roles
SET SESSION AUTHORIZATION regress_role_admin;
-DROP ROLE regress_nosuch_superuser;
-DROP ROLE regress_nosuch_replication_bypassrls;
-DROP ROLE regress_nosuch_replication;
-DROP ROLE regress_nosuch_bypassrls;
+ALTER ROLE regress_plainrole OWNER TO regress_role_admin;
+REASSIGN OWNED BY regress_plainrole TO regress_role_admin;
+
+-- ok, superuser roles can drop superuser roles they own
+SET SESSION AUTHORIZATION regress_role_super;
+DROP ROLE regress_superuser;
+
+-- ok, non-superuser roles can drop non-superuser roles they own
+SET SESSION AUTHORIZATION regress_role_admin;
+DROP ROLE regress_replication_bypassrls;
+DROP ROLE regress_replication;
+DROP ROLE regress_bypassrls;
DROP ROLE regress_nosuch_super;
DROP ROLE regress_nosuch_dbowner;
DROP ROLE regress_nosuch_recursive;
DROP ROLE regress_nosuch_admin_recursive;
DROP ROLE regress_plainrole;
--- ok, should be able to drop non-superuser roles we created
-DROP ROLE regress_createdb;
+-- fail, cannot drop roles that own other roles
DROP ROLE regress_createrole;
+
+-- ok, should be able to drop these non-superuser roles
+DROP ROLE regress_createdb;
DROP ROLE regress_login;
DROP ROLE regress_inherit;
DROP ROLE regress_connection_limit;
@@ -110,6 +170,7 @@ DROP ROLE regress_noiseword;
DROP ROLE regress_inroles;
DROP ROLE regress_adminroles;
DROP ROLE regress_rolecreator;
+DROP ROLE regress_tenant;
DROP ROLE regress_read_all_data;
DROP ROLE regress_write_all_data;
DROP ROLE regress_monitor;
@@ -121,18 +182,19 @@ DROP ROLE regress_write_server_files;
DROP ROLE regress_execute_server_program;
DROP ROLE regress_signal_backend;
--- fail, role still owns database objects
-DROP ROLE regress_tenant;
-
-- fail, cannot drop ourself nor superusers
DROP ROLE regress_role_super;
DROP ROLE regress_role_admin;
--- ok
+-- ok, no more owned roles remain
+DROP ROLE regress_createrole;
+
+-- fail, cannot drop role with remaining privileges
RESET SESSION AUTHORIZATION;
-DROP INDEX tenant_idx;
-DROP TABLE tenant_table;
-DROP VIEW tenant_view;
-DROP ROLE regress_tenant;
DROP ROLE regress_role_admin;
+
+-- ok, can drop role if we revoke privileges first
+REVOKE CREATE ON DATABASE regression FROM regress_role_admin;
+DROP ROLE regress_role_admin;
+DROP ROLE regress_role_bystander;
DROP ROLE regress_role_super;
diff --git a/src/test/regress/sql/privileges.sql b/src/test/regress/sql/privileges.sql
index c8c545b64c..dcf84f91fd 100644
--- a/src/test/regress/sql/privileges.sql
+++ b/src/test/regress/sql/privileges.sql
@@ -30,8 +30,10 @@ CREATE USER regress_priv_user3;
CREATE USER regress_priv_user4;
CREATE USER regress_priv_user5;
CREATE USER regress_priv_user5; -- duplicate
-CREATE USER regress_priv_user6;
+CREATE USER regress_priv_user6 CREATEROLE;
+SET SESSION AUTHORIZATION regress_priv_user6;
CREATE USER regress_priv_user7;
+RESET SESSION AUTHORIZATION;
CREATE USER regress_priv_user8;
CREATE USER regress_priv_user9;
CREATE USER regress_priv_user10;
@@ -1426,8 +1428,14 @@ DROP USER regress_priv_user2;
DROP USER regress_priv_user3;
DROP USER regress_priv_user4;
DROP USER regress_priv_user5;
+
DROP USER regress_priv_user6;
+
+SET SESSION AUTHORIZATION regress_priv_user6;
DROP USER regress_priv_user7;
+RESET SESSION AUTHORIZATION;
+
+DROP USER regress_priv_user6;
DROP USER regress_priv_user8; -- does not exist
--
2.21.1 (Apple Git-122.3)
[application/octet-stream] v7-0002-Restrict-power-granted-via-CREATEROLE.patch (43.9K, ../../[email protected]/3-v7-0002-Restrict-power-granted-via-CREATEROLE.patch)
download | inline diff:
From e6faa2a27cf01fd85c8d95101d89eb78416d4df6 Mon Sep 17 00:00:00 2001
From: Mark Dilger <[email protected]>
Date: Tue, 25 Jan 2022 14:54:06 -0800
Subject: [PATCH v7 2/2] Restrict power granted via CREATEROLE.
The CREATEROLE attribute no longer has anything to do with the power
to alter roles or to grant or revoke role membership, but merely the
ability to create new roles, as its name suggests. The ability to
alter a role is based on role ownership; the ability to grant and
revoke role membership is based on having admin privilege on the
relevant role or alternatively on role ownership, as owners now
implicitly have admin privileges on roles they own.
A role must either be superuser or have the CREATEROLE attribute to
create roles. This is unchanged from the prior behavior. A new
principle is adopted, though, to make CREATEROLE less dangerous: a
role may not create new roles with privileges that the creating role
lacks. This new principle is intended to prevent privilege
escalation attacks stemming from giving CREATEROLE to a user. This
is not backwards compatible. The idea is to fix the CREATEROLE
privilege to not be pathway to gaining superuser, and no
non-breaking change to accomplish that is apparent.
SUPERUSER, REPLICATION, BYPASSRLS, CREATEDB, CREATEROLE and LOGIN
privilege can only be given to new roles by creators who have the
same privilege. In the case of the CREATEROLE privilege, this is
trivially true, as the creator must necessarily have it or they
couldn't be creating the role to begin with.
The INHERIT attribute is not considered a privilege, and since a
user who belongs to a role may SET ROLE to that role and do anything
that role can do, it isn't clear that treating it as a privilege
would stop any privilege escalation attacks.
The CONNECTION LIMIT and VALID UNTIL attributes are also not
considered privileges, but this design choice is debatable. One
could think of the ability to log in during a given window of time,
or up to a certain number of connections as a privilege, and
allowing such a restricted role to create a new role with unlimited
connections or no expiration as a privilege escalation which escapes
the intended restrictions. However, it is just as easy to think of
these limitations as being used to guard against badly written
client programs connecting too many times, or connecting at a time
of day that is not intended. Since it is unclear which design is
better, this commit is conservative and the handling of these
attributes is unchanged relative to prior behavior.
Since the grammar of the CREATE ROLE command allows specifying roles
into which the new role should be enrolled, and also lists of roles
which become members of the newly created role (as admin or not),
the CREATE ROLE command may now fail if the creating role has
insufficient privilege on the roles so listed. Such failures were
not possible before, since the CREATEROLE privilege was always
sufficient.
---
doc/src/sgml/ddl.sgml | 12 +--
doc/src/sgml/ref/alter_role.sgml | 20 ++--
doc/src/sgml/ref/comment.sgml | 8 +-
doc/src/sgml/ref/create_role.sgml | 26 +++--
doc/src/sgml/ref/drop_role.sgml | 3 +-
doc/src/sgml/ref/dropuser.sgml | 6 +-
doc/src/sgml/ref/grant.sgml | 4 +-
doc/src/sgml/user-manag.sgml | 44 +++++----
src/backend/catalog/aclchk.c | 111 ++++++++++++++++++++++
src/backend/commands/user.c | 60 +++++-------
src/backend/utils/adt/acl.c | 21 +---
src/include/utils/acl.h | 5 +
src/test/regress/expected/create_role.out | 73 ++++++--------
src/test/regress/sql/create_role.sql | 45 ++++-----
14 files changed, 264 insertions(+), 174 deletions(-)
diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml
index 7cf0f0da3b..85c133d00d 100644
--- a/doc/src/sgml/ddl.sgml
+++ b/doc/src/sgml/ddl.sgml
@@ -3132,9 +3132,7 @@ REVOKE CREATE ON SCHEMA public FROM PUBLIC;
doesn't preserve that DROP.
A database owner can attack the database's users via "CREATE SCHEMA
- trojan; ALTER DATABASE $mydb SET search_path = trojan, public;". A
- CREATEROLE user can issue "GRANT $dbowner TO $me" and then use the
- database owner attack. -->
+ trojan; ALTER DATABASE $mydb SET search_path = trojan, public;". -->
<para>
Constrain ordinary users to user-private schemas. To implement this,
first issue <literal>REVOKE CREATE ON SCHEMA public FROM
@@ -3146,9 +3144,8 @@ REVOKE CREATE ON SCHEMA public FROM PUBLIC;
pattern in a database where untrusted users had already logged in,
consider auditing the public schema for objects named like objects in
schema <literal>pg_catalog</literal>. This pattern is a secure schema
- usage pattern unless an untrusted user is the database owner or holds
- the <literal>CREATEROLE</literal> privilege, in which case no secure
- schema usage pattern exists.
+ usage pattern unless an untrusted user is the database owner, in which
+ case no secure schema usage pattern exists.
</para>
<para>
If the database originated in an upgrade
@@ -3170,8 +3167,7 @@ REVOKE CREATE ON SCHEMA public FROM PUBLIC;
schema <link linkend="typeconv-func">will be unsafe or
unreliable</link>. If you create functions or extensions in the public
schema, use the first pattern instead. Otherwise, like the first
- pattern, this is secure unless an untrusted user is the database owner
- or holds the <literal>CREATEROLE</literal> privilege.
+ pattern, this is secure unless an untrusted user is the database owner.
</para>
</listitem>
diff --git a/doc/src/sgml/ref/alter_role.sgml b/doc/src/sgml/ref/alter_role.sgml
index 5aa5648ae7..fc9bea8072 100644
--- a/doc/src/sgml/ref/alter_role.sgml
+++ b/doc/src/sgml/ref/alter_role.sgml
@@ -70,18 +70,18 @@ ALTER ROLE { <replaceable class="parameter">role_specification</replaceable> | A
<link linkend="sql-revoke"><command>REVOKE</command></link> for that.)
Attributes not mentioned in the command retain their previous settings.
Database superusers can change any of these settings for any role.
- Roles having <literal>CREATEROLE</literal> privilege can change any of these
- settings except <literal>SUPERUSER</literal>, <literal>REPLICATION</literal>,
- and <literal>BYPASSRLS</literal>; but only for non-superuser and
- non-replication roles.
- Ordinary roles can only change their own password.
+ Role owners can change any of these settings on roles they directly or
+ indirectly own except <literal>SUPERUSER</literal>,
+ <literal>REPLICATION</literal>, and <literal>BYPASSRLS</literal>; but only
+ for non-superuser and non-replication roles, and only if the role owner does
+ not alter the target role to have a privilege which the role owner itself
+ lacks. Ordinary roles can only change their own password.
</para>
<para>
The second variant changes the name of the role.
Database superusers can rename any role.
- Roles having <literal>CREATEROLE</literal> privilege can rename non-superuser
- roles.
+ Roles can rename non-superuser roles they directly or indirectly own.
The current session user cannot be renamed.
(Connect as a different user if you need to do that.)
Because <literal>MD5</literal>-encrypted passwords use the role name as
@@ -114,9 +114,9 @@ ALTER ROLE { <replaceable class="parameter">role_specification</replaceable> | A
</para>
<para>
- Superusers can change anyone's session defaults. Roles having
- <literal>CREATEROLE</literal> privilege can change defaults for non-superuser
- roles. Ordinary roles can only set defaults for themselves.
+ Superusers can change anyone's session defaults. Roles may change
+ privilege for non-superuser roles they directly or indirectly own. Ordinary roles can only set
+ defaults for themselves.
Certain configuration variables cannot be set this way, or can only be
set if a superuser issues the command. Only superusers can change a setting
for all roles in all databases.
diff --git a/doc/src/sgml/ref/comment.sgml b/doc/src/sgml/ref/comment.sgml
index b12796095f..38c78c9971 100644
--- a/doc/src/sgml/ref/comment.sgml
+++ b/doc/src/sgml/ref/comment.sgml
@@ -97,12 +97,8 @@ COMMENT ON
<para>
For most kinds of object, only the object's owner can set the comment.
- Roles don't have owners, so the rule for <literal>COMMENT ON ROLE</literal> is
- that you must be superuser to comment on a superuser role, or have the
- <literal>CREATEROLE</literal> privilege to comment on non-superuser roles.
- Likewise, access methods don't have owners either; you must be superuser
- to comment on an access method.
- Of course, a superuser can comment on anything.
+ Access methods don't have owners; you must be superuser to comment on an
+ access method. Of course, a superuser can comment on anything.
</para>
<para>
diff --git a/doc/src/sgml/ref/create_role.sgml b/doc/src/sgml/ref/create_role.sgml
index 1e9347b2ce..59bbd418f4 100644
--- a/doc/src/sgml/ref/create_role.sgml
+++ b/doc/src/sgml/ref/create_role.sgml
@@ -126,8 +126,10 @@ in sync when changing the above synopsis!
<literal>CREATEDB</literal> is specified, the role being
defined will be allowed to create new databases. Specifying
<literal>NOCREATEDB</literal> will deny a role the ability to
- create databases. If not specified,
- <literal>NOCREATEDB</literal> is the default.
+ create databases. Only roles with the <literal>CREATEDB</literal>
+ attribute may create roles with the <literal>CREATEDB</literal>
+ attribute. If not specified, <literal>NOCREATEDB</literal> is the
+ default.
</para>
</listitem>
</varlistentry>
@@ -139,8 +141,6 @@ in sync when changing the above synopsis!
<para>
These clauses determine whether a role will be permitted to
create new roles (that is, execute <command>CREATE ROLE</command>).
- A role with <literal>CREATEROLE</literal> privilege can also alter
- and drop other roles.
If not specified,
<literal>NOCREATEROLE</literal> is the default.
</para>
@@ -182,6 +182,8 @@ in sync when changing the above synopsis!
<literal>NOLOGIN</literal> is the default, except when
<command>CREATE ROLE</command> is invoked through its alternative spelling
<link linkend="sql-createuser"><command>CREATE USER</command></link>.
+ You must have the <literal>LOGIN</literal> attribute to create a new role
+ with the <literal>LOGIN</literal> attribute.
</para>
</listitem>
</varlistentry>
@@ -213,8 +215,8 @@ in sync when changing the above synopsis!
<para>
These clauses determine whether a role bypasses every row-level
security (RLS) policy. <literal>NOBYPASSRLS</literal> is the default.
- You must be a superuser to create a new role having
- the <literal>BYPASSRLS</literal> attribute.
+ You must have the <literal>BYPASSRLS</literal> attribute to create a
+ new role having the <literal>BYPASSRLS</literal> attribute.
</para>
<para>
@@ -300,6 +302,10 @@ in sync when changing the above synopsis!
member. (Note that there is no option to add the new role as an
administrator; use a separate <command>GRANT</command> command to do that.)
</para>
+ <para>
+ If not a superuser, the creating role must either own or have admin
+ privilege on each listed role.
+ </para>
</listitem>
</varlistentry>
@@ -320,6 +326,10 @@ in sync when changing the above synopsis!
roles which are automatically added as members of the new role.
(This in effect makes the new role a <quote>group</quote>.)
</para>
+ <para>
+ If not a superuser, the creating role must either own or have admin
+ privilege on each listed role.
+ </para>
</listitem>
</varlistentry>
@@ -332,6 +342,10 @@ in sync when changing the above synopsis!
OPTION</literal>, giving them the right to grant membership in this role
to others.
</para>
+ <para>
+ If not a superuser, the creating role must either own or have admin
+ privilege on each listed role.
+ </para>
</listitem>
</varlistentry>
diff --git a/doc/src/sgml/ref/drop_role.sgml b/doc/src/sgml/ref/drop_role.sgml
index 13dc1cc649..c3d57ee8db 100644
--- a/doc/src/sgml/ref/drop_role.sgml
+++ b/doc/src/sgml/ref/drop_role.sgml
@@ -31,8 +31,7 @@ DROP ROLE [ IF EXISTS ] <replaceable class="parameter">name</replaceable> [, ...
<para>
<command>DROP ROLE</command> removes the specified role(s).
To drop a superuser role, you must be a superuser yourself;
- to drop non-superuser roles, you must have <literal>CREATEROLE</literal>
- privilege.
+ to drop non-superuser roles, you must own the target role.
</para>
<para>
diff --git a/doc/src/sgml/ref/dropuser.sgml b/doc/src/sgml/ref/dropuser.sgml
index 81580507e8..30a99eaf68 100644
--- a/doc/src/sgml/ref/dropuser.sgml
+++ b/doc/src/sgml/ref/dropuser.sgml
@@ -35,9 +35,9 @@ PostgreSQL documentation
<para>
<application>dropuser</application> removes an existing
<productname>PostgreSQL</productname> user.
- Only superusers and users with the <literal>CREATEROLE</literal> privilege can
- remove <productname>PostgreSQL</productname> users. (To remove a
- superuser, you must yourself be a superuser.)
+ A <productname>PostgreSQL</productname> user may only be removed by its
+ owner or by a superuser. (To remove a superuser, you must yourself be a
+ superuser.)
</para>
<para>
diff --git a/doc/src/sgml/ref/grant.sgml b/doc/src/sgml/ref/grant.sgml
index a897712de2..86fc387af2 100644
--- a/doc/src/sgml/ref/grant.sgml
+++ b/doc/src/sgml/ref/grant.sgml
@@ -254,8 +254,8 @@ GRANT <replaceable class="parameter">role_name</replaceable> [, ...] TO <replace
OPTION</literal> on itself, but it may grant or revoke membership in
itself from a database session where the session user matches the
role. Database superusers can grant or revoke membership in any role
- to anyone. Roles having <literal>CREATEROLE</literal> privilege can grant
- or revoke membership in any role that is not a superuser.
+ to anyone. Roles can revoke membership in any role they own, and
+ may grant membership in any role they own to any role they own.
</para>
<para>
diff --git a/doc/src/sgml/user-manag.sgml b/doc/src/sgml/user-manag.sgml
index 9067be1d9c..e65b55a004 100644
--- a/doc/src/sgml/user-manag.sgml
+++ b/doc/src/sgml/user-manag.sgml
@@ -198,9 +198,10 @@ CREATE USER <replaceable>name</replaceable>;
(except for superusers, since those bypass all permission
checks). To create such a role, use <literal>CREATE ROLE
<replaceable>name</replaceable> CREATEROLE</literal>.
- A role with <literal>CREATEROLE</literal> privilege can alter and drop
- other roles, too, as well as grant or revoke membership in them.
- However, to create, alter, drop, or change membership of a
+ A role which creates a new role becomes the new role's owner, able to
+ alter or drop that new role, and sharing ownership of any additional
+ objects (including additional roles) that new role creates.
+ To create, alter, drop, or change membership of a
superuser role, superuser status is required;
<literal>CREATEROLE</literal> is insufficient for that.
</para>
@@ -246,11 +247,14 @@ CREATE USER <replaceable>name</replaceable>;
<tip>
<para>
- It is good practice to create a role that has the <literal>CREATEDB</literal>
- and <literal>CREATEROLE</literal> privileges, but is not a superuser, and then
+ It is good practice to create a role that has the
+ <literal>CREATEDB</literal>, <literal>LOGIN</literal> and
+ <literal>CREATEROLE</literal> privileges, but is not a superuser, and then
use this role for all routine management of databases and roles. This
- approach avoids the dangers of operating as a superuser for tasks that
- do not really require it.
+ approach avoids the dangers of operating as a superuser for tasks that do
+ not really require it. This role must also have
+ <literal>REPLICATION</literal> if it will create replication users, and
+ must have <literal>BYPASSRLS</literal> if it will create bypassrls users.
</para>
</tip>
@@ -387,15 +391,22 @@ RESET ROLE;
<para>
The role attributes <literal>LOGIN</literal>, <literal>SUPERUSER</literal>,
- <literal>CREATEDB</literal>, and <literal>CREATEROLE</literal> can be thought of as
- special privileges, but they are never inherited as ordinary privileges
- on database objects are. You must actually <command>SET ROLE</command> to a
- specific role having one of these attributes in order to make use of
- the attribute. Continuing the above example, we might choose to
+ <literal>CREATEDB</literal>, <literal>REPLICATION</literal>,
+ <literal>BYPASSRLS</literal>, and <literal>CREATEROLE</literal> can be
+ thought of as special privileges, but they are never inherited as ordinary
+ privileges on database objects are. You must actually <command>SET
+ ROLE</command> to a specific role having one of these attributes in order to
+ make use of the attribute. Continuing the above example, we might choose to
grant <literal>CREATEDB</literal> and <literal>CREATEROLE</literal> to the
- <literal>admin</literal> role. Then a session connecting as role <literal>joe</literal>
- would not have these privileges immediately, only after doing
- <command>SET ROLE admin</command>.
+ <literal>admin</literal> role. Then a session connecting as role
+ <literal>joe</literal> would not have these privileges immediately, only
+ after doing <command>SET ROLE admin</command>. Roles with these attributes
+ may only be created by roles which themselves have these attributes.
+ Superusers may always do so, but non-superuser roles with
+ <literal>CREATEROLE</literal> may only create new roles with
+ <literal>LOGIN</literal>, <literal>CREATEDB</literal>,
+ <literal>REPLICATION</literal>, or <literal>BYPASSRLS</literal> if they
+ themselves have the same attribute.
</para>
<para>
@@ -493,8 +504,7 @@ DROP ROLE doomed_role;
<para>
<productname>PostgreSQL</productname> provides a set of predefined roles
that provide access to certain, commonly needed, privileged capabilities
- and information. Administrators (including roles that have the
- <literal>CREATEROLE</literal> privilege) can <command>GRANT</command> these
+ and information. Administrators can <command>GRANT</command> these
roles to users and/or other roles in their environment, providing those
users with access to the specified capabilities and information.
</para>
diff --git a/src/backend/catalog/aclchk.c b/src/backend/catalog/aclchk.c
index 1e0ee503e4..8327005404 100644
--- a/src/backend/catalog/aclchk.c
+++ b/src/backend/catalog/aclchk.c
@@ -5470,6 +5470,9 @@ has_createrole_privilege(Oid roleid)
return result;
}
+/*
+ * Check whether specified role has BYPASSRLS privilege (or is a superuser)
+ */
bool
has_bypassrls_privilege(Oid roleid)
{
@@ -5489,6 +5492,114 @@ has_bypassrls_privilege(Oid roleid)
return result;
}
+/*
+ * Check whether specified role has INHERIT privilege (or is a superuser)
+ */
+bool
+has_inherit_privilege(Oid roleid)
+{
+ bool result = false;
+ HeapTuple utup;
+
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(roleid))
+ return true;
+
+ utup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(roleid));
+ if (HeapTupleIsValid(utup))
+ {
+ result = ((Form_pg_authid) GETSTRUCT(utup))->rolinherit;
+ ReleaseSysCache(utup);
+ }
+ return result;
+}
+
+/*
+ * Check whether specified role has CREATEDB privilege (or is a superuser)
+ */
+bool
+has_createdb_privilege(Oid roleid)
+{
+ bool result = false;
+ HeapTuple utup;
+
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(roleid))
+ return true;
+
+ utup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(roleid));
+ if (HeapTupleIsValid(utup))
+ {
+ result = ((Form_pg_authid) GETSTRUCT(utup))->rolcreatedb;
+ ReleaseSysCache(utup);
+ }
+ return result;
+}
+
+/*
+ * Check whether specified role has LOGIN privilege (or is a superuser)
+ */
+bool
+has_login_privilege(Oid roleid)
+{
+ bool result = false;
+ HeapTuple utup;
+
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(roleid))
+ return true;
+
+ utup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(roleid));
+ if (HeapTupleIsValid(utup))
+ {
+ result = ((Form_pg_authid) GETSTRUCT(utup))->rolcanlogin;
+ ReleaseSysCache(utup);
+ }
+ return result;
+}
+
+/*
+ * Check whether specified role has REPLICATION privilege (or is a superuser)
+ */
+bool
+has_replication_privilege(Oid roleid)
+{
+ bool result = false;
+ HeapTuple utup;
+
+ /* Superusers bypass all permission checking. */
+ if (superuser_arg(roleid))
+ return true;
+
+ utup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(roleid));
+ if (HeapTupleIsValid(utup))
+ {
+ result = ((Form_pg_authid) GETSTRUCT(utup))->rolreplication;
+ ReleaseSysCache(utup);
+ }
+ return result;
+}
+
+/*
+ * Get the connection limit for the specified role.
+ *
+ * Returns -1 if the role has no connection limit.
+ */
+int32
+role_connection_limit(Oid roleid)
+{
+ int32 result = -1;
+ HeapTuple utup;
+
+ utup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(roleid));
+ if (HeapTupleIsValid(utup))
+ {
+ result = ((Form_pg_authid) GETSTRUCT(utup))->rolconnlimit;
+ ReleaseSysCache(utup);
+ }
+ return result;
+}
+
/*
* Fetch pg_default_acl entry for given role, namespace and object type
* (object type must be given in pg_default_acl's encoding).
diff --git a/src/backend/commands/user.c b/src/backend/commands/user.c
index 3726e40f36..13265a0ed2 100644
--- a/src/backend/commands/user.c
+++ b/src/backend/commands/user.c
@@ -58,15 +58,6 @@ static void DelRoleMems(const char *rolename, Oid roleid,
static void AlterRoleOwner_internal(HeapTuple tup, Relation rel,
Oid newOwnerId);
-
-/* Check if current user has createrole privileges */
-static bool
-have_createrole_privilege(void)
-{
- return has_createrole_privilege(GetUserId());
-}
-
-
/*
* CREATE ROLE
*/
@@ -279,24 +270,32 @@ CreateRole(ParseState *pstate, CreateRoleStmt *stmt)
}
else if (isreplication)
{
- if (!superuser())
+ if (!has_replication_privilege(GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser to create replication users")));
+ errmsg("must have replication privilege to create replication users")));
}
else if (bypassrls)
{
- if (!superuser())
+ if (!has_bypassrls_privilege(GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
- errmsg("must be superuser to create bypassrls users")));
+ errmsg("must have bypassrls privilege to create bypassrls users")));
}
- else
+ else if (!superuser())
{
- if (!have_createrole_privilege())
+ if (!has_createrole_privilege(GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("permission denied to create role")));
+ if (createdb && !has_createdb_privilege(GetUserId()))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must have createdb privilege to create createdb users")));
+ if (canlogin && !has_login_privilege(GetUserId()))
+ ereport(ERROR,
+ (errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
+ errmsg("must have login privilege to create login users")));
}
/*
@@ -692,7 +691,7 @@ AlterRole(ParseState *pstate, AlterRoleStmt *stmt)
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to change bypassrls attribute")));
}
- else if (!have_createrole_privilege())
+ else if (!superuser())
{
/* check the rest */
if (dinherit || dcreaterole || dcreatedb || dcanlogin || dconnlimit ||
@@ -891,7 +890,7 @@ AlterRoleSet(AlterRoleSetStmt *stmt)
/*
* To mess with a superuser you gotta be superuser; else you need
- * createrole, or just want to change your own settings
+ * to own the role, or just want to change your own settings
*/
if (roleform->rolsuper)
{
@@ -902,8 +901,7 @@ AlterRoleSet(AlterRoleSetStmt *stmt)
}
else
{
- if (!have_createrole_privilege() &&
- !pg_role_ownercheck(roleid, GetUserId()))
+ if (!pg_role_ownercheck(roleid, GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("permission denied")));
@@ -1016,18 +1014,12 @@ DropRole(DropRoleStmt *stmt)
(errcode(ERRCODE_OBJECT_IN_USE),
errmsg("session user cannot be dropped")));
- /*
- * For safety's sake, we allow createrole holders to drop ordinary
- * roles but not superuser roles. This is mainly to avoid the
- * scenario where you accidentally drop the last superuser.
- */
if (roleform->rolsuper && !superuser())
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to drop superusers")));
- if (!have_createrole_privilege() &&
- !pg_role_ownercheck(roleid, GetUserId()))
+ if (!pg_role_ownercheck(roleid, GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("permission denied to drop role")));
@@ -1208,7 +1200,7 @@ RenameRole(const char *oldname, const char *newname)
errmsg("role \"%s\" already exists", newname)));
/*
- * createrole is enough privilege unless you want to mess with a superuser
+ * role ownership is enough privilege unless you want to mess with a superuser
*/
if (((Form_pg_authid) GETSTRUCT(oldtuple))->rolsuper)
{
@@ -1219,7 +1211,7 @@ RenameRole(const char *oldname, const char *newname)
}
else
{
- if (!have_createrole_privilege())
+ if (!pg_role_ownercheck(roleid, GetUserId()))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("permission denied to rename role")));
@@ -1434,7 +1426,7 @@ AddRoleMems(const char *rolename, Oid roleid,
return;
/*
- * Check permissions: must have createrole or admin option on the role to
+ * Check permissions: must be owner or have admin option on the role to
* be changed. To mess with a superuser role, you gotta be superuser.
*/
if (superuser_arg(roleid))
@@ -1444,9 +1436,9 @@ AddRoleMems(const char *rolename, Oid roleid,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to alter superusers")));
}
- else
+ else if (!superuser())
{
- if (!have_createrole_privilege() &&
+ if (!pg_role_ownercheck(roleid, grantorId) &&
!is_admin_of_role(grantorId, roleid))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
@@ -1622,9 +1614,9 @@ DelRoleMems(const char *rolename, Oid roleid,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
errmsg("must be superuser to alter superusers")));
}
- else
+ else if (!superuser())
{
- if (!have_createrole_privilege() &&
+ if (!pg_role_ownercheck(roleid, GetUserId()) &&
!is_admin_of_role(GetUserId(), roleid))
ereport(ERROR,
(errcode(ERRCODE_INSUFFICIENT_PRIVILEGE),
@@ -1780,7 +1772,7 @@ AlterRoleOwner_internal(HeapTuple tup, Relation rel, Oid newOwnerId)
* consistent with the CREATE case for roles. Because superusers will
* always have this right, we need no special case for them.
*/
- if (!have_createrole_privilege())
+ if (!has_createrole_privilege(GetUserId()))
aclcheck_error(ACLCHECK_NO_PRIV, OBJECT_ROLE,
NameStr(authForm->rolname));
diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c
index 97336db058..cdcce9c569 100644
--- a/src/backend/utils/adt/acl.c
+++ b/src/backend/utils/adt/acl.c
@@ -4656,7 +4656,7 @@ initialize_acl(void)
/*
* In normal mode, set a callback on any syscache invalidation of rows
* of pg_auth_members (for roles_is_member_of()), pg_authid (for
- * has_rolinherit()), or pg_database (for roles_is_member_of())
+ * has_inherit_privilege()), or pg_database (for roles_is_member_of())
*/
CacheRegisterSyscacheCallback(AUTHMEMROLEMEM,
RoleMembershipCacheCallback,
@@ -4690,23 +4690,6 @@ RoleMembershipCacheCallback(Datum arg, int cacheid, uint32 hashvalue)
}
-/* Check if specified role has rolinherit set */
-static bool
-has_rolinherit(Oid roleid)
-{
- bool result = false;
- HeapTuple utup;
-
- utup = SearchSysCache1(AUTHOID, ObjectIdGetDatum(roleid));
- if (HeapTupleIsValid(utup))
- {
- result = ((Form_pg_authid) GETSTRUCT(utup))->rolinherit;
- ReleaseSysCache(utup);
- }
- return result;
-}
-
-
/*
* Get a list of roles that the specified roleid is a member of
*
@@ -4776,7 +4759,7 @@ roles_is_member_of(Oid roleid, enum RoleRecurseType type,
CatCList *memlist;
int i;
- if (type == ROLERECURSE_PRIVS && !has_rolinherit(memberid))
+ if (type == ROLERECURSE_PRIVS && !has_inherit_privilege(memberid))
continue; /* ignore non-inheriting roles */
/* Find roles that memberid is directly a member of */
diff --git a/src/include/utils/acl.h b/src/include/utils/acl.h
index 572cae0f27..7a6640dfc9 100644
--- a/src/include/utils/acl.h
+++ b/src/include/utils/acl.h
@@ -320,5 +320,10 @@ extern bool pg_statistics_object_ownercheck(Oid stat_oid, Oid roleid);
extern bool pg_role_ownercheck(Oid owned_role_oid, Oid owner_roleid);
extern bool has_createrole_privilege(Oid roleid);
extern bool has_bypassrls_privilege(Oid roleid);
+extern bool has_inherit_privilege(Oid roleid);
+extern bool has_createdb_privilege(Oid roleid);
+extern bool has_login_privilege(Oid roleid);
+extern bool has_replication_privilege(Oid roleid);
+extern int32 role_connection_limit(Oid roleid);
#endif /* ACL_H */
diff --git a/src/test/regress/expected/create_role.out b/src/test/regress/expected/create_role.out
index 4ad6fd4161..61f143a191 100644
--- a/src/test/regress/expected/create_role.out
+++ b/src/test/regress/expected/create_role.out
@@ -3,16 +3,17 @@ CREATE ROLE regress_role_super SUPERUSER;
CREATE ROLE regress_role_bystander;
CREATE ROLE regress_role_admin CREATEDB CREATEROLE REPLICATION BYPASSRLS;
GRANT CREATE ON DATABASE regression TO regress_role_admin;
--- fail, only superusers can create users with these privileges
+-- fail, only superusers can create users with superuser
SET SESSION AUTHORIZATION regress_role_admin;
CREATE ROLE regress_nosuch_superuser SUPERUSER;
ERROR: must be superuser to create superusers
-CREATE ROLE regress_nosuch_replication_bypassrls REPLICATION BYPASSRLS;
-ERROR: must be superuser to create replication users
-CREATE ROLE regress_nosuch_replication REPLICATION;
-ERROR: must be superuser to create replication users
-CREATE ROLE regress_nosuch_bypassrls BYPASSRLS;
-ERROR: must be superuser to create bypassrls users
+-- fail, only login users can create other login users
+CREATE ROLE regress_nosuch_login LOGIN;
+ERROR: must have login privilege to create login users
+-- ok, can assign privileges the creator has
+CREATE ROLE regress_replication_bypassrls REPLICATION BYPASSRLS;
+CREATE ROLE regress_replication REPLICATION;
+CREATE ROLE regress_bypassrls BYPASSRLS;
-- fail, only superusers can own superusers
RESET SESSION AUTHORIZATION;
CREATE ROLE regress_nosuch_superuser AUTHORIZATION regress_role_admin SUPERUSER;
@@ -29,9 +30,6 @@ CREATE ROLE regress_nosuch_charlie AUTHORIZATION regress_role_bystander;
ERROR: must be member of role "regress_role_bystander"
-- ok, superuser can create users with these privileges for normal role
RESET SESSION AUTHORIZATION;
-CREATE ROLE regress_replication_bypassrls AUTHORIZATION regress_role_admin REPLICATION BYPASSRLS;
-CREATE ROLE regress_replication AUTHORIZATION regress_role_admin REPLICATION;
-CREATE ROLE regress_bypassrls AUTHORIZATION regress_role_admin BYPASSRLS;
\du+ regress_superuser
List of roles
Role name | Owner | Attributes | Member of | Description
@@ -63,7 +61,10 @@ ERROR: role may not own itself
SET SESSION AUTHORIZATION regress_role_admin;
CREATE ROLE regress_createdb CREATEDB;
CREATE ROLE regress_createrole CREATEROLE;
+-- fail, cannot assign LOGIN privilege that creator lacks
CREATE ROLE regress_login LOGIN;
+ERROR: must have login privilege to create login users
+-- ok, having CREATEROLE is enough for these
CREATE ROLE regress_inherit INHERIT;
CREATE ROLE regress_connection_limit CONNECTION LIMIT 5;
CREATE ROLE regress_encrypted_password PASSWORD NULL;
@@ -83,12 +84,6 @@ COMMENT ON ROLE regress_password_null IS 'no login test role';
--------------------+--------------------+---------------------------+-----------+-------------
regress_createrole | regress_role_admin | Create role, Cannot login | {} |
-\du+ regress_login
- List of roles
- Role name | Owner | Attributes | Member of | Description
----------------+--------------------+------------+-----------+-------------
- regress_login | regress_role_admin | | {} |
-
\du+ regress_inherit
List of roles
Role name | Owner | Attributes | Member of | Description
@@ -121,19 +116,19 @@ NOTICE: SYSID can no longer be specified
-- fail, cannot grant membership in superuser role
CREATE ROLE regress_nosuch_super IN ROLE regress_role_super;
ERROR: must be superuser to alter superusers
--- fail, database owner cannot have members
+-- fail, do not have ADMIN privilege on database owner
CREATE ROLE regress_nosuch_dbowner IN ROLE pg_database_owner;
-ERROR: role "pg_database_owner" cannot have explicit members
+ERROR: must have admin option on role "pg_database_owner"
-- ok, can grant other users into a role
CREATE ROLE regress_inroles ROLE
- regress_role_super, regress_createdb, regress_createrole, regress_login,
+ regress_role_super, regress_createdb, regress_createrole,
regress_inherit, regress_connection_limit, regress_encrypted_password, regress_password_null;
-- fail, cannot grant a role into itself
CREATE ROLE regress_nosuch_recursive ROLE regress_nosuch_recursive;
ERROR: role "regress_nosuch_recursive" is a member of role "regress_nosuch_recursive"
-- ok, can grant other users into a role with admin option
CREATE ROLE regress_adminroles ADMIN
- regress_role_super, regress_createdb, regress_createrole, regress_login,
+ regress_role_super, regress_createdb, regress_createrole,
regress_inherit, regress_connection_limit, regress_encrypted_password, regress_password_null;
-- fail, cannot grant a role into itself with admin option
CREATE ROLE regress_nosuch_admin_recursive ADMIN regress_nosuch_admin_recursive;
@@ -146,8 +141,11 @@ ERROR: permission denied to create database
CREATE ROLE regress_plainrole;
-- ok, roles with CREATEROLE can create new roles with it
CREATE ROLE regress_rolecreator CREATEROLE;
--- ok, roles with CREATEROLE can create new roles with privilege they lack
+-- fail, roles with CREATEROLE cannot create new roles with privilege they lack
CREATE ROLE regress_tenant CREATEDB CREATEROLE LOGIN INHERIT CONNECTION LIMIT 5;
+ERROR: must have createdb privilege to create createdb users
+-- ok, roles with CREATEROLE can create new roles with privilege they have
+CREATE ROLE regress_tenant CREATEROLE INHERIT CONNECTION LIMIT 5;
-- ok, regress_tenant can create objects within the database
SET SESSION AUTHORIZATION regress_tenant;
CREATE TABLE tenant_table (i integer);
@@ -166,17 +164,27 @@ ERROR: must be member of role "regress_role_admin"
DROP VIEW tenant_view;
-- ok, can take ownership objects from owned roles
REASSIGN OWNED BY regress_tenant TO regress_createrole;
--- ok, having CREATEROLE is enough to create roles in privileged roles
+-- fail, having CREATEROLE is not enough to create roles in privileged roles
CREATE ROLE regress_read_all_data IN ROLE pg_read_all_data;
+ERROR: must have admin option on role "pg_read_all_data"
CREATE ROLE regress_write_all_data IN ROLE pg_write_all_data;
+ERROR: must have admin option on role "pg_write_all_data"
CREATE ROLE regress_monitor IN ROLE pg_monitor;
+ERROR: must have admin option on role "pg_monitor"
CREATE ROLE regress_read_all_settings IN ROLE pg_read_all_settings;
+ERROR: must have admin option on role "pg_read_all_settings"
CREATE ROLE regress_read_all_stats IN ROLE pg_read_all_stats;
+ERROR: must have admin option on role "pg_read_all_stats"
CREATE ROLE regress_stat_scan_tables IN ROLE pg_stat_scan_tables;
+ERROR: must have admin option on role "pg_stat_scan_tables"
CREATE ROLE regress_read_server_files IN ROLE pg_read_server_files;
+ERROR: must have admin option on role "pg_read_server_files"
CREATE ROLE regress_write_server_files IN ROLE pg_write_server_files;
+ERROR: must have admin option on role "pg_write_server_files"
CREATE ROLE regress_execute_server_program IN ROLE pg_execute_server_program;
+ERROR: must have admin option on role "pg_execute_server_program"
CREATE ROLE regress_signal_backend IN ROLE pg_signal_backend;
+ERROR: must have admin option on role "pg_signal_backend"
-- fail, cannot create ownership cycles
RESET SESSION AUTHORIZATION;
REASSIGN OWNED BY regress_role_admin TO regress_tenant;
@@ -209,19 +217,8 @@ DROP ROLE regress_createrole;
ERROR: role "regress_createrole" cannot be dropped because some objects depend on it
DETAIL: owner of role regress_rolecreator
owner of role regress_tenant
-owner of role regress_read_all_data
-owner of role regress_write_all_data
-owner of role regress_monitor
-owner of role regress_read_all_settings
-owner of role regress_read_all_stats
-owner of role regress_stat_scan_tables
-owner of role regress_read_server_files
-owner of role regress_write_server_files
-owner of role regress_execute_server_program
-owner of role regress_signal_backend
-- ok, should be able to drop these non-superuser roles
DROP ROLE regress_createdb;
-DROP ROLE regress_login;
DROP ROLE regress_inherit;
DROP ROLE regress_connection_limit;
DROP ROLE regress_encrypted_password;
@@ -231,16 +228,6 @@ DROP ROLE regress_inroles;
DROP ROLE regress_adminroles;
DROP ROLE regress_rolecreator;
DROP ROLE regress_tenant;
-DROP ROLE regress_read_all_data;
-DROP ROLE regress_write_all_data;
-DROP ROLE regress_monitor;
-DROP ROLE regress_read_all_settings;
-DROP ROLE regress_read_all_stats;
-DROP ROLE regress_stat_scan_tables;
-DROP ROLE regress_read_server_files;
-DROP ROLE regress_write_server_files;
-DROP ROLE regress_execute_server_program;
-DROP ROLE regress_signal_backend;
-- fail, cannot drop ourself nor superusers
DROP ROLE regress_role_super;
ERROR: must be superuser to drop superusers
diff --git a/src/test/regress/sql/create_role.sql b/src/test/regress/sql/create_role.sql
index 6266586b8b..36b9b04322 100644
--- a/src/test/regress/sql/create_role.sql
+++ b/src/test/regress/sql/create_role.sql
@@ -4,12 +4,17 @@ CREATE ROLE regress_role_bystander;
CREATE ROLE regress_role_admin CREATEDB CREATEROLE REPLICATION BYPASSRLS;
GRANT CREATE ON DATABASE regression TO regress_role_admin;
--- fail, only superusers can create users with these privileges
+-- fail, only superusers can create users with superuser
SET SESSION AUTHORIZATION regress_role_admin;
CREATE ROLE regress_nosuch_superuser SUPERUSER;
-CREATE ROLE regress_nosuch_replication_bypassrls REPLICATION BYPASSRLS;
-CREATE ROLE regress_nosuch_replication REPLICATION;
-CREATE ROLE regress_nosuch_bypassrls BYPASSRLS;
+
+-- fail, only login users can create other login users
+CREATE ROLE regress_nosuch_login LOGIN;
+
+-- ok, can assign privileges the creator has
+CREATE ROLE regress_replication_bypassrls REPLICATION BYPASSRLS;
+CREATE ROLE regress_replication REPLICATION;
+CREATE ROLE regress_bypassrls BYPASSRLS;
-- fail, only superusers can own superusers
RESET SESSION AUTHORIZATION;
@@ -26,9 +31,6 @@ CREATE ROLE regress_nosuch_charlie AUTHORIZATION regress_role_bystander;
-- ok, superuser can create users with these privileges for normal role
RESET SESSION AUTHORIZATION;
-CREATE ROLE regress_replication_bypassrls AUTHORIZATION regress_role_admin REPLICATION BYPASSRLS;
-CREATE ROLE regress_replication AUTHORIZATION regress_role_admin REPLICATION;
-CREATE ROLE regress_bypassrls AUTHORIZATION regress_role_admin BYPASSRLS;
\du+ regress_superuser
\du+ regress_replication_bypassrls
@@ -42,7 +44,11 @@ ALTER ROLE regress_bypassrls OWNER TO regress_bypassrls;
SET SESSION AUTHORIZATION regress_role_admin;
CREATE ROLE regress_createdb CREATEDB;
CREATE ROLE regress_createrole CREATEROLE;
+
+-- fail, cannot assign LOGIN privilege that creator lacks
CREATE ROLE regress_login LOGIN;
+
+-- ok, having CREATEROLE is enough for these
CREATE ROLE regress_inherit INHERIT;
CREATE ROLE regress_connection_limit CONNECTION LIMIT 5;
CREATE ROLE regress_encrypted_password PASSWORD NULL;
@@ -53,7 +59,6 @@ COMMENT ON ROLE regress_password_null IS 'no login test role';
\du+ regress_createdb
\du+ regress_createrole
-\du+ regress_login
\du+ regress_inherit
\du+ regress_connection_limit
\du+ regress_encrypted_password
@@ -65,12 +70,12 @@ CREATE ROLE regress_noiseword SYSID 12345;
-- fail, cannot grant membership in superuser role
CREATE ROLE regress_nosuch_super IN ROLE regress_role_super;
--- fail, database owner cannot have members
+-- fail, do not have ADMIN privilege on database owner
CREATE ROLE regress_nosuch_dbowner IN ROLE pg_database_owner;
-- ok, can grant other users into a role
CREATE ROLE regress_inroles ROLE
- regress_role_super, regress_createdb, regress_createrole, regress_login,
+ regress_role_super, regress_createdb, regress_createrole,
regress_inherit, regress_connection_limit, regress_encrypted_password, regress_password_null;
-- fail, cannot grant a role into itself
@@ -78,7 +83,7 @@ CREATE ROLE regress_nosuch_recursive ROLE regress_nosuch_recursive;
-- ok, can grant other users into a role with admin option
CREATE ROLE regress_adminroles ADMIN
- regress_role_super, regress_createdb, regress_createrole, regress_login,
+ regress_role_super, regress_createdb, regress_createrole,
regress_inherit, regress_connection_limit, regress_encrypted_password, regress_password_null;
-- fail, cannot grant a role into itself with admin option
@@ -94,9 +99,12 @@ CREATE ROLE regress_plainrole;
-- ok, roles with CREATEROLE can create new roles with it
CREATE ROLE regress_rolecreator CREATEROLE;
--- ok, roles with CREATEROLE can create new roles with privilege they lack
+-- fail, roles with CREATEROLE cannot create new roles with privilege they lack
CREATE ROLE regress_tenant CREATEDB CREATEROLE LOGIN INHERIT CONNECTION LIMIT 5;
+-- ok, roles with CREATEROLE can create new roles with privilege they have
+CREATE ROLE regress_tenant CREATEROLE INHERIT CONNECTION LIMIT 5;
+
-- ok, regress_tenant can create objects within the database
SET SESSION AUTHORIZATION regress_tenant;
CREATE TABLE tenant_table (i integer);
@@ -119,7 +127,7 @@ DROP VIEW tenant_view;
-- ok, can take ownership objects from owned roles
REASSIGN OWNED BY regress_tenant TO regress_createrole;
--- ok, having CREATEROLE is enough to create roles in privileged roles
+-- fail, having CREATEROLE is not enough to create roles in privileged roles
CREATE ROLE regress_read_all_data IN ROLE pg_read_all_data;
CREATE ROLE regress_write_all_data IN ROLE pg_write_all_data;
CREATE ROLE regress_monitor IN ROLE pg_monitor;
@@ -161,7 +169,6 @@ DROP ROLE regress_createrole;
-- ok, should be able to drop these non-superuser roles
DROP ROLE regress_createdb;
-DROP ROLE regress_login;
DROP ROLE regress_inherit;
DROP ROLE regress_connection_limit;
DROP ROLE regress_encrypted_password;
@@ -171,16 +178,6 @@ DROP ROLE regress_inroles;
DROP ROLE regress_adminroles;
DROP ROLE regress_rolecreator;
DROP ROLE regress_tenant;
-DROP ROLE regress_read_all_data;
-DROP ROLE regress_write_all_data;
-DROP ROLE regress_monitor;
-DROP ROLE regress_read_all_settings;
-DROP ROLE regress_read_all_stats;
-DROP ROLE regress_stat_scan_tables;
-DROP ROLE regress_read_server_files;
-DROP ROLE regress_write_server_files;
-DROP ROLE regress_execute_server_program;
-DROP ROLE regress_signal_backend;
-- fail, cannot drop ourself nor superusers
DROP ROLE regress_role_super;
--
2.21.1 (Apple Git-122.3)
^ permalink raw reply [nested|flat] 28+ messages in thread
* [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s)
@ 2023-09-24 20:49 Karl O. Pinc <[email protected]>
0 siblings, 0 replies; 28+ messages in thread
From: Karl O. Pinc @ 2023-09-24 20:49 UTC (permalink / raw)
The current section title of "Executing a Command with a Single-Row
Result" does not reflect what the section is really about. Other
sections make clear how to _execute_ commands, single-row result or not.
What this section is about is how to _save_ a single row of results into
variable(s).
It would be nice to talk about saving results into variables in the
section heading but I couldn't come up with anything pithy. "Saving a
Single-Row of a Command's Result" seems good enough, especially since
there's few other places to save results other than in variables.
---
doc/src/sgml/plpgsql.sgml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/doc/src/sgml/plpgsql.sgml b/doc/src/sgml/plpgsql.sgml
index f55e901c7e..8747e84245 100644
--- a/doc/src/sgml/plpgsql.sgml
+++ b/doc/src/sgml/plpgsql.sgml
@@ -1126,7 +1126,7 @@ PERFORM create_mv('cs_session_page_requests_mv', my_query);
</sect2>
<sect2 id="plpgsql-statements-sql-onerow">
- <title>Executing a Command with a Single-Row Result</title>
+ <title>Saving a Single-Row of a Command's Result</title>
<indexterm zone="plpgsql-statements-sql-onerow">
<primary>SELECT INTO</primary>
--
2.30.2
--MP_/OOXZvOwbpccKfGOtE9/SwX6
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v4-0002-Change-section-heading-to-better-describe-referen.patch
^ permalink raw reply [nested|flat] 28+ messages in thread
end of thread, other threads:[~2023-09-24 20:49 UTC | newest]
Thread overview: 28+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2018-10-16 04:04 [PATCH 2/4] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2018-10-16 04:04 [PATCH 1/4] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2018-10-16 04:04 [PATCH 3/5] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2018-10-16 04:04 [PATCH 2/3] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2018-10-16 04:04 [PATCH 1/4] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2018-10-16 04:04 [PATCH 2/2] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2018-10-16 04:04 [PATCH 1/4] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2018-10-16 04:04 [PATCH 2/4] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2019-03-01 04:32 [PATCH 2/6] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2019-03-01 04:32 [PATCH] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2019-03-01 04:32 [PATCH 2/2] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2019-03-01 04:32 [PATCH 1/2] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2019-03-01 04:32 [PATCH] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2019-03-01 04:32 [PATCH] Remove entries that haven't been used for a certain time Kyotaro Horiguchi <[email protected]>
2022-01-22 21:20 Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 19:19 ` Re: CREATEROLE and role ownership hierarchies Andrew Dunstan <[email protected]>
2022-01-24 20:33 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 21:00 ` Re: CREATEROLE and role ownership hierarchies Andrew Dunstan <[email protected]>
2022-01-24 21:23 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 21:41 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-24 22:21 ` Re: CREATEROLE and role ownership hierarchies Stephen Frost <[email protected]>
2022-01-24 23:18 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
2022-01-25 06:55 ` Re: CREATEROLE and role ownership hierarchies Fujii Masao <[email protected]>
2022-01-25 16:21 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
2022-01-25 16:42 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
2022-01-25 19:04 ` Re: CREATEROLE and role ownership hierarchies Robert Haas <[email protected]>
2022-01-25 23:17 ` Re: CREATEROLE and role ownership hierarchies Mark Dilger <[email protected]>
2023-09-24 20:49 [PATCH v4 01/12] Change section heading to better reflect saving a result in variable(s) Karl O. Pinc <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox