public inbox for [email protected]
help / color / mirror / Atom feed[PATCH v20 2/3] Add WAL relief vent for replication slots
86+ messages / 18 participants
[nested] [flat]
* [PATCH v20 2/3] Add WAL relief vent for replication slots
@ 2017-12-21 12:20 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2017-12-21 12:20 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
contrib/test_decoding/expected/ddl.out | 4 +-
contrib/test_decoding/sql/ddl.sql | 2 +
doc/src/sgml/catalogs.sgml | 48 +++
doc/src/sgml/config.sgml | 23 ++
doc/src/sgml/high-availability.sgml | 8 +-
src/backend/access/transam/xlog.c | 361 ++++++++++++++++--
src/backend/catalog/system_views.sql | 4 +-
src/backend/replication/slot.c | 1 +
src/backend/replication/slotfuncs.c | 39 +-
src/backend/utils/misc/guc.c | 13 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 19 +
src/include/catalog/pg_proc.dat | 6 +-
src/test/recovery/t/018_replslot_limit.pl | 202 ++++++++++
src/test/regress/expected/rules.out | 6 +-
15 files changed, 699 insertions(+), 38 deletions(-)
create mode 100644 src/test/recovery/t/018_replslot_limit.pl
diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out
index 2c999fd3eb..cf0318f697 100644
--- a/contrib/test_decoding/expected/ddl.out
+++ b/contrib/test_decoding/expected/ddl.out
@@ -723,8 +723,8 @@ SELECT pg_drop_replication_slot('regression_slot');
(1 row)
/* check that the slot is gone */
+\x
SELECT * FROM pg_replication_slots;
- slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn
------------+--------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+---------------------
(0 rows)
+\x
diff --git a/contrib/test_decoding/sql/ddl.sql b/contrib/test_decoding/sql/ddl.sql
index 856495c952..0f2b9992f7 100644
--- a/contrib/test_decoding/sql/ddl.sql
+++ b/contrib/test_decoding/sql/ddl.sql
@@ -387,4 +387,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
SELECT pg_drop_replication_slot('regression_slot');
/* check that the slot is gone */
+\x
SELECT * FROM pg_replication_slots;
+\x
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 64614b569c..01a7802ed4 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9907,6 +9907,54 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</entry>
</row>
+ <row>
+ <entry><structfield>wal_status</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry></entry>
+
+ <entry>Availability of WAL files claimed by this slot.
+ Valid values are:
+ <simplelist>
+ <member>
+ <literal>normal</literal> means that the claimed files
+ are within <varname>max_wal_size</varname>
+ </member>
+ <member>
+ <literal>keeping</literal> means that <varname>max_wal_size</varname>
+ is exceeded but still held by replication slots or
+ <varname>wal_keep_segments</varname>
+ </member>
+ <member>
+ <literal>losing</literal> means that some of the files are on the verge
+ of deletion, but can still be accessed by a session that's currently
+ reading it
+ </member>
+ <member>
+ <literal>lost</literal> means that some of them are definitely lost
+ and the session using this slot cannot continue replication.
+ This state also implies that the session using this slot has been
+ stopped.
+ </member>
+ </simplelist>
+ The last two states are seen only when
+ <xref linkend="guc-max-slot-wal-keep-size"/> is
+ non-negative. If <structfield>restart_lsn</structfield> is NULL, this
+ field is null.
+ </entry>
+ </row>
+
+ <row>
+ <entry><structfield>remain</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry></entry>
+ <entry>The amount in bytes of WAL that can be written before this slot
+ loses required WAL files.
+ If <structfield>restart_lsn</structfield> is null or
+ <structfield>wal_status</structfield> is <literal>losing</literal>
+ or <literal>lost</literal>, this field is null.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 2de21903a1..dc99c6868a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -3758,6 +3758,29 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-slot-wal-keep-size" xreflabel="max_slot_wal_keep_size">
+ <term><varname>max_slot_wal_keep_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_slot_wal_keep_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specify the maximum size of WAL files
+ that <link linkend="streaming-replication-slots">replication
+ slots</link> are allowed to retain in the <filename>pg_wal</filename>
+ directory at checkpoint time.
+ If <varname>max_slot_wal_keep_size</varname> is -1 (the default),
+ replication slots retain unlimited amount of WAL files. If
+ restart_lsn of a replication slot gets behind more than that megabytes
+ from the current LSN, the standby using the slot may no longer be able
+ to continue replication due to removal of required WAL files. You
+ can see the WAL availability of replication slots
+ in <link linkend="view-pg-replication-slots">pg_replication_slots</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-wal-sender-timeout" xreflabel="wal_sender_timeout">
<term><varname>wal_sender_timeout</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index b5d32bb720..624e5f94ad 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -925,9 +925,11 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
<xref linkend="guc-archive-command"/>.
However, these methods often result in retaining more WAL segments than
required, whereas replication slots retain only the number of segments
- known to be needed. An advantage of these methods is that they bound
- the space requirement for <literal>pg_wal</literal>; there is currently no way
- to do this using replication slots.
+ known to be needed. On the other hand, replication slots can retain so
+ many WAL segments that they fill up the space allocated
+ for <literal>pg_wal</literal>;
+ <xref linkend="guc-max-slot-wal-keep-size"/> limits the size of WAL files
+ retained by replication slots.
</para>
<para>
Similarly, <xref linkend="guc-hot-standby-feedback"/>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1951103b26..e68a89fded 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -105,6 +105,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -758,7 +759,7 @@ static ControlFileData *ControlFile = NULL;
*/
#define UsableBytesInPage (XLOG_BLCKSZ - SizeOfXLogShortPHD)
-/* Convert min_wal_size_mb and max_wal_size_mb to equivalent segment count */
+/* Convert values of GUCs measured in megabytes to equiv. segment count */
#define ConvertToXSegs(x, segsize) \
(x / ((segsize) / (1024 * 1024)))
@@ -895,6 +896,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -3929,8 +3931,56 @@ XLogGetLastRemovedSegno(void)
}
/*
- * Update the last removed segno pointer in shared memory, to reflect
- * that the given XLOG file has been removed.
+ * Scan the WAL directory then set the lastRemovedSegNo.
+ *
+ * In the case we need to know the last removed segment before the first
+ * checkpoint runs, call this function to initialize the variable by scanning
+ * the WAL directory.
+ */
+XLogSegNo
+ScanAndSetLastRemovedSegno(void)
+{
+ DIR *xldir;
+ struct dirent *xlde;
+ XLogSegNo segno;
+
+ xldir = AllocateDir(XLOGDIR);
+ while ((xlde = ReadDir(xldir, XLOGDIR)) != NULL)
+ {
+ TimeLineID tli;
+ XLogSegNo fsegno;
+
+ /* Ignore files that are not XLOG segments */
+ if (!IsXLogFileName(xlde->d_name) &&
+ !IsPartialXLogFileName(xlde->d_name))
+ continue;
+
+ XLogFromFileName(xlde->d_name, &tli, &fsegno, wal_segment_size);
+
+ /*
+ * Get minimum segment ignoring timeline ID, the same way with
+ * RemoveOldXlogFiles().
+ */
+ if (segno == 0 || fsegno < segno)
+ segno = fsegno;
+ }
+
+ FreeDir(xldir);
+
+ /* Update the last removed segno, not making retrogression. */
+ SpinLockAcquire(&XLogCtl->info_lck);
+ if (segno > XLogCtl->lastRemovedSegNo)
+ XLogCtl->lastRemovedSegNo = segno;
+ else
+ segno = XLogCtl->lastRemovedSegNo;
+ SpinLockRelease(&XLogCtl->info_lck);
+
+ return segno;
+}
+
+/*
+ * Update the last removed segno pointer in shared memory, to reflect that the
+ * given XLOG file has been removed.
*/
static void
UpdateLastRemovedPtr(char *filename)
@@ -9441,6 +9491,201 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Detect availability of the record at given targetLSN.
+ *
+ * targetLSN is restart_lsn of a slot.
+ * walsender_pid is the slot's walsender PID.
+ *
+ * Returns one of the following enum values.
+ *
+ * WALAVAIL_NORMAL means targetLSN is available because it is in the range of
+ * max_wal_size. If max_slot_wal_keep_size is smaller than max_wal_size, this
+ * state is not returned.
+ *
+ * WALAVAIL_PRESERVED means it is still available by preserving extra segments
+ * beyond max_wal_size.
+ *
+ * WALAVAIL_BEING_REMOVED means it is being removed or already removed but the
+ * replication stream on the given slot is live yet. The state may transit to
+ * WALAVAIL_PRESERVED or WALAVAIL_NORMAL state if the walsender advances
+ * restart_lsn.
+ *
+ * WALAVAIL_REMOVED means it is definitely lost. The replication stream on the
+ * slot cannot continue.
+ *
+ * returns WALAVAIL_NULL if restart_lsn is invalid.
+ */
+WalAvailability
+GetWalAvailability(XLogRecPtr restart_lsn, pid_t walsender_pid)
+{
+ XLogRecPtr currpos;
+ XLogRecPtr slotPtr;
+ XLogSegNo currSeg; /* segid of currpos */
+ XLogSegNo restartSeg; /* segid of restart_lsn */
+ XLogSegNo oldestSeg; /* actual oldest segid */
+ XLogSegNo oldestSegMaxWalSize; /* oldest segid kept by max_wal_size */
+ XLogSegNo oldestSlotSeg;/* oldest segid kept by slot */
+ uint64 keepSegs;
+
+ /* the case where the slot has never been activated */
+ if (XLogRecPtrIsInvalid(restart_lsn))
+ return WALAVAIL_INVALID_LSN;
+
+ currpos = GetXLogWriteRecPtr();
+
+ /* calculate oldest segment currently needed by slots */
+ XLByteToSeg(restart_lsn, restartSeg, wal_segment_size);
+ slotPtr = XLogGetReplicationSlotMinimumLSN();
+ oldestSlotSeg = GetOldestKeepSegment(currpos, slotPtr);
+
+ /* find the oldest extant segment file */
+ oldestSeg = XLogGetLastRemovedSegno() + 1;
+
+ /* initialize last removed segno if not yet */
+ if (oldestSeg == 1)
+ oldestSeg = ScanAndSetLastRemovedSegno() + 1;
+
+ /* calculate oldest segment by max_wal_size */
+ XLByteToSeg(currpos, currSeg, wal_segment_size);
+ keepSegs = ConvertToXSegs(max_wal_size_mb, wal_segment_size) + 1;
+
+ if (currSeg > keepSegs)
+ oldestSegMaxWalSize = currSeg - keepSegs;
+ else
+ oldestSegMaxWalSize = 1;
+
+ /*
+ * If max_slot_wal_keep_size has changed after the last call, the segment
+ * that would been kept by the current setting might have been lost by the
+ * previous setting. No point in showing normal or keeping status values if
+ * the restartSeg is known to be lost.
+ */
+ if (restartSeg >= oldestSeg)
+ {
+ /*
+ * show "normal" when restartSeg is within max_wal_size. If
+ * max_slot_wal_keep_size is smaller than max_wal_size, there's no
+ * point in showing the status.
+ */
+ if ((max_slot_wal_keep_size_mb <= 0 ||
+ max_slot_wal_keep_size_mb >= max_wal_size_mb) &&
+ oldestSegMaxWalSize <= restartSeg)
+ return WALAVAIL_NORMAL;
+
+ /* being retained by slots */
+ if (oldestSlotSeg <= restartSeg)
+ return WALAVAIL_PRESERVED;
+ }
+
+ /*
+ * The segment is already lost or being lost. If the oldest segment is just
+ * after the restartSeg, running walsender may be reading the just removed
+ * segment. The walsender may safely move to the oldest existing segment in
+ * that case.
+ */
+ if (oldestSeg == restartSeg + 1 && walsender_pid != 0)
+ return WALAVAIL_BEING_REMOVED;
+
+ /* definitely lost. the walsender can no longer restart */
+ return WALAVAIL_REMOVED;
+}
+
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
+/*
+ * Calculate remaining bytes until WAL segment for targetLSN will be removed.
+ */
+int64
+DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN)
+{
+ XLogSegNo currSeg;
+ uint64 limitSegs = 0;
+ int64 restbytes;
+ uint64 fragbytes;
+ XLogSegNo targetSeg;
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+
+ /* Calculate how far back WAL segments are preserved */
+ if (max_slot_wal_keep_size_mb >= 0)
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ if (wal_keep_segments > 0 && limitSegs < wal_keep_segments)
+ limitSegs = wal_keep_segments;
+
+ XLByteToSeg(targetLSN, targetSeg, wal_segment_size);
+
+ /* avoid underflow */
+ if (targetSeg + limitSegs < currSeg)
+ return 0;
+
+ /*
+ * This slot still has all required segments. Calculate how
+ * many LSN bytes the slot has until it loses targetLSN.
+ */
+ fragbytes = wal_segment_size - (currLSN % wal_segment_size);
+ XLogSegNoOffsetToRecPtr(targetSeg + limitSegs - currSeg,
+ fragbytes, wal_segment_size,
+ restbytes);
+
+ /*
+ * Not realistic, but make sure that it is not out of the
+ * range of int64. No problem to do so since such large values
+ * have no significant difference.
+ */
+ if (restbytes > PG_INT64_MAX)
+ restbytes = PG_INT64_MAX;
+
+ return restbytes;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9452,38 +9697,102 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ static XLogSegNo last_lost_segs = 0;
+ static int last_nslots = 0;
+ static char *last_slot_name = NULL;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo minSlotSegNo;
+ int nslots_affected = 0;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
+
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
- else
- segno = segno - wal_keep_segments;
+ Assert (max_replication_slots > 0);
+
+ XLByteToSeg(slotminptr, minSlotSegNo, wal_segment_size);
+
+ if (minSlotSegNo < minSegNo)
+ {
+ /* Some slots has lost required segments */
+ XLogSegNo lost_segs = minSegNo - minSlotSegNo;
+ ReplicationSlot *earliest = NULL;
+ char *earliest_name = NULL;
+ int i;
+
+ /* Find the most affected slot */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s =
+ &ReplicationSlotCtl->replication_slots[i];
+ XLogSegNo slotSegNo;
+
+ XLByteToSeg(s->data.restart_lsn, slotSegNo, wal_segment_size);
+
+ if (s->in_use && s->active_pid == 0 && slotSegNo < minSegNo)
+ {
+ nslots_affected++;
+
+ if (earliest == NULL ||
+ s->data.restart_lsn < earliest->data.restart_lsn)
+ earliest = s;
+ }
+ }
+
+ if (earliest)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+ earliest_name = pstrdup(NameStr(earliest->data.name));
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* Emit WARNING if something has changed */
+ if (earliest_name &&
+ (last_lost_segs != lost_segs || last_nslots != nslots_affected))
+ {
+ ereport(WARNING,
+ (errmsg_plural ("%d replication slot has lost required WAL segments by %lu segments",
+ "%d replication slots have lost required WAL segments by %lu segments",
+ nslots_affected, nslots_affected,
+ lost_segs),
+ errdetail("Most affected slot is %s.",
+ earliest_name)));
+
+ if (last_slot_name)
+ pfree(last_slot_name);
+ last_slot_name = earliest_name;
+ last_lost_segs = lost_segs;
+ last_nslots = nslots_affected;
+ }
+ }
}
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
+ /* Reset the state if no affected slots remain. */
+ if (nslots_affected == 0 && last_slot_name)
{
- XLogSegNo slotSegNo;
-
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
-
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ pfree(last_slot_name);
+ last_slot_name = NULL;
+ last_lost_segs = 0;
+ last_nslots = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 83d00c6cde..775b8b7f20 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -863,7 +863,9 @@ CREATE VIEW pg_replication_slots AS
L.xmin,
L.catalog_xmin,
L.restart_lsn,
- L.confirmed_flush_lsn
+ L.confirmed_flush_lsn,
+ L.wal_status,
+ L.remain
FROM pg_get_replication_slots() AS L
LEFT JOIN pg_database D ON (L.datoid = D.oid);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d90c7235e9..a26f7999aa 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -49,6 +49,7 @@
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/memutils.h"
/*
* Replication slot on-disk data structure.
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ce0c9127bc..47cd4375a1 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -234,7 +234,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_get_replication_slots(PG_FUNCTION_ARGS)
{
-#define PG_GET_REPLICATION_SLOTS_COLS 11
+#define PG_GET_REPLICATION_SLOTS_COLS 13
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -288,6 +288,7 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
Oid database;
NameData slot_name;
NameData plugin;
+ WalAvailability walstate;
int i;
if (!slot->in_use)
@@ -355,6 +356,42 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
else
nulls[i++] = true;
+ walstate = GetWalAvailability(restart_lsn, active_pid);
+
+ switch (walstate)
+ {
+ case WALAVAIL_INVALID_LSN:
+ nulls[i++] = true;
+ break;
+
+ case WALAVAIL_NORMAL:
+ values[i++] = CStringGetTextDatum("normal");
+ break;
+
+ case WALAVAIL_PRESERVED:
+ values[i++] = CStringGetTextDatum("keeping");
+ break;
+
+ case WALAVAIL_BEING_REMOVED:
+ values[i++] = CStringGetTextDatum("losing");
+ break;
+
+ case WALAVAIL_REMOVED:
+ values[i++] = CStringGetTextDatum("lost");
+ break;
+ }
+
+ if (max_slot_wal_keep_size_mb >= 0 &&
+ (walstate == WALAVAIL_NORMAL ||
+ walstate == WALAVAIL_PRESERVED))
+ {
+ values[i++] =
+ Int64GetDatum(DistanceToWalRemoval(GetXLogWriteRecPtr(),
+ restart_lsn));
+ }
+ else
+ nulls[i++] = true;
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
LWLockRelease(ReplicationSlotControlLock);
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 79bc7ac8ca..54cd5f6420 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2771,6 +2771,19 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum size of WAL space reserved by replication slots."),
+ gettext_noop("Replication slots will be marked as failed, and segments released "
+ "for deletion or recycling, if this much space is occupied by WAL "
+ "on disk."),
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e9f8ca775d..0b696e7044 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -287,6 +287,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..c1994eec6f 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
@@ -255,6 +256,20 @@ typedef struct CheckpointStatsData
extern CheckpointStatsData CheckpointStats;
+/*
+ * WAL segment availability status
+ *
+ * This is used as the return value of GetWalAvailability.
+ */
+typedef enum WalAvailability
+{
+ WALAVAIL_INVALID_LSN, /* parameter error */
+ WALAVAIL_NORMAL, /* WAL segment is within max_wal_size */
+ WALAVAIL_PRESERVED, /* WAL segment is preserved by repslots */
+ WALAVAIL_BEING_REMOVED, /* WAL segment is no longer preserved */
+ WALAVAIL_REMOVED /* WAL segment has been removed */
+} WalAvailability;
+
struct XLogRecData;
extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
@@ -268,6 +283,7 @@ extern int XLogFileOpen(XLogSegNo segno);
extern void CheckXLogRemoved(XLogSegNo segno, TimeLineID tli);
extern XLogSegNo XLogGetLastRemovedSegno(void);
+extern XLogSegNo ScanAndSetLastRemovedSegno(void);
extern void XLogSetAsyncXactLSN(XLogRecPtr record);
extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);
@@ -305,6 +321,9 @@ extern void ShutdownXLOG(int code, Datum arg);
extern void InitXLOGAccess(void);
extern void CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
+extern WalAvailability GetWalAvailability(XLogRecPtr restart_lsn,
+ pid_t walsender_pid);
+extern int64 DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN);
extern void XLogPutNextOid(Oid nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a6a708cca9..2025f34bfd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9971,9 +9971,9 @@
proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', prorettype => 'record',
proargtypes => '',
- proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn}',
+ proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,remain}',
prosrc => 'pg_get_replication_slots' },
{ oid => '3786', descr => 'set up a logical replication slot',
proname => 'pg_create_logical_replication_slot', provolatile => 'v',
diff --git a/src/test/recovery/t/018_replslot_limit.pl b/src/test/recovery/t/018_replslot_limit.pl
new file mode 100644
index 0000000000..84a1f3a9dd
--- /dev/null
+++ b/src/test/recovery/t/018_replslot_limit.pl
@@ -0,0 +1,202 @@
+# Test for replication slot limit
+# Ensure that max_slot_wal_keep_size limits the number of WAL files to
+# be kept by replication slots.
+
+use strict;
+use warnings;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 13;
+use Time::HiRes qw(usleep);
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node, setting wal-segsize to 1MB
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1,
+ extra => ['--wal-segsize=1'],
+ min_wal_size => '2MB',
+ max_wal_size => '4MB',
+ log_checkpoints => 'yes');
+$node_master->start;
+$node_master->safe_psql('postgres', "SELECT pg_create_physical_replication_slot('rep1')");
+
+# The slot state and remain should be null before the first connection
+my $result = $node_master->safe_psql('postgres', "SELECT restart_lsn is NULL, wal_status is NULL, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "t|t|t", 'check the state of non-reserved slot is "unknown"');
+
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create a standby linking to it using the replication slot
+my $node_standby = get_new_node('standby_1');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1,
+ primary_slot_name => 'rep1');
+
+$node_standby->start;
+
+# Wait until standby has replayed enough data
+my $start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+# Stop standby
+$node_standby->stop;
+
+
+# Preparation done, the slot is the state "normal" now
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check the catching-up state');
+
+# Advance WAL by five segments (= 5MB) on master
+advance_wal($node_master, 1);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when fitting max_wal_size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that restart_lsn is in max_wal_size');
+
+advance_wal($node_master, 4);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when max_slot_wal_keep_size is not set
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that slot is working');
+
+# The standby can reconnect to master
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+# Set max_slot_wal_keep_size on master
+my $max_slot_wal_keep_size_mb = 6;
+$node_master->append_conf('postgresql.conf', qq(
+max_slot_wal_keep_size = ${max_slot_wal_keep_size_mb}MB
+));
+$node_master->reload;
+
+# The slot is in safe state. The remaining bytes should be as almost
+# (max_slot_wal_keep_size + 1) times large as the segment size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that max_slot_wal_keep_size is working');
+
+# Advance WAL again then checkpoint, reducing remain by 2 MB.
+advance_wal($node_master, 2);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is still working
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|5120 kB", 'check that remaining byte is calculated correctly');
+
+# wal_keep_segments overrides max_slot_wal_keep_size
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 8; SELECT pg_reload_conf();");
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that wal_keep_segments overrides max_slot_wal_keep_size');
+
+# restore wal_keep_segments
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 0; SELECT pg_reload_conf();");
+
+# Advance WAL again without checkpoint, reducing remain by 4 MB.
+advance_wal($node_master, 4);
+
+# Slot gets into 'keeping' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|keeping|1024 kB", 'check that the slot state changes to "keeping"');
+
+# do checkpoint so that the next checkpoint runs too early
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# Advance WAL again without checkpoint; remain goes to 0.
+advance_wal($node_master, 1);
+
+# Slot gets into 'lost' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby still can connect to master before a checkpoint
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+ok(!find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed"),
+ 'check that required WAL segments are still available');
+
+# Advance WAL again, the slot loses the oldest segment.
+my $logstart = get_log_size($node_master);
+advance_wal($node_master, 7);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# WARNING should be issued
+ok(find_in_log($node_master,
+ "1 replication slot has lost required WAL segments by 1 segments\n".
+ ".*Most affected slot is rep1.",
+ $logstart),
+ 'check that the warning is logged');
+
+# This slot should be broken
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby no longer can connect to the master
+$logstart = get_log_size($node_standby);
+$node_standby->start;
+
+my $failed = 0;
+for (my $i = 0 ; $i < 10000 ; $i++)
+{
+ if (find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed",
+ $logstart))
+ {
+ $failed = 1;
+ last;
+ }
+ usleep(100_000);
+}
+ok($failed, 'check that replication has been broken');
+
+$node_standby->stop;
+
+#####################################
+# Advance WAL of $node by $n segments
+sub advance_wal
+{
+ my ($node, $n) = @_;
+
+ # Advance by $n segments (= (16 * $n) MB) on master
+ for (my $i = 0 ; $i < $n ; $i++)
+ {
+ $node->safe_psql('postgres', "CREATE TABLE t (); DROP TABLE t; SELECT pg_switch_wal();");
+ }
+}
+
+# return the size of logfile of $node in bytes
+sub get_log_size
+{
+ my ($node) = @_;
+
+ return (stat $node->logfile)[7];
+}
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = TestLib::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 7245b0e13b..8688f7138f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1462,8 +1462,10 @@ pg_replication_slots| SELECT l.slot_name,
l.xmin,
l.catalog_xmin,
l.restart_lsn,
- l.confirmed_flush_lsn
- FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn)
+ l.confirmed_flush_lsn,
+ l.wal_status,
+ l.remain
+ FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, remain)
LEFT JOIN pg_database d ON ((l.datoid = d.oid)));
pg_roles| SELECT pg_authid.rolname,
pg_authid.rolsuper,
--
2.18.2
----Next_Part(Tue_Mar_31_14_10_14_2020_637)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v20-0003-Allow-init-and-init_from_backup-to-set-arbitrary.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 1/6] Add WAL relief vent for replication slots
@ 2017-12-21 12:20 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2017-12-21 12:20 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
src/backend/access/transam/xlog.c | 128 +++++++++++++++++++++-----
src/backend/replication/slot.c | 62 +++++++++++++
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 1 +
src/include/replication/slot.h | 1 +
6 files changed, 180 insertions(+), 25 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f553523857..3989f6e54a 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -104,6 +104,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -872,6 +873,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -9288,6 +9290,54 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9299,38 +9349,66 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo slotSegNo;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
+
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
+ static XLogSegNo prev_lost_segs = 0; /* avoid duplicate messages */
+
+ XLByteToSeg(slotminptr, slotSegNo, wal_segment_size);
+
+ if (slotSegNo < minSegNo)
+ {
+ XLogSegNo lost_segs = minSegNo - slotSegNo;
+ if (prev_lost_segs != lost_segs)
+ {
+ /* We have lost a new segment, warn it.*/
+ XLogRecPtr minlsn;
+ char *slot_names;
+ int nslots;
+
+ XLogSegNoOffsetToRecPtr(minSegNo, 0, wal_segment_size, minlsn);
+ slot_names =
+ ReplicationSlotsEnumerateBehinds(minlsn, ", ", &nslots);
+
+ /*
+ * Some of the affected slots could have just been removed. We
+ * don't need show anything here if no affected slots are
+ * remaining.
+ */
+ if (slot_names)
+ {
+ ereport(WARNING,
+ (errmsg ("some replication slots have lost required WAL segments"),
+ errdetail_plural(
+ "Slot %s lost %ld segment(s).",
+ "Slots %s lost at most %ld segment(s).",
+ nslots, slot_names, lost_segs)));
+ }
+ }
+ prev_lost_segs = lost_segs;
+ }
else
- segno = segno - wal_keep_segments;
- }
-
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
- {
- XLogSegNo slotSegNo;
-
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
-
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ prev_lost_segs = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 62342a69cb..5bdf1e90fb 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1064,6 +1064,68 @@ ReplicationSlotReserveWal(void)
}
}
+/*
+ * Returns names of inactive replication slots that their restart_lsn are
+ * behind specified LSN for the purpose of error message, in palloc'ed
+ * character array stuffed with slot names delimited by the given
+ * separator. Returns NULL if no slot matches. If pnslots is given, the number
+ * of the returned slots is returned there.
+ */
+char *
+ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots)
+{
+ static StringInfoData retstr;
+ static bool retstr_initialized = false;
+ bool insert_separator = false;
+ int i;
+ int nslots = 0;
+
+ Assert (separator);
+ if (max_replication_slots <= 0)
+ return NULL;
+
+ if (!retstr_initialized)
+ {
+ initStringInfo(&retstr);
+ retstr_initialized = true;
+ }
+ else
+ resetStringInfo(&retstr);
+
+ /* construct name list */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ /*
+ * We are collecting slots that are definitely behind the given target
+ * LSN. Active slots are exluded since they can catch up later.
+ */
+ if (s->in_use && s->active_pid == 0 && s->data.restart_lsn < target)
+ {
+ if (insert_separator)
+ appendStringInfoString(&retstr, separator);
+
+ /*
+ * Slot names consist only with lower-case letters. We don't
+ * bother quoting.
+ */
+ appendStringInfoString(&retstr, NameStr(s->data.name));
+ insert_separator = true;
+ nslots++;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* return the number of slots in the list if requested */
+ if (pnslots)
+ *pnslots = nslots;
+
+ /* return NULL instead of an empty string */
+ return retstr.data[0] ? retstr.data : NULL;
+}
+
/*
* Flush all replication slots to disk.
*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fc463601ff..f8e796b6c1 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2654,6 +2654,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum size of extra WALs kept by replication slots."),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1,
+ MAX_KILOBYTES, /* XXX: This is in megabytes, like max/min_wal_size */
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index cfad86c02a..aadbc76d85 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -286,6 +286,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252aad..b355452072 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 8fbddea78f..e0fee0663c 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -199,6 +199,7 @@ extern void ReplicationSlotsComputeRequiredLSN(void);
extern XLogRecPtr ReplicationSlotsComputeLogicalRestartLSN(void);
extern bool ReplicationSlotsCountDBSlots(Oid dboid, int *nslots, int *nactive);
extern void ReplicationSlotsDropDBSlots(Oid dboid);
+extern char *ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
--
2.16.3
----Next_Part(Wed_Jul_31_16_56_16_2019_668)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v16-0002-Add-monitoring-aid-for-max_slot_wal_keep_size.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 1/6] Add WAL relief vent for replication slots
@ 2017-12-21 12:20 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2017-12-21 12:20 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
src/backend/access/transam/xlog.c | 128 +++++++++++++++++++++-----
src/backend/replication/slot.c | 58 ++++++++++++
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 1 +
src/include/replication/slot.h | 1 +
6 files changed, 176 insertions(+), 25 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index f553523857..fcb076100f 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -104,6 +104,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -872,6 +873,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -9288,6 +9290,54 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9299,38 +9349,66 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo slotSegNo;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
+
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
+ static XLogSegNo prev_lost_segs = 0; /* avoid duplicate messages */
+
+ XLByteToSeg(slotminptr, slotSegNo, wal_segment_size);
+
+ if (slotSegNo < minSegNo)
+ {
+ XLogSegNo lost_segs = minSegNo - slotSegNo;
+ if (prev_lost_segs != lost_segs)
+ {
+ /* We have lost a new segment, warn it.*/
+ XLogRecPtr minlsn;
+ char *slot_names;
+ int nslots;
+
+ XLogSegNoOffsetToRecPtr(minSegNo, 0, wal_segment_size, minlsn);
+ slot_names =
+ ReplicationSlotsEnumerateBehinds(minlsn, ", ", &nslots);
+
+ /*
+ * Some of the affected slots could have just been removed.
+ * We don't need show anything here if no affected slot
+ * remains.
+ */
+ if (slot_names)
+ {
+ ereport(WARNING,
+ (errmsg ("some replication slots have lost required WAL segments"),
+ errdetail_plural(
+ "Slot %s lost %ld segment(s).",
+ "Slots %s lost at most %ld segment(s).",
+ nslots, slot_names, lost_segs)));
+ }
+ }
+ prev_lost_segs = lost_segs;
+ }
else
- segno = segno - wal_keep_segments;
- }
-
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
- {
- XLogSegNo slotSegNo;
-
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
-
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ prev_lost_segs = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 62342a69cb..24b8d42eab 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1064,6 +1064,64 @@ ReplicationSlotReserveWal(void)
}
}
+/*
+ * Returns names of replication slots that their restart_lsn are behind
+ * specified LSN, in palloc'ed character array stuffed with slot names
+ * delimited by the given separator. Returns NULL if no slot matches. If
+ * pnslots is given, the number of the returned slots is returned there.
+ */
+char *
+ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots)
+{
+ static StringInfoData retstr;
+ static bool retstr_initialized = false;
+ bool insert_separator = false;
+ int i;
+ int nslots = 0;
+
+ Assert (separator);
+ if (max_replication_slots <= 0)
+ return NULL;
+
+ if (!retstr_initialized)
+ {
+ initStringInfo(&retstr);
+ retstr_initialized = true;
+ }
+ else
+ resetStringInfo(&retstr);
+
+ /* construct name list */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ /* Exclude active walsenders */
+ if (s->in_use && s->active_pid == 0 && s->data.restart_lsn < target)
+ {
+ if (insert_separator)
+ appendStringInfoString(&retstr, separator);
+
+ /*
+ * Slot names consist only with lower-case letters. We don't
+ * bother quoting.
+ */
+ appendStringInfoString(&retstr, NameStr(s->data.name));
+ insert_separator = true;
+ nslots++;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* return the number of slots in the list if requested */
+ if (pnslots)
+ *pnslots = nslots;
+
+ /* return NULL instead of an empty string */
+ return retstr.data[0] ? retstr.data : NULL;
+}
+
/*
* Flush all replication slots to disk.
*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fc463601ff..f8e796b6c1 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2654,6 +2654,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum size of extra WALs kept by replication slots."),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1,
+ MAX_KILOBYTES, /* XXX: This is in megabytes, like max/min_wal_size */
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index cfad86c02a..aadbc76d85 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -286,6 +286,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index d519252aad..b355452072 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 8fbddea78f..e0fee0663c 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -199,6 +199,7 @@ extern void ReplicationSlotsComputeRequiredLSN(void);
extern XLogRecPtr ReplicationSlotsComputeLogicalRestartLSN(void);
extern bool ReplicationSlotsCountDBSlots(Oid dboid, int *nslots, int *nactive);
extern void ReplicationSlotsDropDBSlots(Oid dboid);
+extern char *ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
--
2.16.3
----Next_Part(Tue_Jul_30_21_30_45_2019_680)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v15-0002-Add-monitoring-aid-for-max_slot_wal_keep_size.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 1/6] Add WAL relief vent for replication slots
@ 2017-12-21 12:20 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2017-12-21 12:20 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
src/backend/access/transam/xlog.c | 128 +++++++++++++++++++++-----
src/backend/replication/slot.c | 57 ++++++++++++
src/backend/utils/misc/guc.c | 12 +++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 1 +
src/include/replication/slot.h | 1 +
6 files changed, 175 insertions(+), 25 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index ecd12fc53a..998b779277 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -100,6 +100,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -872,6 +873,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -9329,6 +9331,54 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9340,38 +9390,66 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo slotSegNo;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
+
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
+ static XLogSegNo prev_lost_segs = 0; /* avoid duplicate messages */
+
+ XLByteToSeg(slotminptr, slotSegNo, wal_segment_size);
+
+ if (slotSegNo < minSegNo)
+ {
+ XLogSegNo lost_segs = minSegNo - slotSegNo;
+ if (prev_lost_segs != lost_segs)
+ {
+ /* We have lost a new segment, warn it.*/
+ XLogRecPtr minlsn;
+ char *slot_names;
+ int nslots;
+
+ XLogSegNoOffsetToRecPtr(minSegNo, 0, wal_segment_size, minlsn);
+ slot_names =
+ ReplicationSlotsEnumerateBehinds(minlsn, ", ", &nslots);
+
+ /*
+ * Some of the affected slots could have just been removed.
+ * We don't need show anything here if no affected slot
+ * remains.
+ */
+ if (slot_names)
+ {
+ ereport(WARNING,
+ (errmsg ("some replication slots have lost required WAL segments"),
+ errdetail_plural(
+ "Slot %s lost %ld segment(s).",
+ "Slots %s lost at most %ld segment(s).",
+ nslots, slot_names, lost_segs)));
+ }
+ }
+ prev_lost_segs = lost_segs;
+ }
else
- segno = segno - wal_keep_segments;
- }
-
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
- {
- XLogSegNo slotSegNo;
-
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
-
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ prev_lost_segs = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 33b23b6b6d..6ef63ae7c0 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1064,6 +1064,63 @@ ReplicationSlotReserveWal(void)
}
}
+/*
+ * Returns names of replication slots that their restart_lsn are behind
+ * specified LSN, in palloc'ed character array stuffed with slot names
+ * delimited by the given separator. Returns NULL if no slot matches. If
+ * pnslots is given, the number of the returned slots is returned there.
+ */
+char *
+ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots)
+{
+ static StringInfoData retstr;
+ static bool retstr_initialized = false;
+ bool insert_separator = false;
+ int i;
+ int nslots = 0;
+
+ Assert (separator);
+ if (max_replication_slots <= 0)
+ return NULL;
+
+ if (!retstr_initialized)
+ {
+ initStringInfo(&retstr);
+ retstr_initialized = true;
+ }
+ else
+ resetStringInfo(&retstr);
+
+ /* construct name list */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (s->in_use && s->data.restart_lsn < target)
+ {
+ if (insert_separator)
+ appendStringInfoString(&retstr, separator);
+
+ /*
+ * Slot names consist only with lower-case letters. We don't
+ * bother quoting.
+ */
+ appendStringInfoString(&retstr, NameStr(s->data.name));
+ insert_separator = true;
+ nslots++;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* return the number of slots in the list if requested */
+ if (pnslots)
+ *pnslots = nslots;
+
+ /* return NULL instead of an empty string */
+ return retstr.data[0] ? retstr.data : NULL;
+}
+
/*
* Flush all replication slots to disk.
*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 156d147c85..c5f04fb8a5 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2629,6 +2629,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum size of extra WALs kept by replication slots."),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1,
+ MAX_KILOBYTES, /* XXX: This is in megabytes, like max/min_wal_size */
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 194f312096..6f96177bbd 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -282,6 +282,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index f90a6a9139..b2eb30b779 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index a8f1d66bae..9c3635dc0e 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -199,6 +199,7 @@ extern void ReplicationSlotsComputeRequiredLSN(void);
extern XLogRecPtr ReplicationSlotsComputeLogicalRestartLSN(void);
extern bool ReplicationSlotsCountDBSlots(Oid dboid, int *nslots, int *nactive);
extern void ReplicationSlotsDropDBSlots(Oid dboid);
+extern char *ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
--
2.16.3
----Next_Part(Fri_Feb_22_14_12_28_2019_076)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v13-0002-Add-monitoring-aid-for-max_slot_wal_keep_size.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH 1/6] Add WAL relief vent for replication slots
@ 2017-12-21 12:20 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2017-12-21 12:20 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
src/backend/access/transam/xlog.c | 124 ++++++++++++++----
src/backend/replication/slot.c | 57 ++++++++
src/backend/utils/misc/guc.c | 12 ++
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 1 +
src/include/replication/slot.h | 1 +
6 files changed, 173 insertions(+), 23 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index e08320e829..9bea2a403f 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -104,6 +104,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -871,6 +872,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -9272,6 +9274,54 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9283,38 +9333,66 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo slotSegNo;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
- {
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
- else
- segno = segno - wal_keep_segments;
- }
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- XLogSegNo slotSegNo;
+ static XLogSegNo prev_lost_segs = 0; /* avoid duplicate messages */
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
+ XLByteToSeg(slotminptr, slotSegNo, wal_segment_size);
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ if (slotSegNo < minSegNo)
+ {
+ XLogSegNo lost_segs = minSegNo - slotSegNo;
+ if (prev_lost_segs != lost_segs)
+ {
+ /* We have lost a new segment, warn it.*/
+ XLogRecPtr minlsn;
+ char *slot_names;
+ int nslots;
+
+ XLogSegNoOffsetToRecPtr(minSegNo, 0, wal_segment_size, minlsn);
+ slot_names =
+ ReplicationSlotsEnumerateBehinds(minlsn, ", ", &nslots);
+
+ /*
+ * Some of the affected slots could have just been removed.
+ * We don't need show anything here if no affected slot
+ * remains.
+ */
+ if (slot_names)
+ {
+ ereport(WARNING,
+ (errmsg ("some replication slots have lost required WAL segments"),
+ errdetail_plural(
+ "Slot %s lost %ld segment(s).",
+ "Slots %s lost at most %ld segment(s).",
+ nslots, slot_names, lost_segs)));
+ }
+ }
+ prev_lost_segs = lost_segs;
+ }
+ else
+ prev_lost_segs = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index 55c306e465..7f9eab6edf 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -1064,6 +1064,63 @@ ReplicationSlotReserveWal(void)
}
}
+/*
+ * Returns names of replication slots that their restart_lsn are behind
+ * specified LSN, in palloc'ed character array stuffed with slot names
+ * delimited by the given separator. Returns NULL if no slot matches. If
+ * pnslots is given, the number of the returned slots is returned there.
+ */
+char *
+ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots)
+{
+ static StringInfoData retstr;
+ static bool retstr_initialized = false;
+ bool insert_separator = false;
+ int i;
+ int nslots = 0;
+
+ Assert (separator);
+ if (max_replication_slots <= 0)
+ return NULL;
+
+ if (!retstr_initialized)
+ {
+ initStringInfo(&retstr);
+ retstr_initialized = true;
+ }
+ else
+ resetStringInfo(&retstr);
+
+ /* construct name list */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s = &ReplicationSlotCtl->replication_slots[i];
+
+ if (s->in_use && s->data.restart_lsn < target)
+ {
+ if (insert_separator)
+ appendStringInfoString(&retstr, separator);
+
+ /*
+ * Slot names consist only with lower-case letters. We don't
+ * bother quoting.
+ */
+ appendStringInfoString(&retstr, NameStr(s->data.name));
+ insert_separator = true;
+ nslots++;
+ }
+ }
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* return the number of slots in the list if requested */
+ if (pnslots)
+ *pnslots = nslots;
+
+ /* return NULL instead of an empty string */
+ return retstr.data[0] ? retstr.data : NULL;
+}
+
/*
* Flush all replication slots to disk.
*
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 1208eb9a68..80245e9def 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2655,6 +2655,18 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum size of extra WALs kept by replication slots."),
+ NULL,
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1,
+ MAX_KILOBYTES, /* XXX: This is in megabytes, like max/min_wal_size */
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index 5ee5e09ddf..5da2706a99 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -286,6 +286,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 237f4e0350..e5322abdf5 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
diff --git a/src/include/replication/slot.h b/src/include/replication/slot.h
index 8fbddea78f..e0fee0663c 100644
--- a/src/include/replication/slot.h
+++ b/src/include/replication/slot.h
@@ -199,6 +199,7 @@ extern void ReplicationSlotsComputeRequiredLSN(void);
extern XLogRecPtr ReplicationSlotsComputeLogicalRestartLSN(void);
extern bool ReplicationSlotsCountDBSlots(Oid dboid, int *nslots, int *nactive);
extern void ReplicationSlotsDropDBSlots(Oid dboid);
+extern char *ReplicationSlotsEnumerateBehinds(XLogRecPtr target, char *separator, int *pnslots);
extern void StartupReplicationSlots(void);
extern void CheckPointReplicationSlots(void);
--
2.20.1
--MP_/68vU=6Ib.yoYOTdA.M=EVne
Content-Type: text/x-patch
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename=v14-0002-Add-monitoring-aid-for-max_slot_wal_keep_size.patch
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v19 2/3] Add WAL relief vent for replication slots
@ 2017-12-21 12:20 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2017-12-21 12:20 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
contrib/test_decoding/expected/ddl.out | 4 +-
contrib/test_decoding/sql/ddl.sql | 2 +
doc/src/sgml/catalogs.sgml | 48 +++
doc/src/sgml/config.sgml | 23 ++
doc/src/sgml/high-availability.sgml | 8 +-
src/backend/access/transam/xlog.c | 354 ++++++++++++++++--
src/backend/catalog/system_views.sql | 4 +-
src/backend/replication/slot.c | 1 +
src/backend/replication/slotfuncs.c | 39 +-
src/backend/utils/misc/guc.c | 13 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 19 +
src/include/catalog/pg_proc.dat | 6 +-
src/test/recovery/t/018_replslot_limit.pl | 202 ++++++++++
src/test/regress/expected/rules.out | 6 +-
15 files changed, 694 insertions(+), 36 deletions(-)
create mode 100644 src/test/recovery/t/018_replslot_limit.pl
diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out
index 2c999fd3eb..cf0318f697 100644
--- a/contrib/test_decoding/expected/ddl.out
+++ b/contrib/test_decoding/expected/ddl.out
@@ -723,8 +723,8 @@ SELECT pg_drop_replication_slot('regression_slot');
(1 row)
/* check that the slot is gone */
+\x
SELECT * FROM pg_replication_slots;
- slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn
------------+--------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+---------------------
(0 rows)
+\x
diff --git a/contrib/test_decoding/sql/ddl.sql b/contrib/test_decoding/sql/ddl.sql
index 856495c952..0f2b9992f7 100644
--- a/contrib/test_decoding/sql/ddl.sql
+++ b/contrib/test_decoding/sql/ddl.sql
@@ -387,4 +387,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
SELECT pg_drop_replication_slot('regression_slot');
/* check that the slot is gone */
+\x
SELECT * FROM pg_replication_slots;
+\x
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 64614b569c..01a7802ed4 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9907,6 +9907,54 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</entry>
</row>
+ <row>
+ <entry><structfield>wal_status</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry></entry>
+
+ <entry>Availability of WAL files claimed by this slot.
+ Valid values are:
+ <simplelist>
+ <member>
+ <literal>normal</literal> means that the claimed files
+ are within <varname>max_wal_size</varname>
+ </member>
+ <member>
+ <literal>keeping</literal> means that <varname>max_wal_size</varname>
+ is exceeded but still held by replication slots or
+ <varname>wal_keep_segments</varname>
+ </member>
+ <member>
+ <literal>losing</literal> means that some of the files are on the verge
+ of deletion, but can still be accessed by a session that's currently
+ reading it
+ </member>
+ <member>
+ <literal>lost</literal> means that some of them are definitely lost
+ and the session using this slot cannot continue replication.
+ This state also implies that the session using this slot has been
+ stopped.
+ </member>
+ </simplelist>
+ The last two states are seen only when
+ <xref linkend="guc-max-slot-wal-keep-size"/> is
+ non-negative. If <structfield>restart_lsn</structfield> is NULL, this
+ field is null.
+ </entry>
+ </row>
+
+ <row>
+ <entry><structfield>remain</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry></entry>
+ <entry>The amount in bytes of WAL that can be written before this slot
+ loses required WAL files.
+ If <structfield>restart_lsn</structfield> is null or
+ <structfield>wal_status</structfield> is <literal>losing</literal>
+ or <literal>lost</literal>, this field is null.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 2de21903a1..dc99c6868a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -3758,6 +3758,29 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-slot-wal-keep-size" xreflabel="max_slot_wal_keep_size">
+ <term><varname>max_slot_wal_keep_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_slot_wal_keep_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specify the maximum size of WAL files
+ that <link linkend="streaming-replication-slots">replication
+ slots</link> are allowed to retain in the <filename>pg_wal</filename>
+ directory at checkpoint time.
+ If <varname>max_slot_wal_keep_size</varname> is -1 (the default),
+ replication slots retain unlimited amount of WAL files. If
+ restart_lsn of a replication slot gets behind more than that megabytes
+ from the current LSN, the standby using the slot may no longer be able
+ to continue replication due to removal of required WAL files. You
+ can see the WAL availability of replication slots
+ in <link linkend="view-pg-replication-slots">pg_replication_slots</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-wal-sender-timeout" xreflabel="wal_sender_timeout">
<term><varname>wal_sender_timeout</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index b5d32bb720..624e5f94ad 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -925,9 +925,11 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
<xref linkend="guc-archive-command"/>.
However, these methods often result in retaining more WAL segments than
required, whereas replication slots retain only the number of segments
- known to be needed. An advantage of these methods is that they bound
- the space requirement for <literal>pg_wal</literal>; there is currently no way
- to do this using replication slots.
+ known to be needed. On the other hand, replication slots can retain so
+ many WAL segments that they fill up the space allocated
+ for <literal>pg_wal</literal>;
+ <xref linkend="guc-max-slot-wal-keep-size"/> limits the size of WAL files
+ retained by replication slots.
</para>
<para>
Similarly, <xref linkend="guc-hot-standby-feedback"/>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1951103b26..be9f218e7c 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -105,6 +105,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -758,7 +759,7 @@ static ControlFileData *ControlFile = NULL;
*/
#define UsableBytesInPage (XLOG_BLCKSZ - SizeOfXLogShortPHD)
-/* Convert min_wal_size_mb and max_wal_size_mb to equivalent segment count */
+/* Convert values of GUCs measured in megabytes to equiv. segment count */
#define ConvertToXSegs(x, segsize) \
(x / ((segsize) / (1024 * 1024)))
@@ -895,6 +896,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -3928,6 +3930,55 @@ XLogGetLastRemovedSegno(void)
return lastRemovedSegNo;
}
+/*
+ * Return the oldest WAL segment file.
+ *
+ * The returned value is XLogGetLastRemovedSegno() + 1 when that function
+ * returns a valid value. Otherwise this function scans over WAL files and
+ * finds the oldest segment at the first time, which could be very slow.
+ */
+XLogSegNo
+FindOldestXLogFileSegNo(void)
+{
+ static XLogSegNo lastFoundOldestSeg = 0;
+ DIR *xldir;
+ struct dirent *xlde;
+ XLogSegNo segno = XLogGetLastRemovedSegno();
+
+ if (segno > 0)
+ return segno + 1;
+
+ if (lastFoundOldestSeg > 0)
+ return lastFoundOldestSeg;
+
+ xldir = AllocateDir(XLOGDIR);
+ while ((xlde = ReadDir(xldir, XLOGDIR)) != NULL)
+ {
+ TimeLineID tli;
+ XLogSegNo fsegno;
+
+ /* Ignore files that are not XLOG segments */
+ if (!IsXLogFileName(xlde->d_name) &&
+ !IsPartialXLogFileName(xlde->d_name))
+ continue;
+
+ XLogFromFileName(xlde->d_name, &tli, &fsegno, wal_segment_size);
+
+ /*
+ * Get minimum segment ignoring timeline ID, the same way with
+ * RemoveOldXlogFiles().
+ */
+ if (segno == 0 || fsegno < segno)
+ segno = fsegno;
+ }
+
+ FreeDir(xldir);
+
+ lastFoundOldestSeg = segno;
+
+ return segno;
+}
+
/*
* Update the last removed segno pointer in shared memory, to reflect
* that the given XLOG file has been removed.
@@ -9441,6 +9492,197 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Detect availability of the record at given targetLSN.
+ *
+ * targetLSN is restart_lsn of a slot.
+ * walsender_pid is the slot's walsender PID.
+ *
+ * Returns one of the following enum values.
+ *
+ * WALAVAIL_NORMAL means targetLSN is available because it is in the range of
+ * max_wal_size. If max_slot_wal_keep_size is smaller than max_wal_size, this
+ * state is not returned.
+ *
+ * WALAVAIL_PRESERVED means it is still available by preserving extra segments
+ * beyond max_wal_size.
+ *
+ * WALAVAIL_BEING_REMOVED means it is being removed or already removed but the
+ * replication stream on the given slot is live yet. The state may transit to
+ * WALAVAIL_PRESERVED or WALAVAIL_NORMAL state if the walsender advances
+ * restart_lsn.
+ *
+ * WALAVAIL_REMOVED means it is definitly lost. The replication stream on the
+ * slot cannot continue.
+ *
+ * returns WALAVAIL_NULL if restart_lsn is invalid.
+ */
+WalAvailability
+GetWalAvailability(XLogRecPtr restart_lsn, pid_t walsender_pid)
+{
+ XLogRecPtr currpos;
+ XLogRecPtr slotPtr;
+ XLogSegNo currSeg; /* segid of currpos */
+ XLogSegNo restartSeg; /* segid of restart_lsn */
+ XLogSegNo oldestSeg; /* actual oldest segid */
+ XLogSegNo oldestSegMaxWalSize; /* oldest segid kept by max_wal_size */
+ XLogSegNo oldestSlotSeg;/* oldest segid kept by slot */
+ uint64 keepSegs;
+
+ /* the case where the slot has never been activated */
+ if (XLogRecPtrIsInvalid(restart_lsn))
+ return WALAVAIL_INVALID_LSN;
+
+ currpos = GetXLogWriteRecPtr();
+
+ /* calculate oldest segment currently needed by slots */
+ XLByteToSeg(restart_lsn, restartSeg, wal_segment_size);
+ slotPtr = XLogGetReplicationSlotMinimumLSN();
+ oldestSlotSeg = GetOldestKeepSegment(currpos, slotPtr);
+
+ /* find the oldest extant segment file */
+ oldestSeg = FindOldestXLogFileSegNo();
+
+ /* calculate oldest segment by max_wal_size */
+ XLByteToSeg(currpos, currSeg, wal_segment_size);
+ keepSegs = ConvertToXSegs(max_wal_size_mb, wal_segment_size) + 1;
+
+ if (currSeg > keepSegs)
+ oldestSegMaxWalSize = currSeg - keepSegs;
+ else
+ oldestSegMaxWalSize = 1;
+
+ /*
+ * If max_slot_wal_keep_size has changed after the last call, the segment
+ * that would been kept by the current setting might have been lost by the
+ * previous setting. No point in showing normal or keeping status values if
+ * the restartSeg is known to be lost.
+ */
+ if (restartSeg >= oldestSeg)
+ {
+ /*
+ * show "normal" when restartSeg is within max_wal_size. If
+ * max_slot_wal_keep_size is smaller than max_wal_size, there's no
+ * point in showing the status.
+ */
+ if ((max_slot_wal_keep_size_mb <= 0 ||
+ max_slot_wal_keep_size_mb >= max_wal_size_mb) &&
+ oldestSegMaxWalSize <= restartSeg)
+ return WALAVAIL_NORMAL;
+
+ /* being retained by slots */
+ if (oldestSlotSeg <= restartSeg)
+ return WALAVAIL_PRESERVED;
+ }
+
+ /*
+ * The segment is already lost or being lost. If the oldest segment is just
+ * after the restartSeg, running walsender may be reading the just removed
+ * segment. The walsender may safely move to the oldest existing segment in
+ * that case.
+ */
+ if (oldestSeg == restartSeg + 1 && walsender_pid != 0)
+ return WALAVAIL_BEING_REMOVED;
+
+ /* definitely lost. the walsender can no longer restart */
+ return WALAVAIL_REMOVED;
+}
+
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
+/*
+ * Calculate remaining bytes until WAL segment for targetLSN will be removed.
+ */
+int64
+DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN)
+{
+ XLogSegNo currSeg;
+ uint64 limitSegs = 0;
+ int64 restbytes;
+ uint64 fragbytes;
+ XLogSegNo targetSeg;
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+
+ /* Calculate how far back WAL segments are preserved */
+ if (max_slot_wal_keep_size_mb >= 0)
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ if (wal_keep_segments > 0 && limitSegs < wal_keep_segments)
+ limitSegs = wal_keep_segments;
+
+ XLByteToSeg(targetLSN, targetSeg, wal_segment_size);
+
+ /* avoid underflow */
+ if (targetSeg + limitSegs < currSeg)
+ return 0;
+
+ /*
+ * This slot still has all required segments. Calculate how
+ * many LSN bytes the slot has until it loses targetLSN.
+ */
+ fragbytes = wal_segment_size - (currLSN % wal_segment_size);
+ XLogSegNoOffsetToRecPtr(targetSeg + limitSegs - currSeg,
+ fragbytes, wal_segment_size,
+ restbytes);
+
+ /*
+ * not realistic, but make sure that it is not out of the
+ * range of int64. No problem to do so since such large values
+ * have no significant difference.
+ */
+ if (restbytes > PG_INT64_MAX)
+ restbytes = PG_INT64_MAX;
+
+ return restbytes;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9452,38 +9694,102 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ static XLogSegNo last_lost_segs = 0;
+ static int last_nslots = 0;
+ static char *last_slot_name = NULL;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo minSlotSegNo;
+ int nslots_affected = 0;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
+
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
- else
- segno = segno - wal_keep_segments;
+ Assert (max_replication_slots > 0);
+
+ XLByteToSeg(slotminptr, minSlotSegNo, wal_segment_size);
+
+ if (minSlotSegNo < minSegNo)
+ {
+ /* Some slots has lost requred segments */
+ XLogSegNo lost_segs = minSegNo - minSlotSegNo;
+ ReplicationSlot *earliest = NULL;
+ char *earliest_name = NULL;
+ int i;
+
+ /* Find the most affected slot */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s =
+ &ReplicationSlotCtl->replication_slots[i];
+ XLogSegNo slotSegNo;
+
+ XLByteToSeg(s->data.restart_lsn, slotSegNo, wal_segment_size);
+
+ if (s->in_use && s->active_pid == 0 && slotSegNo < minSegNo)
+ {
+ nslots_affected++;
+
+ if (earliest == NULL ||
+ s->data.restart_lsn < earliest->data.restart_lsn)
+ earliest = s;
+ }
+ }
+
+ if (earliest)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+ earliest_name = pstrdup(NameStr(earliest->data.name));
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* Emit WARNING if something has changed */
+ if (earliest_name &&
+ (last_lost_segs != lost_segs || last_nslots != nslots_affected))
+ {
+ ereport(WARNING,
+ (errmsg_plural ("%d replication slot has lost required WAL segments by %lu segments",
+ "%d replication slots have lost required WAL segments by %lu segments",
+ nslots_affected, nslots_affected,
+ lost_segs),
+ errdetail("Most affected slot is %s.",
+ earliest_name)));
+
+ if (last_slot_name)
+ pfree(last_slot_name);
+ last_slot_name = earliest_name;
+ last_lost_segs = lost_segs;
+ last_nslots = nslots_affected;
+ }
+ }
}
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
+ /* Reset the state if no affected slots remain. */
+ if (nslots_affected == 0 && last_slot_name)
{
- XLogSegNo slotSegNo;
-
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
-
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ pfree(last_slot_name);
+ last_slot_name = NULL;
+ last_lost_segs = 0;
+ last_nslots = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 83d00c6cde..775b8b7f20 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -863,7 +863,9 @@ CREATE VIEW pg_replication_slots AS
L.xmin,
L.catalog_xmin,
L.restart_lsn,
- L.confirmed_flush_lsn
+ L.confirmed_flush_lsn,
+ L.wal_status,
+ L.remain
FROM pg_get_replication_slots() AS L
LEFT JOIN pg_database D ON (L.datoid = D.oid);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d90c7235e9..a26f7999aa 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -49,6 +49,7 @@
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/memutils.h"
/*
* Replication slot on-disk data structure.
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ce0c9127bc..47cd4375a1 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -234,7 +234,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_get_replication_slots(PG_FUNCTION_ARGS)
{
-#define PG_GET_REPLICATION_SLOTS_COLS 11
+#define PG_GET_REPLICATION_SLOTS_COLS 13
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -288,6 +288,7 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
Oid database;
NameData slot_name;
NameData plugin;
+ WalAvailability walstate;
int i;
if (!slot->in_use)
@@ -355,6 +356,42 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
else
nulls[i++] = true;
+ walstate = GetWalAvailability(restart_lsn, active_pid);
+
+ switch (walstate)
+ {
+ case WALAVAIL_INVALID_LSN:
+ nulls[i++] = true;
+ break;
+
+ case WALAVAIL_NORMAL:
+ values[i++] = CStringGetTextDatum("normal");
+ break;
+
+ case WALAVAIL_PRESERVED:
+ values[i++] = CStringGetTextDatum("keeping");
+ break;
+
+ case WALAVAIL_BEING_REMOVED:
+ values[i++] = CStringGetTextDatum("losing");
+ break;
+
+ case WALAVAIL_REMOVED:
+ values[i++] = CStringGetTextDatum("lost");
+ break;
+ }
+
+ if (max_slot_wal_keep_size_mb >= 0 &&
+ (walstate == WALAVAIL_NORMAL ||
+ walstate == WALAVAIL_PRESERVED))
+ {
+ values[i++] =
+ Int64GetDatum(DistanceToWalRemoval(GetXLogWriteRecPtr(),
+ restart_lsn));
+ }
+ else
+ nulls[i++] = true;
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
LWLockRelease(ReplicationSlotControlLock);
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 79bc7ac8ca..54cd5f6420 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2771,6 +2771,19 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum size of WAL space reserved by replication slots."),
+ gettext_noop("Replication slots will be marked as failed, and segments released "
+ "for deletion or recycling, if this much space is occupied by WAL "
+ "on disk."),
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e9f8ca775d..0b696e7044 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -287,6 +287,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..16d33b41f7 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
@@ -255,6 +256,20 @@ typedef struct CheckpointStatsData
extern CheckpointStatsData CheckpointStats;
+/*
+ * WAL segment availability status
+ *
+ * This is used as the return value of GetWalAvailability.
+ */
+typedef enum WalAvailability
+{
+ WALAVAIL_INVALID_LSN, /* parameter errror */
+ WALAVAIL_NORMAL, /* WAL segment is within max_wal_size */
+ WALAVAIL_PRESERVED, /* WAL segment is preserved by repslots */
+ WALAVAIL_BEING_REMOVED, /* WAL segment is no longer preserved */
+ WALAVAIL_REMOVED /* WAL segment has been removed */
+} WalAvailability;
+
struct XLogRecData;
extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
@@ -268,6 +283,7 @@ extern int XLogFileOpen(XLogSegNo segno);
extern void CheckXLogRemoved(XLogSegNo segno, TimeLineID tli);
extern XLogSegNo XLogGetLastRemovedSegno(void);
+extern XLogSegNo FindOldestXLogFileSegNo(void);
extern void XLogSetAsyncXactLSN(XLogRecPtr record);
extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);
@@ -305,6 +321,9 @@ extern void ShutdownXLOG(int code, Datum arg);
extern void InitXLOGAccess(void);
extern void CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
+extern WalAvailability GetWalAvailability(XLogRecPtr restart_lsn,
+ pid_t walsender_pid);
+extern int64 DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN);
extern void XLogPutNextOid(Oid nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a6a708cca9..2025f34bfd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9971,9 +9971,9 @@
proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', prorettype => 'record',
proargtypes => '',
- proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn}',
+ proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,remain}',
prosrc => 'pg_get_replication_slots' },
{ oid => '3786', descr => 'set up a logical replication slot',
proname => 'pg_create_logical_replication_slot', provolatile => 'v',
diff --git a/src/test/recovery/t/018_replslot_limit.pl b/src/test/recovery/t/018_replslot_limit.pl
new file mode 100644
index 0000000000..6688167546
--- /dev/null
+++ b/src/test/recovery/t/018_replslot_limit.pl
@@ -0,0 +1,202 @@
+# Test for replication slot limit
+# Ensure that max_slot_wal_keep_size limits the number of WAL files to
+# be kept by replication slots.
+
+use strict;
+use warnings;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 13;
+use Time::HiRes qw(usleep);
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node, setting wal-segsize to 1MB
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
+$node_master->append_conf('postgresql.conf', qq(
+min_wal_size = 2MB
+max_wal_size = 4MB
+log_checkpoints = yes
+));
+$node_master->start;
+$node_master->safe_psql('postgres', "SELECT pg_create_physical_replication_slot('rep1')");
+
+# The slot state and remain should be null before the first connection
+my $result = $node_master->safe_psql('postgres', "SELECT restart_lsn is NULL, wal_status is NULL, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "t|t|t", 'check the state of non-reserved slot is "unknown"');
+
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create a standby linking to it using the replication slot
+my $node_standby = get_new_node('standby_1');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1, primary_slot_name => 'rep1');
+
+$node_standby->start;
+
+# Wait until standby has replayed enough data
+my $start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+# Stop standby
+$node_standby->stop;
+
+
+# Preparation done, the slot is the state "normal" now
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check the catching-up state');
+
+# Advance WAL by five segments (= 5MB) on master
+advance_wal($node_master, 1);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when fitting max_wal_size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that restart_lsn is in max_wal_size');
+
+advance_wal($node_master, 4);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when max_slot_wal_keep_size is not set
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that slot is working');
+
+# The standby can reconnect to master
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+# Set max_slot_wal_keep_size on master
+my $max_slot_wal_keep_size_mb = 6;
+$node_master->append_conf('postgresql.conf', qq(
+max_slot_wal_keep_size = ${max_slot_wal_keep_size_mb}MB
+));
+$node_master->reload;
+
+# The slot is in safe state. The remaining bytes should be as almost
+# (max_slot_wal_keep_size + 1) times large as the segment size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that max_slot_wal_keep_size is working');
+
+# Advance WAL again then checkpoint, reducing remain by 2 MB.
+advance_wal($node_master, 2);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is still working
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|5120 kB", 'check that remaining byte is calculated correctly');
+
+# wal_keep_segments overrides max_slot_wal_keep_size
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 8; SELECT pg_reload_conf();");
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that wal_keep_segments overrides max_slot_wal_keep_size');
+
+# restore wal_keep_segments
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 0; SELECT pg_reload_conf();");
+
+# Advance WAL again without checkpoint, reducing remain by 4 MB.
+advance_wal($node_master, 4);
+
+# Slot gets into 'keeping' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|keeping|1024 kB", 'check that the slot state changes to "keeping"');
+
+# do checkpoint so that the next checkpoint runs too early
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# Advance WAL again without checkpoint; remain goes to 0.
+advance_wal($node_master, 1);
+
+# Slot gets into 'lost' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby still can connect to master before a checkpoint
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+ok(!find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed"),
+ 'check that required WAL segments are still available');
+
+# Advance WAL again, the slot loses the oldest segment.
+my $logstart = get_log_size($node_master);
+advance_wal($node_master, 7);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# WARNING should be issued
+ok(find_in_log($node_master,
+ "1 replication slot has lost required WAL segments by 1 segments\n".
+ ".*Most affected slot is rep1.",
+ $logstart),
+ 'check that the warning is logged');
+
+# This slot should be broken
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby no longer can connect to the master
+$logstart = get_log_size($node_standby);
+$node_standby->start;
+
+my $failed = 0;
+for (my $i = 0 ; $i < 10000 ; $i++)
+{
+ if (find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed",
+ $logstart))
+ {
+ $failed = 1;
+ last;
+ }
+ usleep(100_000);
+}
+ok($failed, 'check that replication has been broken');
+
+$node_standby->stop;
+
+#####################################
+# Advance WAL of $node by $n segments
+sub advance_wal
+{
+ my ($node, $n) = @_;
+
+ # Advance by $n segments (= (16 * $n) MB) on master
+ for (my $i = 0 ; $i < $n ; $i++)
+ {
+ $node->safe_psql('postgres', "CREATE TABLE t (); DROP TABLE t; SELECT pg_switch_wal();");
+ }
+}
+
+# return the size of logfile of $node in bytes
+sub get_log_size
+{
+ my ($node) = @_;
+
+ return (stat $node->logfile)[7];
+}
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = TestLib::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 7245b0e13b..8688f7138f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1462,8 +1462,10 @@ pg_replication_slots| SELECT l.slot_name,
l.xmin,
l.catalog_xmin,
l.restart_lsn,
- l.confirmed_flush_lsn
- FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn)
+ l.confirmed_flush_lsn,
+ l.wal_status,
+ l.remain
+ FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, remain)
LEFT JOIN pg_database d ON ((l.datoid = d.oid)));
pg_roles| SELECT pg_authid.rolname,
pg_authid.rolsuper,
--
2.20.1
--d6Gm4EdcadzBjdND
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v19-0003-Check-removal-of-in-reading-segment-file.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v20 2/3] Add WAL relief vent for replication slots
@ 2017-12-21 12:20 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2017-12-21 12:20 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
contrib/test_decoding/expected/ddl.out | 4 +-
contrib/test_decoding/sql/ddl.sql | 2 +
doc/src/sgml/catalogs.sgml | 48 +++
doc/src/sgml/config.sgml | 23 ++
doc/src/sgml/high-availability.sgml | 8 +-
src/backend/access/transam/xlog.c | 361 ++++++++++++++++--
src/backend/catalog/system_views.sql | 4 +-
src/backend/replication/slot.c | 1 +
src/backend/replication/slotfuncs.c | 39 +-
src/backend/utils/misc/guc.c | 13 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 19 +
src/include/catalog/pg_proc.dat | 6 +-
src/test/recovery/t/018_replslot_limit.pl | 202 ++++++++++
src/test/regress/expected/rules.out | 6 +-
15 files changed, 699 insertions(+), 38 deletions(-)
create mode 100644 src/test/recovery/t/018_replslot_limit.pl
diff --git a/contrib/test_decoding/expected/ddl.out b/contrib/test_decoding/expected/ddl.out
index 2c999fd3eb..cf0318f697 100644
--- a/contrib/test_decoding/expected/ddl.out
+++ b/contrib/test_decoding/expected/ddl.out
@@ -723,8 +723,8 @@ SELECT pg_drop_replication_slot('regression_slot');
(1 row)
/* check that the slot is gone */
+\x
SELECT * FROM pg_replication_slots;
- slot_name | plugin | slot_type | datoid | database | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn
------------+--------+-----------+--------+----------+-----------+--------+------------+------+--------------+-------------+---------------------
(0 rows)
+\x
diff --git a/contrib/test_decoding/sql/ddl.sql b/contrib/test_decoding/sql/ddl.sql
index 856495c952..0f2b9992f7 100644
--- a/contrib/test_decoding/sql/ddl.sql
+++ b/contrib/test_decoding/sql/ddl.sql
@@ -387,4 +387,6 @@ SELECT data FROM pg_logical_slot_get_changes('regression_slot', NULL, NULL, 'inc
SELECT pg_drop_replication_slot('regression_slot');
/* check that the slot is gone */
+\x
SELECT * FROM pg_replication_slots;
+\x
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 64614b569c..01a7802ed4 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9907,6 +9907,54 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</entry>
</row>
+ <row>
+ <entry><structfield>wal_status</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry></entry>
+
+ <entry>Availability of WAL files claimed by this slot.
+ Valid values are:
+ <simplelist>
+ <member>
+ <literal>normal</literal> means that the claimed files
+ are within <varname>max_wal_size</varname>
+ </member>
+ <member>
+ <literal>keeping</literal> means that <varname>max_wal_size</varname>
+ is exceeded but still held by replication slots or
+ <varname>wal_keep_segments</varname>
+ </member>
+ <member>
+ <literal>losing</literal> means that some of the files are on the verge
+ of deletion, but can still be accessed by a session that's currently
+ reading it
+ </member>
+ <member>
+ <literal>lost</literal> means that some of them are definitely lost
+ and the session using this slot cannot continue replication.
+ This state also implies that the session using this slot has been
+ stopped.
+ </member>
+ </simplelist>
+ The last two states are seen only when
+ <xref linkend="guc-max-slot-wal-keep-size"/> is
+ non-negative. If <structfield>restart_lsn</structfield> is NULL, this
+ field is null.
+ </entry>
+ </row>
+
+ <row>
+ <entry><structfield>remain</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry></entry>
+ <entry>The amount in bytes of WAL that can be written before this slot
+ loses required WAL files.
+ If <structfield>restart_lsn</structfield> is null or
+ <structfield>wal_status</structfield> is <literal>losing</literal>
+ or <literal>lost</literal>, this field is null.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 2de21903a1..dc99c6868a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -3758,6 +3758,29 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-slot-wal-keep-size" xreflabel="max_slot_wal_keep_size">
+ <term><varname>max_slot_wal_keep_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_slot_wal_keep_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specify the maximum size of WAL files
+ that <link linkend="streaming-replication-slots">replication
+ slots</link> are allowed to retain in the <filename>pg_wal</filename>
+ directory at checkpoint time.
+ If <varname>max_slot_wal_keep_size</varname> is -1 (the default),
+ replication slots retain unlimited amount of WAL files. If
+ restart_lsn of a replication slot gets behind more than that megabytes
+ from the current LSN, the standby using the slot may no longer be able
+ to continue replication due to removal of required WAL files. You
+ can see the WAL availability of replication slots
+ in <link linkend="view-pg-replication-slots">pg_replication_slots</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-wal-sender-timeout" xreflabel="wal_sender_timeout">
<term><varname>wal_sender_timeout</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index b5d32bb720..624e5f94ad 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -925,9 +925,11 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
<xref linkend="guc-archive-command"/>.
However, these methods often result in retaining more WAL segments than
required, whereas replication slots retain only the number of segments
- known to be needed. An advantage of these methods is that they bound
- the space requirement for <literal>pg_wal</literal>; there is currently no way
- to do this using replication slots.
+ known to be needed. On the other hand, replication slots can retain so
+ many WAL segments that they fill up the space allocated
+ for <literal>pg_wal</literal>;
+ <xref linkend="guc-max-slot-wal-keep-size"/> limits the size of WAL files
+ retained by replication slots.
</para>
<para>
Similarly, <xref linkend="guc-hot-standby-feedback"/>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 1951103b26..e68a89fded 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -105,6 +105,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -758,7 +759,7 @@ static ControlFileData *ControlFile = NULL;
*/
#define UsableBytesInPage (XLOG_BLCKSZ - SizeOfXLogShortPHD)
-/* Convert min_wal_size_mb and max_wal_size_mb to equivalent segment count */
+/* Convert values of GUCs measured in megabytes to equiv. segment count */
#define ConvertToXSegs(x, segsize) \
(x / ((segsize) / (1024 * 1024)))
@@ -895,6 +896,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -3929,8 +3931,56 @@ XLogGetLastRemovedSegno(void)
}
/*
- * Update the last removed segno pointer in shared memory, to reflect
- * that the given XLOG file has been removed.
+ * Scan the WAL directory then set the lastRemovedSegNo.
+ *
+ * In the case we need to know the last removed segment before the first
+ * checkpoint runs, call this function to initialize the variable by scanning
+ * the WAL directory.
+ */
+XLogSegNo
+ScanAndSetLastRemovedSegno(void)
+{
+ DIR *xldir;
+ struct dirent *xlde;
+ XLogSegNo segno;
+
+ xldir = AllocateDir(XLOGDIR);
+ while ((xlde = ReadDir(xldir, XLOGDIR)) != NULL)
+ {
+ TimeLineID tli;
+ XLogSegNo fsegno;
+
+ /* Ignore files that are not XLOG segments */
+ if (!IsXLogFileName(xlde->d_name) &&
+ !IsPartialXLogFileName(xlde->d_name))
+ continue;
+
+ XLogFromFileName(xlde->d_name, &tli, &fsegno, wal_segment_size);
+
+ /*
+ * Get minimum segment ignoring timeline ID, the same way with
+ * RemoveOldXlogFiles().
+ */
+ if (segno == 0 || fsegno < segno)
+ segno = fsegno;
+ }
+
+ FreeDir(xldir);
+
+ /* Update the last removed segno, not making retrogression. */
+ SpinLockAcquire(&XLogCtl->info_lck);
+ if (segno > XLogCtl->lastRemovedSegNo)
+ XLogCtl->lastRemovedSegNo = segno;
+ else
+ segno = XLogCtl->lastRemovedSegNo;
+ SpinLockRelease(&XLogCtl->info_lck);
+
+ return segno;
+}
+
+/*
+ * Update the last removed segno pointer in shared memory, to reflect that the
+ * given XLOG file has been removed.
*/
static void
UpdateLastRemovedPtr(char *filename)
@@ -9441,6 +9491,201 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Detect availability of the record at given targetLSN.
+ *
+ * targetLSN is restart_lsn of a slot.
+ * walsender_pid is the slot's walsender PID.
+ *
+ * Returns one of the following enum values.
+ *
+ * WALAVAIL_NORMAL means targetLSN is available because it is in the range of
+ * max_wal_size. If max_slot_wal_keep_size is smaller than max_wal_size, this
+ * state is not returned.
+ *
+ * WALAVAIL_PRESERVED means it is still available by preserving extra segments
+ * beyond max_wal_size.
+ *
+ * WALAVAIL_BEING_REMOVED means it is being removed or already removed but the
+ * replication stream on the given slot is live yet. The state may transit to
+ * WALAVAIL_PRESERVED or WALAVAIL_NORMAL state if the walsender advances
+ * restart_lsn.
+ *
+ * WALAVAIL_REMOVED means it is definitely lost. The replication stream on the
+ * slot cannot continue.
+ *
+ * returns WALAVAIL_NULL if restart_lsn is invalid.
+ */
+WalAvailability
+GetWalAvailability(XLogRecPtr restart_lsn, pid_t walsender_pid)
+{
+ XLogRecPtr currpos;
+ XLogRecPtr slotPtr;
+ XLogSegNo currSeg; /* segid of currpos */
+ XLogSegNo restartSeg; /* segid of restart_lsn */
+ XLogSegNo oldestSeg; /* actual oldest segid */
+ XLogSegNo oldestSegMaxWalSize; /* oldest segid kept by max_wal_size */
+ XLogSegNo oldestSlotSeg;/* oldest segid kept by slot */
+ uint64 keepSegs;
+
+ /* the case where the slot has never been activated */
+ if (XLogRecPtrIsInvalid(restart_lsn))
+ return WALAVAIL_INVALID_LSN;
+
+ currpos = GetXLogWriteRecPtr();
+
+ /* calculate oldest segment currently needed by slots */
+ XLByteToSeg(restart_lsn, restartSeg, wal_segment_size);
+ slotPtr = XLogGetReplicationSlotMinimumLSN();
+ oldestSlotSeg = GetOldestKeepSegment(currpos, slotPtr);
+
+ /* find the oldest extant segment file */
+ oldestSeg = XLogGetLastRemovedSegno() + 1;
+
+ /* initialize last removed segno if not yet */
+ if (oldestSeg == 1)
+ oldestSeg = ScanAndSetLastRemovedSegno() + 1;
+
+ /* calculate oldest segment by max_wal_size */
+ XLByteToSeg(currpos, currSeg, wal_segment_size);
+ keepSegs = ConvertToXSegs(max_wal_size_mb, wal_segment_size) + 1;
+
+ if (currSeg > keepSegs)
+ oldestSegMaxWalSize = currSeg - keepSegs;
+ else
+ oldestSegMaxWalSize = 1;
+
+ /*
+ * If max_slot_wal_keep_size has changed after the last call, the segment
+ * that would been kept by the current setting might have been lost by the
+ * previous setting. No point in showing normal or keeping status values if
+ * the restartSeg is known to be lost.
+ */
+ if (restartSeg >= oldestSeg)
+ {
+ /*
+ * show "normal" when restartSeg is within max_wal_size. If
+ * max_slot_wal_keep_size is smaller than max_wal_size, there's no
+ * point in showing the status.
+ */
+ if ((max_slot_wal_keep_size_mb <= 0 ||
+ max_slot_wal_keep_size_mb >= max_wal_size_mb) &&
+ oldestSegMaxWalSize <= restartSeg)
+ return WALAVAIL_NORMAL;
+
+ /* being retained by slots */
+ if (oldestSlotSeg <= restartSeg)
+ return WALAVAIL_PRESERVED;
+ }
+
+ /*
+ * The segment is already lost or being lost. If the oldest segment is just
+ * after the restartSeg, running walsender may be reading the just removed
+ * segment. The walsender may safely move to the oldest existing segment in
+ * that case.
+ */
+ if (oldestSeg == restartSeg + 1 && walsender_pid != 0)
+ return WALAVAIL_BEING_REMOVED;
+
+ /* definitely lost. the walsender can no longer restart */
+ return WALAVAIL_REMOVED;
+}
+
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
+/*
+ * Calculate remaining bytes until WAL segment for targetLSN will be removed.
+ */
+int64
+DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN)
+{
+ XLogSegNo currSeg;
+ uint64 limitSegs = 0;
+ int64 restbytes;
+ uint64 fragbytes;
+ XLogSegNo targetSeg;
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+
+ /* Calculate how far back WAL segments are preserved */
+ if (max_slot_wal_keep_size_mb >= 0)
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ if (wal_keep_segments > 0 && limitSegs < wal_keep_segments)
+ limitSegs = wal_keep_segments;
+
+ XLByteToSeg(targetLSN, targetSeg, wal_segment_size);
+
+ /* avoid underflow */
+ if (targetSeg + limitSegs < currSeg)
+ return 0;
+
+ /*
+ * This slot still has all required segments. Calculate how
+ * many LSN bytes the slot has until it loses targetLSN.
+ */
+ fragbytes = wal_segment_size - (currLSN % wal_segment_size);
+ XLogSegNoOffsetToRecPtr(targetSeg + limitSegs - currSeg,
+ fragbytes, wal_segment_size,
+ restbytes);
+
+ /*
+ * Not realistic, but make sure that it is not out of the
+ * range of int64. No problem to do so since such large values
+ * have no significant difference.
+ */
+ if (restbytes > PG_INT64_MAX)
+ restbytes = PG_INT64_MAX;
+
+ return restbytes;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9452,38 +9697,102 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ static XLogSegNo last_lost_segs = 0;
+ static int last_nslots = 0;
+ static char *last_slot_name = NULL;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo minSlotSegNo;
+ int nslots_affected = 0;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
+
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
- else
- segno = segno - wal_keep_segments;
+ Assert (max_replication_slots > 0);
+
+ XLByteToSeg(slotminptr, minSlotSegNo, wal_segment_size);
+
+ if (minSlotSegNo < minSegNo)
+ {
+ /* Some slots has lost required segments */
+ XLogSegNo lost_segs = minSegNo - minSlotSegNo;
+ ReplicationSlot *earliest = NULL;
+ char *earliest_name = NULL;
+ int i;
+
+ /* Find the most affected slot */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s =
+ &ReplicationSlotCtl->replication_slots[i];
+ XLogSegNo slotSegNo;
+
+ XLByteToSeg(s->data.restart_lsn, slotSegNo, wal_segment_size);
+
+ if (s->in_use && s->active_pid == 0 && slotSegNo < minSegNo)
+ {
+ nslots_affected++;
+
+ if (earliest == NULL ||
+ s->data.restart_lsn < earliest->data.restart_lsn)
+ earliest = s;
+ }
+ }
+
+ if (earliest)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+ earliest_name = pstrdup(NameStr(earliest->data.name));
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* Emit WARNING if something has changed */
+ if (earliest_name &&
+ (last_lost_segs != lost_segs || last_nslots != nslots_affected))
+ {
+ ereport(WARNING,
+ (errmsg_plural ("%d replication slot has lost required WAL segments by %lu segments",
+ "%d replication slots have lost required WAL segments by %lu segments",
+ nslots_affected, nslots_affected,
+ lost_segs),
+ errdetail("Most affected slot is %s.",
+ earliest_name)));
+
+ if (last_slot_name)
+ pfree(last_slot_name);
+ last_slot_name = earliest_name;
+ last_lost_segs = lost_segs;
+ last_nslots = nslots_affected;
+ }
+ }
}
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
+ /* Reset the state if no affected slots remain. */
+ if (nslots_affected == 0 && last_slot_name)
{
- XLogSegNo slotSegNo;
-
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
-
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ pfree(last_slot_name);
+ last_slot_name = NULL;
+ last_lost_segs = 0;
+ last_nslots = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 83d00c6cde..775b8b7f20 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -863,7 +863,9 @@ CREATE VIEW pg_replication_slots AS
L.xmin,
L.catalog_xmin,
L.restart_lsn,
- L.confirmed_flush_lsn
+ L.confirmed_flush_lsn,
+ L.wal_status,
+ L.remain
FROM pg_get_replication_slots() AS L
LEFT JOIN pg_database D ON (L.datoid = D.oid);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d90c7235e9..a26f7999aa 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -49,6 +49,7 @@
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/memutils.h"
/*
* Replication slot on-disk data structure.
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ce0c9127bc..47cd4375a1 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -234,7 +234,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_get_replication_slots(PG_FUNCTION_ARGS)
{
-#define PG_GET_REPLICATION_SLOTS_COLS 11
+#define PG_GET_REPLICATION_SLOTS_COLS 13
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -288,6 +288,7 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
Oid database;
NameData slot_name;
NameData plugin;
+ WalAvailability walstate;
int i;
if (!slot->in_use)
@@ -355,6 +356,42 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
else
nulls[i++] = true;
+ walstate = GetWalAvailability(restart_lsn, active_pid);
+
+ switch (walstate)
+ {
+ case WALAVAIL_INVALID_LSN:
+ nulls[i++] = true;
+ break;
+
+ case WALAVAIL_NORMAL:
+ values[i++] = CStringGetTextDatum("normal");
+ break;
+
+ case WALAVAIL_PRESERVED:
+ values[i++] = CStringGetTextDatum("keeping");
+ break;
+
+ case WALAVAIL_BEING_REMOVED:
+ values[i++] = CStringGetTextDatum("losing");
+ break;
+
+ case WALAVAIL_REMOVED:
+ values[i++] = CStringGetTextDatum("lost");
+ break;
+ }
+
+ if (max_slot_wal_keep_size_mb >= 0 &&
+ (walstate == WALAVAIL_NORMAL ||
+ walstate == WALAVAIL_PRESERVED))
+ {
+ values[i++] =
+ Int64GetDatum(DistanceToWalRemoval(GetXLogWriteRecPtr(),
+ restart_lsn));
+ }
+ else
+ nulls[i++] = true;
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
LWLockRelease(ReplicationSlotControlLock);
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 79bc7ac8ca..54cd5f6420 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2771,6 +2771,19 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum size of WAL space reserved by replication slots."),
+ gettext_noop("Replication slots will be marked as failed, and segments released "
+ "for deletion or recycling, if this much space is occupied by WAL "
+ "on disk."),
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e9f8ca775d..0b696e7044 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -287,6 +287,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..c1994eec6f 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
@@ -255,6 +256,20 @@ typedef struct CheckpointStatsData
extern CheckpointStatsData CheckpointStats;
+/*
+ * WAL segment availability status
+ *
+ * This is used as the return value of GetWalAvailability.
+ */
+typedef enum WalAvailability
+{
+ WALAVAIL_INVALID_LSN, /* parameter error */
+ WALAVAIL_NORMAL, /* WAL segment is within max_wal_size */
+ WALAVAIL_PRESERVED, /* WAL segment is preserved by repslots */
+ WALAVAIL_BEING_REMOVED, /* WAL segment is no longer preserved */
+ WALAVAIL_REMOVED /* WAL segment has been removed */
+} WalAvailability;
+
struct XLogRecData;
extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
@@ -268,6 +283,7 @@ extern int XLogFileOpen(XLogSegNo segno);
extern void CheckXLogRemoved(XLogSegNo segno, TimeLineID tli);
extern XLogSegNo XLogGetLastRemovedSegno(void);
+extern XLogSegNo ScanAndSetLastRemovedSegno(void);
extern void XLogSetAsyncXactLSN(XLogRecPtr record);
extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);
@@ -305,6 +321,9 @@ extern void ShutdownXLOG(int code, Datum arg);
extern void InitXLOGAccess(void);
extern void CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
+extern WalAvailability GetWalAvailability(XLogRecPtr restart_lsn,
+ pid_t walsender_pid);
+extern int64 DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN);
extern void XLogPutNextOid(Oid nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a6a708cca9..2025f34bfd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9971,9 +9971,9 @@
proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', prorettype => 'record',
proargtypes => '',
- proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn}',
+ proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,remain}',
prosrc => 'pg_get_replication_slots' },
{ oid => '3786', descr => 'set up a logical replication slot',
proname => 'pg_create_logical_replication_slot', provolatile => 'v',
diff --git a/src/test/recovery/t/018_replslot_limit.pl b/src/test/recovery/t/018_replslot_limit.pl
new file mode 100644
index 0000000000..84a1f3a9dd
--- /dev/null
+++ b/src/test/recovery/t/018_replslot_limit.pl
@@ -0,0 +1,202 @@
+# Test for replication slot limit
+# Ensure that max_slot_wal_keep_size limits the number of WAL files to
+# be kept by replication slots.
+
+use strict;
+use warnings;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 13;
+use Time::HiRes qw(usleep);
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node, setting wal-segsize to 1MB
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1,
+ extra => ['--wal-segsize=1'],
+ min_wal_size => '2MB',
+ max_wal_size => '4MB',
+ log_checkpoints => 'yes');
+$node_master->start;
+$node_master->safe_psql('postgres', "SELECT pg_create_physical_replication_slot('rep1')");
+
+# The slot state and remain should be null before the first connection
+my $result = $node_master->safe_psql('postgres', "SELECT restart_lsn is NULL, wal_status is NULL, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "t|t|t", 'check the state of non-reserved slot is "unknown"');
+
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create a standby linking to it using the replication slot
+my $node_standby = get_new_node('standby_1');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1,
+ primary_slot_name => 'rep1');
+
+$node_standby->start;
+
+# Wait until standby has replayed enough data
+my $start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+# Stop standby
+$node_standby->stop;
+
+
+# Preparation done, the slot is the state "normal" now
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check the catching-up state');
+
+# Advance WAL by five segments (= 5MB) on master
+advance_wal($node_master, 1);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when fitting max_wal_size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that restart_lsn is in max_wal_size');
+
+advance_wal($node_master, 4);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when max_slot_wal_keep_size is not set
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that slot is working');
+
+# The standby can reconnect to master
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+# Set max_slot_wal_keep_size on master
+my $max_slot_wal_keep_size_mb = 6;
+$node_master->append_conf('postgresql.conf', qq(
+max_slot_wal_keep_size = ${max_slot_wal_keep_size_mb}MB
+));
+$node_master->reload;
+
+# The slot is in safe state. The remaining bytes should be as almost
+# (max_slot_wal_keep_size + 1) times large as the segment size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that max_slot_wal_keep_size is working');
+
+# Advance WAL again then checkpoint, reducing remain by 2 MB.
+advance_wal($node_master, 2);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is still working
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|5120 kB", 'check that remaining byte is calculated correctly');
+
+# wal_keep_segments overrides max_slot_wal_keep_size
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 8; SELECT pg_reload_conf();");
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that wal_keep_segments overrides max_slot_wal_keep_size');
+
+# restore wal_keep_segments
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 0; SELECT pg_reload_conf();");
+
+# Advance WAL again without checkpoint, reducing remain by 4 MB.
+advance_wal($node_master, 4);
+
+# Slot gets into 'keeping' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|keeping|1024 kB", 'check that the slot state changes to "keeping"');
+
+# do checkpoint so that the next checkpoint runs too early
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# Advance WAL again without checkpoint; remain goes to 0.
+advance_wal($node_master, 1);
+
+# Slot gets into 'lost' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby still can connect to master before a checkpoint
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+ok(!find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed"),
+ 'check that required WAL segments are still available');
+
+# Advance WAL again, the slot loses the oldest segment.
+my $logstart = get_log_size($node_master);
+advance_wal($node_master, 7);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# WARNING should be issued
+ok(find_in_log($node_master,
+ "1 replication slot has lost required WAL segments by 1 segments\n".
+ ".*Most affected slot is rep1.",
+ $logstart),
+ 'check that the warning is logged');
+
+# This slot should be broken
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby no longer can connect to the master
+$logstart = get_log_size($node_standby);
+$node_standby->start;
+
+my $failed = 0;
+for (my $i = 0 ; $i < 10000 ; $i++)
+{
+ if (find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed",
+ $logstart))
+ {
+ $failed = 1;
+ last;
+ }
+ usleep(100_000);
+}
+ok($failed, 'check that replication has been broken');
+
+$node_standby->stop;
+
+#####################################
+# Advance WAL of $node by $n segments
+sub advance_wal
+{
+ my ($node, $n) = @_;
+
+ # Advance by $n segments (= (16 * $n) MB) on master
+ for (my $i = 0 ; $i < $n ; $i++)
+ {
+ $node->safe_psql('postgres', "CREATE TABLE t (); DROP TABLE t; SELECT pg_switch_wal();");
+ }
+}
+
+# return the size of logfile of $node in bytes
+sub get_log_size
+{
+ my ($node) = @_;
+
+ return (stat $node->logfile)[7];
+}
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = TestLib::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 7245b0e13b..8688f7138f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1462,8 +1462,10 @@ pg_replication_slots| SELECT l.slot_name,
l.xmin,
l.catalog_xmin,
l.restart_lsn,
- l.confirmed_flush_lsn
- FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn)
+ l.confirmed_flush_lsn,
+ l.wal_status,
+ l.remain
+ FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, remain)
LEFT JOIN pg_database d ON ((l.datoid = d.oid)));
pg_roles| SELECT pg_authid.rolname,
pg_authid.rolsuper,
--
2.18.2
----Next_Part(Tue_Mar_31_14_20_16_2020_885)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v20-0003-Allow-init-and-init_from_backup-to-set-arbitrary.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v21 1/2] Add WAL relief vent for replication slots
@ 2018-12-19 03:43 Kyotaro Horiguchi <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Kyotaro Horiguchi @ 2018-12-19 03:43 UTC (permalink / raw)
Replication slot is useful to maintain replication connection in the
configurations where replication is so delayed that connection is
broken. On the other hand so many WAL files can fill up disk that the
master downs by a long delay. This feature, which is activated by a
GUC "max_slot_wal_keep_size", protects master servers from suffering
disk full by limiting the number of WAL files reserved by replication
slots.
---
doc/src/sgml/catalogs.sgml | 48 +++
doc/src/sgml/config.sgml | 23 ++
doc/src/sgml/high-availability.sgml | 8 +-
src/backend/access/transam/xlog.c | 322 ++++++++++++++++--
src/backend/catalog/system_views.sql | 4 +-
src/backend/replication/slot.c | 1 +
src/backend/replication/slotfuncs.c | 39 ++-
src/backend/utils/misc/guc.c | 13 +
src/backend/utils/misc/postgresql.conf.sample | 1 +
src/include/access/xlog.h | 19 ++
src/include/catalog/pg_proc.dat | 6 +-
src/test/recovery/t/018_replslot_limit.pl | 203 +++++++++++
src/test/regress/expected/rules.out | 6 +-
13 files changed, 657 insertions(+), 36 deletions(-)
create mode 100644 src/test/recovery/t/018_replslot_limit.pl
diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml
index 64614b569c..01a7802ed4 100644
--- a/doc/src/sgml/catalogs.sgml
+++ b/doc/src/sgml/catalogs.sgml
@@ -9907,6 +9907,54 @@ SELECT * FROM pg_locks pl LEFT JOIN pg_prepared_xacts ppx
</entry>
</row>
+ <row>
+ <entry><structfield>wal_status</structfield></entry>
+ <entry><type>text</type></entry>
+ <entry></entry>
+
+ <entry>Availability of WAL files claimed by this slot.
+ Valid values are:
+ <simplelist>
+ <member>
+ <literal>normal</literal> means that the claimed files
+ are within <varname>max_wal_size</varname>
+ </member>
+ <member>
+ <literal>keeping</literal> means that <varname>max_wal_size</varname>
+ is exceeded but still held by replication slots or
+ <varname>wal_keep_segments</varname>
+ </member>
+ <member>
+ <literal>losing</literal> means that some of the files are on the verge
+ of deletion, but can still be accessed by a session that's currently
+ reading it
+ </member>
+ <member>
+ <literal>lost</literal> means that some of them are definitely lost
+ and the session using this slot cannot continue replication.
+ This state also implies that the session using this slot has been
+ stopped.
+ </member>
+ </simplelist>
+ The last two states are seen only when
+ <xref linkend="guc-max-slot-wal-keep-size"/> is
+ non-negative. If <structfield>restart_lsn</structfield> is NULL, this
+ field is null.
+ </entry>
+ </row>
+
+ <row>
+ <entry><structfield>remain</structfield></entry>
+ <entry><type>bigint</type></entry>
+ <entry></entry>
+ <entry>The amount in bytes of WAL that can be written before this slot
+ loses required WAL files.
+ If <structfield>restart_lsn</structfield> is null or
+ <structfield>wal_status</structfield> is <literal>losing</literal>
+ or <literal>lost</literal>, this field is null.
+ </entry>
+ </row>
+
</tbody>
</tgroup>
</table>
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 2de21903a1..dc99c6868a 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -3758,6 +3758,29 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</listitem>
</varlistentry>
+ <varlistentry id="guc-max-slot-wal-keep-size" xreflabel="max_slot_wal_keep_size">
+ <term><varname>max_slot_wal_keep_size</varname> (<type>integer</type>)
+ <indexterm>
+ <primary><varname>max_slot_wal_keep_size</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Specify the maximum size of WAL files
+ that <link linkend="streaming-replication-slots">replication
+ slots</link> are allowed to retain in the <filename>pg_wal</filename>
+ directory at checkpoint time.
+ If <varname>max_slot_wal_keep_size</varname> is -1 (the default),
+ replication slots retain unlimited amount of WAL files. If
+ restart_lsn of a replication slot gets behind more than that megabytes
+ from the current LSN, the standby using the slot may no longer be able
+ to continue replication due to removal of required WAL files. You
+ can see the WAL availability of replication slots
+ in <link linkend="view-pg-replication-slots">pg_replication_slots</link>.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-wal-sender-timeout" xreflabel="wal_sender_timeout">
<term><varname>wal_sender_timeout</varname> (<type>integer</type>)
<indexterm>
diff --git a/doc/src/sgml/high-availability.sgml b/doc/src/sgml/high-availability.sgml
index b5d32bb720..624e5f94ad 100644
--- a/doc/src/sgml/high-availability.sgml
+++ b/doc/src/sgml/high-availability.sgml
@@ -925,9 +925,11 @@ primary_conninfo = 'host=192.168.1.50 port=5432 user=foo password=foopass'
<xref linkend="guc-archive-command"/>.
However, these methods often result in retaining more WAL segments than
required, whereas replication slots retain only the number of segments
- known to be needed. An advantage of these methods is that they bound
- the space requirement for <literal>pg_wal</literal>; there is currently no way
- to do this using replication slots.
+ known to be needed. On the other hand, replication slots can retain so
+ many WAL segments that they fill up the space allocated
+ for <literal>pg_wal</literal>;
+ <xref linkend="guc-max-slot-wal-keep-size"/> limits the size of WAL files
+ retained by replication slots.
</para>
<para>
Similarly, <xref linkend="guc-hot-standby-feedback"/>
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 977d448f50..db7494e741 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -106,6 +106,7 @@ int wal_level = WAL_LEVEL_MINIMAL;
int CommitDelay = 0; /* precommit delay in microseconds */
int CommitSiblings = 5; /* # concurrent xacts needed to sleep */
int wal_retrieve_retry_interval = 5000;
+int max_slot_wal_keep_size_mb = -1;
#ifdef WAL_DEBUG
bool XLOG_DEBUG = false;
@@ -759,7 +760,7 @@ static ControlFileData *ControlFile = NULL;
*/
#define UsableBytesInPage (XLOG_BLCKSZ - SizeOfXLogShortPHD)
-/* Convert min_wal_size_mb and max_wal_size_mb to equivalent segment count */
+/* Convert values of GUCs measured in megabytes to equiv. segment count */
#define ConvertToXSegs(x, segsize) \
(x / ((segsize) / (1024 * 1024)))
@@ -896,6 +897,7 @@ static void checkTimeLineSwitch(XLogRecPtr lsn, TimeLineID newTLI,
static void LocalSetXLogInsertAllowed(void);
static void CreateEndOfRecoveryRecord(void);
static void CheckPointGuts(XLogRecPtr checkPointRedo, int flags);
+static XLogSegNo GetOldestKeepSegment(XLogRecPtr currpos, XLogRecPtr minSlotPtr);
static void KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo);
static XLogRecPtr XLogGetReplicationSlotMinimumLSN(void);
@@ -3930,8 +3932,21 @@ XLogGetLastRemovedSegno(void)
}
/*
- * Update the last removed segno pointer in shared memory, to reflect
- * that the given XLOG file has been removed.
+ * Return the oldest extant WAL segment file.
+ *
+ * When XLogGetLastRemovedSegno returns a non-zero value, then the oldest
+ * valid segment is one further from that; and if it returns zero, then
+ * it is the segment numbered on.
+ */
+XLogSegNo
+FindOldestXLogFileSegNo(void)
+{
+ return XLogGetLastRemovedSegno() + 1;
+}
+
+/*
+ * Update the last removed segno pointer in shared memory, to reflect that the
+ * given XLOG file has been removed.
*/
static void
UpdateLastRemovedPtr(char *filename)
@@ -9451,6 +9466,197 @@ CreateRestartPoint(int flags)
return true;
}
+/*
+ * Detect availability of the record at given targetLSN.
+ *
+ * targetLSN is restart_lsn of a slot.
+ * walsender_pid is the slot's walsender PID.
+ *
+ * Returns one of the following enum values.
+ *
+ * WALAVAIL_NORMAL means targetLSN is available because it is in the range of
+ * max_wal_size. If max_slot_wal_keep_size is smaller than max_wal_size, this
+ * state is not returned.
+ *
+ * WALAVAIL_PRESERVED means it is still available by preserving extra segments
+ * beyond max_wal_size.
+ *
+ * WALAVAIL_BEING_REMOVED means it is being removed or already removed but the
+ * replication stream on the given slot is live yet. The state may transit to
+ * WALAVAIL_PRESERVED or WALAVAIL_NORMAL state if the walsender advances
+ * restart_lsn.
+ *
+ * WALAVAIL_REMOVED means it is definitely lost. The replication stream on the
+ * slot cannot continue.
+ *
+ * returns WALAVAIL_NULL if restart_lsn is invalid.
+ */
+WalAvailability
+GetWalAvailability(XLogRecPtr restart_lsn, pid_t walsender_pid)
+{
+ XLogRecPtr currpos;
+ XLogRecPtr slotPtr;
+ XLogSegNo currSeg; /* segid of currpos */
+ XLogSegNo restartSeg; /* segid of restart_lsn */
+ XLogSegNo oldestSeg; /* actual oldest segid */
+ XLogSegNo oldestSegMaxWalSize; /* oldest segid kept by max_wal_size */
+ XLogSegNo oldestSlotSeg;/* oldest segid kept by slot */
+ uint64 keepSegs;
+
+ /* the case where the slot has never been activated */
+ if (XLogRecPtrIsInvalid(restart_lsn))
+ return WALAVAIL_INVALID_LSN;
+
+ currpos = GetXLogWriteRecPtr();
+
+ /* calculate oldest segment currently needed by slots */
+ XLByteToSeg(restart_lsn, restartSeg, wal_segment_size);
+ slotPtr = XLogGetReplicationSlotMinimumLSN();
+ oldestSlotSeg = GetOldestKeepSegment(currpos, slotPtr);
+
+ /* find the oldest extant segment file */
+ oldestSeg = FindOldestXLogFileSegNo();
+
+ /* calculate oldest segment by max_wal_size */
+ XLByteToSeg(currpos, currSeg, wal_segment_size);
+ keepSegs = ConvertToXSegs(max_wal_size_mb, wal_segment_size) + 1;
+
+ if (currSeg > keepSegs)
+ oldestSegMaxWalSize = currSeg - keepSegs;
+ else
+ oldestSegMaxWalSize = 1;
+
+ /*
+ * If max_slot_wal_keep_size has changed after the last call, the segment
+ * that would been kept by the current setting might have been lost by the
+ * previous setting. No point in showing normal or keeping status values if
+ * the restartSeg is known to be lost.
+ */
+ if (restartSeg >= oldestSeg)
+ {
+ /*
+ * show "normal" when restartSeg is within max_wal_size. If
+ * max_slot_wal_keep_size is smaller than max_wal_size, there's no
+ * point in showing the status.
+ */
+ if ((max_slot_wal_keep_size_mb <= 0 ||
+ max_slot_wal_keep_size_mb >= max_wal_size_mb) &&
+ oldestSegMaxWalSize <= restartSeg)
+ return WALAVAIL_NORMAL;
+
+ /* being retained by slots */
+ if (oldestSlotSeg <= restartSeg)
+ return WALAVAIL_PRESERVED;
+ }
+
+ /*
+ * The segment is already lost or being lost. If the oldest segment is just
+ * after the restartSeg, running walsender may be reading the just removed
+ * segment. The walsender may safely move to the oldest existing segment in
+ * that case.
+ */
+ if (oldestSeg == restartSeg + 1 && walsender_pid != 0)
+ return WALAVAIL_BEING_REMOVED;
+
+ /* definitely lost. the walsender can no longer restart */
+ return WALAVAIL_REMOVED;
+}
+
+/*
+ * Returns minimum segment number that the next checkpoint must leave
+ * considering wal_keep_segments, replication slots and
+ * max_slot_wal_keep_size.
+ *
+ * currLSN is the current insert location.
+ * minSlotLSN is the minimum restart_lsn of all active slots.
+ */
+static XLogSegNo
+GetOldestKeepSegment(XLogRecPtr currLSN, XLogRecPtr minSlotLSN)
+{
+ XLogSegNo currSeg;
+ XLogSegNo minSlotSeg;
+ uint64 keepSegs = 0; /* # of segments actually kept */
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+ XLByteToSeg(minSlotLSN, minSlotSeg, wal_segment_size);
+
+ /*
+ * Calculate how many segments are kept by slots first. The second
+ * term of the condition is just a sanity check.
+ */
+ if (minSlotLSN != InvalidXLogRecPtr && minSlotSeg <= currSeg)
+ keepSegs = currSeg - minSlotSeg;
+
+ /* Cap keepSegs by max_slot_wal_keep_size */
+ if (max_slot_wal_keep_size_mb >= 0)
+ {
+ uint64 limitSegs;
+
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ /* Reduce it if slots already reserves too many. */
+ if (limitSegs < keepSegs)
+ keepSegs = limitSegs;
+ }
+
+ /* but, keep at least wal_keep_segments segments if any */
+ if (wal_keep_segments > 0 && keepSegs < wal_keep_segments)
+ keepSegs = wal_keep_segments;
+
+ /* avoid underflow, don't go below 1 */
+ if (currSeg <= keepSegs)
+ return 1;
+
+ return currSeg - keepSegs;
+}
+
+/*
+ * Calculate remaining bytes until WAL segment for targetLSN will be removed.
+ */
+int64
+DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN)
+{
+ XLogSegNo currSeg;
+ uint64 limitSegs = 0;
+ int64 restbytes;
+ uint64 fragbytes;
+ XLogSegNo targetSeg;
+
+ XLByteToSeg(currLSN, currSeg, wal_segment_size);
+
+ /* Calculate how far back WAL segments are preserved */
+ if (max_slot_wal_keep_size_mb >= 0)
+ limitSegs = ConvertToXSegs(max_slot_wal_keep_size_mb, wal_segment_size);
+
+ if (wal_keep_segments > 0 && limitSegs < wal_keep_segments)
+ limitSegs = wal_keep_segments;
+
+ XLByteToSeg(targetLSN, targetSeg, wal_segment_size);
+
+ /* avoid underflow */
+ if (targetSeg + limitSegs < currSeg)
+ return 0;
+
+ /*
+ * This slot still has all required segments. Calculate how
+ * many LSN bytes the slot has until it loses targetLSN.
+ */
+ fragbytes = wal_segment_size - (currLSN % wal_segment_size);
+ XLogSegNoOffsetToRecPtr(targetSeg + limitSegs - currSeg,
+ fragbytes, wal_segment_size,
+ restbytes);
+
+ /*
+ * Not realistic, but make sure that it is not out of the
+ * range of int64. No problem to do so since such large values
+ * have no significant difference.
+ */
+ if (restbytes > PG_INT64_MAX)
+ restbytes = PG_INT64_MAX;
+
+ return restbytes;
+}
+
/*
* Retreat *logSegNo to the last segment that we need to retain because of
* either wal_keep_segments or replication slots.
@@ -9462,38 +9668,102 @@ CreateRestartPoint(int flags)
static void
KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
{
- XLogSegNo segno;
- XLogRecPtr keep;
+ static XLogSegNo last_lost_segs = 0;
+ static int last_nslots = 0;
+ static char *last_slot_name = NULL;
+ XLogRecPtr slotminptr = InvalidXLogRecPtr;
+ XLogSegNo minSegNo;
+ XLogSegNo minSlotSegNo;
+ int nslots_affected = 0;
- XLByteToSeg(recptr, segno, wal_segment_size);
- keep = XLogGetReplicationSlotMinimumLSN();
+ if (max_replication_slots > 0)
+ slotminptr = XLogGetReplicationSlotMinimumLSN();
- /* compute limit for wal_keep_segments first */
- if (wal_keep_segments > 0)
+ /*
+ * We should keep certain number of WAL segments after this checkpoint.
+ */
+ minSegNo = GetOldestKeepSegment(recptr, slotminptr);
+
+ /*
+ * Warn the checkpoint is going to flush the segments required by
+ * replication slots.
+ */
+ if (!XLogRecPtrIsInvalid(slotminptr))
{
- /* avoid underflow, don't go below 1 */
- if (segno <= wal_keep_segments)
- segno = 1;
- else
- segno = segno - wal_keep_segments;
+ Assert (max_replication_slots > 0);
+
+ XLByteToSeg(slotminptr, minSlotSegNo, wal_segment_size);
+
+ if (minSlotSegNo < minSegNo)
+ {
+ /* Some slots has lost required segments */
+ XLogSegNo lost_segs = minSegNo - minSlotSegNo;
+ ReplicationSlot *earliest = NULL;
+ char *earliest_name = NULL;
+ int i;
+
+ /* Find the most affected slot */
+ LWLockAcquire(ReplicationSlotControlLock, LW_SHARED);
+ for (i = 0 ; i < max_replication_slots ; i++)
+ {
+ ReplicationSlot *s =
+ &ReplicationSlotCtl->replication_slots[i];
+ XLogSegNo slotSegNo;
+
+ XLByteToSeg(s->data.restart_lsn, slotSegNo, wal_segment_size);
+
+ if (s->in_use && s->active_pid == 0 && slotSegNo < minSegNo)
+ {
+ nslots_affected++;
+
+ if (earliest == NULL ||
+ s->data.restart_lsn < earliest->data.restart_lsn)
+ earliest = s;
+ }
+ }
+
+ if (earliest)
+ {
+ MemoryContext oldcxt = MemoryContextSwitchTo(TopMemoryContext);
+ earliest_name = pstrdup(NameStr(earliest->data.name));
+ MemoryContextSwitchTo(oldcxt);
+ }
+
+ LWLockRelease(ReplicationSlotControlLock);
+
+ /* Emit WARNING if something has changed */
+ if (earliest_name &&
+ (last_lost_segs != lost_segs || last_nslots != nslots_affected))
+ {
+ ereport(WARNING,
+ (errmsg_plural ("%d replication slot has lost required WAL segments by %lu segments",
+ "%d replication slots have lost required WAL segments by %lu segments",
+ nslots_affected, nslots_affected,
+ lost_segs),
+ errdetail("Most affected slot is %s.",
+ earliest_name)));
+
+ if (last_slot_name)
+ pfree(last_slot_name);
+ last_slot_name = earliest_name;
+ last_lost_segs = lost_segs;
+ last_nslots = nslots_affected;
+ }
+ }
}
- /* then check whether slots limit removal further */
- if (max_replication_slots > 0 && keep != InvalidXLogRecPtr)
+ /* Reset the state if no affected slots remain. */
+ if (nslots_affected == 0 && last_slot_name)
{
- XLogSegNo slotSegNo;
-
- XLByteToSeg(keep, slotSegNo, wal_segment_size);
-
- if (slotSegNo <= 0)
- segno = 1;
- else if (slotSegNo < segno)
- segno = slotSegNo;
+ pfree(last_slot_name);
+ last_slot_name = NULL;
+ last_lost_segs = 0;
+ last_nslots = 0;
}
/* don't delete WAL segments newer than the calculated segment */
- if (segno < *logSegNo)
- *logSegNo = segno;
+ if (minSegNo < *logSegNo)
+ *logSegNo = minSegNo;
}
/*
diff --git a/src/backend/catalog/system_views.sql b/src/backend/catalog/system_views.sql
index 83d00c6cde..775b8b7f20 100644
--- a/src/backend/catalog/system_views.sql
+++ b/src/backend/catalog/system_views.sql
@@ -863,7 +863,9 @@ CREATE VIEW pg_replication_slots AS
L.xmin,
L.catalog_xmin,
L.restart_lsn,
- L.confirmed_flush_lsn
+ L.confirmed_flush_lsn,
+ L.wal_status,
+ L.remain
FROM pg_get_replication_slots() AS L
LEFT JOIN pg_database D ON (L.datoid = D.oid);
diff --git a/src/backend/replication/slot.c b/src/backend/replication/slot.c
index d90c7235e9..a26f7999aa 100644
--- a/src/backend/replication/slot.c
+++ b/src/backend/replication/slot.c
@@ -49,6 +49,7 @@
#include "storage/proc.h"
#include "storage/procarray.h"
#include "utils/builtins.h"
+#include "utils/memutils.h"
/*
* Replication slot on-disk data structure.
diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c
index ce0c9127bc..47cd4375a1 100644
--- a/src/backend/replication/slotfuncs.c
+++ b/src/backend/replication/slotfuncs.c
@@ -234,7 +234,7 @@ pg_drop_replication_slot(PG_FUNCTION_ARGS)
Datum
pg_get_replication_slots(PG_FUNCTION_ARGS)
{
-#define PG_GET_REPLICATION_SLOTS_COLS 11
+#define PG_GET_REPLICATION_SLOTS_COLS 13
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
TupleDesc tupdesc;
Tuplestorestate *tupstore;
@@ -288,6 +288,7 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
Oid database;
NameData slot_name;
NameData plugin;
+ WalAvailability walstate;
int i;
if (!slot->in_use)
@@ -355,6 +356,42 @@ pg_get_replication_slots(PG_FUNCTION_ARGS)
else
nulls[i++] = true;
+ walstate = GetWalAvailability(restart_lsn, active_pid);
+
+ switch (walstate)
+ {
+ case WALAVAIL_INVALID_LSN:
+ nulls[i++] = true;
+ break;
+
+ case WALAVAIL_NORMAL:
+ values[i++] = CStringGetTextDatum("normal");
+ break;
+
+ case WALAVAIL_PRESERVED:
+ values[i++] = CStringGetTextDatum("keeping");
+ break;
+
+ case WALAVAIL_BEING_REMOVED:
+ values[i++] = CStringGetTextDatum("losing");
+ break;
+
+ case WALAVAIL_REMOVED:
+ values[i++] = CStringGetTextDatum("lost");
+ break;
+ }
+
+ if (max_slot_wal_keep_size_mb >= 0 &&
+ (walstate == WALAVAIL_NORMAL ||
+ walstate == WALAVAIL_PRESERVED))
+ {
+ values[i++] =
+ Int64GetDatum(DistanceToWalRemoval(GetXLogWriteRecPtr(),
+ restart_lsn));
+ }
+ else
+ nulls[i++] = true;
+
tuplestore_putvalues(tupstore, tupdesc, values, nulls);
}
LWLockRelease(ReplicationSlotControlLock);
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 79bc7ac8ca..a4f0a4e0e3 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -2771,6 +2771,19 @@ static struct config_int ConfigureNamesInt[] =
NULL, NULL, NULL
},
+ {
+ {"max_slot_wal_keep_size", PGC_SIGHUP, REPLICATION_SENDING,
+ gettext_noop("Sets the maximum WAL size that can be reserved by replication slots."),
+ gettext_noop("Replication slots will be marked as failed, and segments released "
+ "for deletion or recycling, if this much space is occupied by WAL "
+ "on disk."),
+ GUC_UNIT_MB
+ },
+ &max_slot_wal_keep_size_mb,
+ -1, -1, MAX_KILOBYTES,
+ NULL, NULL, NULL
+ },
+
{
{"wal_sender_timeout", PGC_USERSET, REPLICATION_SENDING,
gettext_noop("Sets the maximum time to wait for WAL replication."),
diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample
index e9f8ca775d..0b696e7044 100644
--- a/src/backend/utils/misc/postgresql.conf.sample
+++ b/src/backend/utils/misc/postgresql.conf.sample
@@ -287,6 +287,7 @@
#max_wal_senders = 10 # max number of walsender processes
# (change requires restart)
#wal_keep_segments = 0 # in logfile segments; 0 disables
+#max_slot_wal_keep_size = -1 # measured in bytes; -1 disables
#wal_sender_timeout = 60s # in milliseconds; 0 disables
#max_replication_slots = 10 # max number of replication slots
diff --git a/src/include/access/xlog.h b/src/include/access/xlog.h
index 9ec7b31cce..9d29d2263f 100644
--- a/src/include/access/xlog.h
+++ b/src/include/access/xlog.h
@@ -108,6 +108,7 @@ extern int wal_segment_size;
extern int min_wal_size_mb;
extern int max_wal_size_mb;
extern int wal_keep_segments;
+extern int max_slot_wal_keep_size_mb;
extern int XLOGbuffers;
extern int XLogArchiveTimeout;
extern int wal_retrieve_retry_interval;
@@ -255,6 +256,20 @@ typedef struct CheckpointStatsData
extern CheckpointStatsData CheckpointStats;
+/*
+ * WAL segment availability status
+ *
+ * This is used as the return value of GetWalAvailability.
+ */
+typedef enum WalAvailability
+{
+ WALAVAIL_INVALID_LSN, /* parameter error */
+ WALAVAIL_NORMAL, /* WAL segment is within max_wal_size */
+ WALAVAIL_PRESERVED, /* WAL segment is preserved by repslots */
+ WALAVAIL_BEING_REMOVED, /* WAL segment is no longer preserved */
+ WALAVAIL_REMOVED /* WAL segment has been removed */
+} WalAvailability;
+
struct XLogRecData;
extern XLogRecPtr XLogInsertRecord(struct XLogRecData *rdata,
@@ -268,6 +283,7 @@ extern int XLogFileOpen(XLogSegNo segno);
extern void CheckXLogRemoved(XLogSegNo segno, TimeLineID tli);
extern XLogSegNo XLogGetLastRemovedSegno(void);
+extern XLogSegNo FindOldestXLogFileSegNo(void);
extern void XLogSetAsyncXactLSN(XLogRecPtr record);
extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);
@@ -305,6 +321,9 @@ extern void ShutdownXLOG(int code, Datum arg);
extern void InitXLOGAccess(void);
extern void CreateCheckPoint(int flags);
extern bool CreateRestartPoint(int flags);
+extern WalAvailability GetWalAvailability(XLogRecPtr restart_lsn,
+ pid_t walsender_pid);
+extern int64 DistanceToWalRemoval(XLogRecPtr currLSN, XLogRecPtr targetLSN);
extern void XLogPutNextOid(Oid nextOid);
extern XLogRecPtr XLogRestorePoint(const char *rpName);
extern void UpdateFullPageWrites(void);
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index a6a708cca9..2025f34bfd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9971,9 +9971,9 @@
proname => 'pg_get_replication_slots', prorows => '10', proisstrict => 'f',
proretset => 't', provolatile => 's', prorettype => 'record',
proargtypes => '',
- proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn}',
- proargmodes => '{o,o,o,o,o,o,o,o,o,o,o}',
- proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn}',
+ proallargtypes => '{name,name,text,oid,bool,bool,int4,xid,xid,pg_lsn,pg_lsn,text,int8}',
+ proargmodes => '{o,o,o,o,o,o,o,o,o,o,o,o,o}',
+ proargnames => '{slot_name,plugin,slot_type,datoid,temporary,active,active_pid,xmin,catalog_xmin,restart_lsn,confirmed_flush_lsn,wal_status,remain}',
prosrc => 'pg_get_replication_slots' },
{ oid => '3786', descr => 'set up a logical replication slot',
proname => 'pg_create_logical_replication_slot', provolatile => 'v',
diff --git a/src/test/recovery/t/018_replslot_limit.pl b/src/test/recovery/t/018_replslot_limit.pl
new file mode 100644
index 0000000000..39a8780b0f
--- /dev/null
+++ b/src/test/recovery/t/018_replslot_limit.pl
@@ -0,0 +1,203 @@
+# Test for replication slot limit
+# Ensure that max_slot_wal_keep_size limits the number of WAL files to
+# be kept by replication slots.
+
+use strict;
+use warnings;
+use File::Path qw(rmtree);
+use PostgresNode;
+use TestLib;
+use Test::More tests => 13;
+use Time::HiRes qw(usleep);
+
+$ENV{PGDATABASE} = 'postgres';
+
+# Initialize master node, setting wal-segsize to 1MB
+my $node_master = get_new_node('master');
+$node_master->init(allows_streaming => 1, extra => ['--wal-segsize=1']);
+$node_master->append_conf('postgresql.conf', qq(
+min_wal_size = 2MB
+max_wal_size = 4MB
+log_checkpoints = yes
+));
+$node_master->start;
+$node_master->safe_psql('postgres', "SELECT pg_create_physical_replication_slot('rep1')");
+
+# The slot state and remain should be null before the first connection
+my $result = $node_master->safe_psql('postgres', "SELECT restart_lsn is NULL, wal_status is NULL, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "t|t|t", 'check the state of non-reserved slot is "unknown"');
+
+
+# Take backup
+my $backup_name = 'my_backup';
+$node_master->backup($backup_name);
+
+# Create a standby linking to it using the replication slot
+my $node_standby = get_new_node('standby_1');
+$node_standby->init_from_backup($node_master, $backup_name, has_streaming => 1);
+$node_standby->append_conf('postgresql.conf', "primary_slot_name = 'rep1'");
+
+$node_standby->start;
+
+# Wait until standby has replayed enough data
+my $start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+# Stop standby
+$node_standby->stop;
+
+
+# Preparation done, the slot is the state "normal" now
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check the catching-up state');
+
+# Advance WAL by five segments (= 5MB) on master
+advance_wal($node_master, 1);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when fitting max_wal_size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that restart_lsn is in max_wal_size');
+
+advance_wal($node_master, 4);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is always "safe" when max_slot_wal_keep_size is not set
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|t", 'check that slot is working');
+
+# The standby can reconnect to master
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+# Set max_slot_wal_keep_size on master
+my $max_slot_wal_keep_size_mb = 6;
+$node_master->append_conf('postgresql.conf', qq(
+max_slot_wal_keep_size = ${max_slot_wal_keep_size_mb}MB
+));
+$node_master->reload;
+
+# The slot is in safe state. The remaining bytes should be as almost
+# (max_slot_wal_keep_size + 1) times large as the segment size
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that max_slot_wal_keep_size is working');
+
+# Advance WAL again then checkpoint, reducing remain by 2 MB.
+advance_wal($node_master, 2);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# The slot is still working
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|5120 kB", 'check that remaining byte is calculated correctly');
+
+# wal_keep_segments overrides max_slot_wal_keep_size
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 8; SELECT pg_reload_conf();");
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|normal|7168 kB", 'check that wal_keep_segments overrides max_slot_wal_keep_size');
+
+# restore wal_keep_segments
+$result = $node_master->safe_psql('postgres', "ALTER SYSTEM SET wal_keep_segments to 0; SELECT pg_reload_conf();");
+
+# Advance WAL again without checkpoint, reducing remain by 4 MB.
+advance_wal($node_master, 4);
+
+# Slot gets into 'keeping' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, pg_size_pretty(remain) as remain FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|keeping|1024 kB", 'check that the slot state changes to "keeping"');
+
+# do checkpoint so that the next checkpoint runs too early
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# Advance WAL again without checkpoint; remain goes to 0.
+advance_wal($node_master, 1);
+
+# Slot gets into 'lost' state
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby still can connect to master before a checkpoint
+$node_standby->start;
+
+$start_lsn = $node_master->lsn('write');
+$node_master->wait_for_catchup($node_standby, 'replay', $start_lsn);
+
+$node_standby->stop;
+
+ok(!find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed"),
+ 'check that required WAL segments are still available');
+
+# Advance WAL again, the slot loses the oldest segment.
+my $logstart = get_log_size($node_master);
+advance_wal($node_master, 7);
+$node_master->safe_psql('postgres', "CHECKPOINT;");
+
+# WARNING should be issued
+ok(find_in_log($node_master,
+ "1 replication slot has lost required WAL segments by 1 segments\n".
+ ".*Most affected slot is rep1.",
+ $logstart),
+ 'check that the warning is logged');
+
+# This slot should be broken
+$result = $node_master->safe_psql('postgres', "SELECT restart_lsn, wal_status, remain is NULL FROM pg_replication_slots WHERE slot_name = 'rep1'");
+is($result, "$start_lsn|lost|t", 'check that the slot state changes to "lost"');
+
+# The standby no longer can connect to the master
+$logstart = get_log_size($node_standby);
+$node_standby->start;
+
+my $failed = 0;
+for (my $i = 0 ; $i < 10000 ; $i++)
+{
+ if (find_in_log($node_standby,
+ "requested WAL segment [0-9A-F]+ has already been removed",
+ $logstart))
+ {
+ $failed = 1;
+ last;
+ }
+ usleep(100_000);
+}
+ok($failed, 'check that replication has been broken');
+
+$node_standby->stop;
+
+#####################################
+# Advance WAL of $node by $n segments
+sub advance_wal
+{
+ my ($node, $n) = @_;
+
+ # Advance by $n segments (= (16 * $n) MB) on master
+ for (my $i = 0 ; $i < $n ; $i++)
+ {
+ $node->safe_psql('postgres', "CREATE TABLE t (); DROP TABLE t; SELECT pg_switch_wal();");
+ }
+}
+
+# return the size of logfile of $node in bytes
+sub get_log_size
+{
+ my ($node) = @_;
+
+ return (stat $node->logfile)[7];
+}
+
+# find $pat in logfile of $node after $off-th byte
+sub find_in_log
+{
+ my ($node, $pat, $off) = @_;
+
+ $off = 0 unless defined $off;
+ my $log = TestLib::slurp_file($node->logfile);
+ return 0 if (length($log) <= $off);
+
+ $log = substr($log, $off);
+
+ return $log =~ m/$pat/;
+}
diff --git a/src/test/regress/expected/rules.out b/src/test/regress/expected/rules.out
index 7245b0e13b..8688f7138f 100644
--- a/src/test/regress/expected/rules.out
+++ b/src/test/regress/expected/rules.out
@@ -1462,8 +1462,10 @@ pg_replication_slots| SELECT l.slot_name,
l.xmin,
l.catalog_xmin,
l.restart_lsn,
- l.confirmed_flush_lsn
- FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn)
+ l.confirmed_flush_lsn,
+ l.wal_status,
+ l.remain
+ FROM (pg_get_replication_slots() l(slot_name, plugin, slot_type, datoid, temporary, active, active_pid, xmin, catalog_xmin, restart_lsn, confirmed_flush_lsn, wal_status, remain)
LEFT JOIN pg_database d ON ((l.datoid = d.oid)));
pg_roles| SELECT pg_authid.rolname,
pg_authid.rolsuper,
--
2.20.1
--T4sUOijqQbZv57TR
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="v21-0002-Check-removal-of-in-reading-segment-file.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v25 01/15] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
7 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 90b5da51c9..80268ac059 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -1426,6 +1426,7 @@ _copyIntoClause(const IntoClause *from)
COPY_STRING_FIELD(tableSpaceName);
COPY_NODE_FIELD(viewQuery);
COPY_SCALAR_FIELD(skipData);
+ COPY_SCALAR_FIELD(ivm);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 06345da3ba..572560b4a2 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -155,6 +155,7 @@ _equalIntoClause(const IntoClause *a, const IntoClause *b)
COMPARE_STRING_FIELD(tableSpaceName);
COMPARE_NODE_FIELD(viewQuery);
COMPARE_SCALAR_FIELD(skipData);
+ COMPARE_SCALAR_FIELD(ivm);
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 2b0236937a..8ec4059cc0 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1119,6 +1119,7 @@ _outIntoClause(StringInfo str, const IntoClause *node)
WRITE_STRING_FIELD(tableSpaceName);
WRITE_NODE_FIELD(viewQuery);
WRITE_BOOL_FIELD(skipData);
+ WRITE_BOOL_FIELD(ivm);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3f68f7c18d..cc6dcb7220 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -563,6 +563,7 @@ _readIntoClause(void)
READ_STRING_FIELD(tableSpaceName);
READ_NODE_FIELD(viewQuery);
READ_BOOL_FIELD(skipData);
+ READ_BOOL_FIELD(ivm);
READ_DONE();
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index b5966712ce..21beb087c2 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -459,6 +459,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -692,7 +693,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4359,30 +4360,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4399,9 +4402,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15752,6 +15760,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
@@ -16304,6 +16313,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index dab5c4ff5d..bc1bcbda13 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index bcef7eed2f..a43af4a597 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -205,6 +205,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.17.1
--Multipart=_Fri__4_Feb_2022_01_48_06_+0900_N8BNZpfOR27sWrgY--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v27 1/9] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
7 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 836f427ea8..0490bce664 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -1434,6 +1434,7 @@ _copyIntoClause(const IntoClause *from)
COPY_STRING_FIELD(tableSpaceName);
COPY_NODE_FIELD(viewQuery);
COPY_SCALAR_FIELD(skipData);
+ COPY_SCALAR_FIELD(ivm);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index e013c1bbfe..55f41263ee 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -211,6 +211,7 @@ _equalIntoClause(const IntoClause *a, const IntoClause *b)
COMPARE_STRING_FIELD(tableSpaceName);
COMPARE_NODE_FIELD(viewQuery);
COMPARE_SCALAR_FIELD(skipData);
+ COMPARE_SCALAR_FIELD(ivm);
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 6a02f81ad5..cfd3ce68b4 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1127,6 +1127,7 @@ _outIntoClause(StringInfo str, const IntoClause *node)
WRITE_STRING_FIELD(tableSpaceName);
WRITE_NODE_FIELD(viewQuery);
WRITE_BOOL_FIELD(skipData);
+ WRITE_BOOL_FIELD(ivm);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ddf76ac778..5165fb3b93 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -605,6 +605,7 @@ _readIntoClause(void)
READ_STRING_FIELD(tableSpaceName);
READ_NODE_FIELD(viewQuery);
READ_BOOL_FIELD(skipData);
+ READ_BOOL_FIELD(ivm);
READ_DONE();
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c9941d9cb4..aa134eafc9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -468,6 +468,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -801,7 +802,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4492,30 +4493,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4532,9 +4535,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17007,6 +17015,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
@@ -17588,6 +17597,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 66d32fc006..acac38b91c 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -126,6 +126,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 8a2ab405a2..7cd9b54a01 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -210,6 +210,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.17.1
--Multipart=_Fri__22_Apr_2022_14_58_01_+0900_MN3L/o2YUVF2g4zw
Content-Type: text/x-diff;
name="v27-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v27-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v27 1/9] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
7 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 836f427ea8..0490bce664 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -1434,6 +1434,7 @@ _copyIntoClause(const IntoClause *from)
COPY_STRING_FIELD(tableSpaceName);
COPY_NODE_FIELD(viewQuery);
COPY_SCALAR_FIELD(skipData);
+ COPY_SCALAR_FIELD(ivm);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index e013c1bbfe..55f41263ee 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -211,6 +211,7 @@ _equalIntoClause(const IntoClause *a, const IntoClause *b)
COMPARE_STRING_FIELD(tableSpaceName);
COMPARE_NODE_FIELD(viewQuery);
COMPARE_SCALAR_FIELD(skipData);
+ COMPARE_SCALAR_FIELD(ivm);
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 6a02f81ad5..cfd3ce68b4 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1127,6 +1127,7 @@ _outIntoClause(StringInfo str, const IntoClause *node)
WRITE_STRING_FIELD(tableSpaceName);
WRITE_NODE_FIELD(viewQuery);
WRITE_BOOL_FIELD(skipData);
+ WRITE_BOOL_FIELD(ivm);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index ddf76ac778..5165fb3b93 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -605,6 +605,7 @@ _readIntoClause(void)
READ_STRING_FIELD(tableSpaceName);
READ_NODE_FIELD(viewQuery);
READ_BOOL_FIELD(skipData);
+ READ_BOOL_FIELD(ivm);
READ_DONE();
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c9941d9cb4..aa134eafc9 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -468,6 +468,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -801,7 +802,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4492,30 +4493,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4532,9 +4535,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17007,6 +17015,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
@@ -17588,6 +17597,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 66d32fc006..acac38b91c 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -126,6 +126,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 8a2ab405a2..7cd9b54a01 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -210,6 +210,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.17.1
--Multipart=_Fri__22_Apr_2022_11_29_39_+0900_ZOAC7UMt5e8j1Nvx
Content-Type: text/x-diff;
name="v27-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v27-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v32 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 682748eb4b..91df005a19 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -468,6 +468,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -732,7 +733,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4734,32 +4735,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4776,9 +4779,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17454,6 +17462,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -18037,6 +18046,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index aa727e722c..9d0ba29a2a 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -154,6 +154,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 6c959e85d5..94c4a3de1a 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -210,6 +210,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Sun__31_Mar_2024_22_59_31_+0900_msknEviJj08_wgqO
Content-Type: text/x-diff;
name="v32-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v32-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v31 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index c1b0cff1c9..3a4746a22b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -467,6 +467,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -730,7 +731,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4732,32 +4733,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4774,9 +4777,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17436,6 +17444,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -18017,6 +18026,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 376f67e6a5..f7bcf78cf2 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -154,6 +154,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 57514d064b..f146d7c41a 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -210,6 +210,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Fri__29_Mar_2024_23_47_00_+0900_KGpmmDOIs1266Ib1
Content-Type: text/x-diff;
name="v31-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v31-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v30 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 130f7fc7c3..7ba972ae5d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -466,6 +466,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -719,7 +720,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4696,32 +4697,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4738,9 +4741,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17252,6 +17260,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -17820,6 +17829,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 4a154606d2..5d61964f37 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -154,6 +154,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 2331acac09..ab92b54642 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -207,6 +207,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v30-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v23 01/15] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 10da5c5c51..9c2382fed7 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -440,6 +440,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -669,7 +670,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4257,30 +4258,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4297,9 +4300,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15584,6 +15592,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
@@ -16136,6 +16145,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index c04282f91f..a0af94eb49 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f836acf876..2cafb4e7fe 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -205,6 +205,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.17.1
--Multipart=_Mon__2_Aug_2021_15_28_34_+0900_wlHCjIpnD/FrGAKu
Content-Type: text/x-diff;
name="v23-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v23-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v39 1/8] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index bb05aeebee4..da35b15cc27 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v38 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index bb05aeebee4..da35b15cc27 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v37 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 7977ee24783..213adeec2e2 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v37 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 7977ee24783..213adeec2e2 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v24 01/15] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
7 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 228387eaee..17be377aa7 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -1423,6 +1423,7 @@ _copyIntoClause(const IntoClause *from)
COPY_STRING_FIELD(tableSpaceName);
COPY_NODE_FIELD(viewQuery);
COPY_SCALAR_FIELD(skipData);
+ COPY_SCALAR_FIELD(ivm);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index 800f588b5c..ed74a5022c 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -155,6 +155,7 @@ _equalIntoClause(const IntoClause *a, const IntoClause *b)
COMPARE_STRING_FIELD(tableSpaceName);
COMPARE_NODE_FIELD(viewQuery);
COMPARE_SCALAR_FIELD(skipData);
+ COMPARE_SCALAR_FIELD(ivm);
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 2e5ed77e18..f6166f7859 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1116,6 +1116,7 @@ _outIntoClause(StringInfo str, const IntoClause *node)
WRITE_STRING_FIELD(tableSpaceName);
WRITE_NODE_FIELD(viewQuery);
WRITE_BOOL_FIELD(skipData);
+ WRITE_BOOL_FIELD(ivm);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index abf08b7a2f..1e55a58f69 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -565,6 +565,7 @@ _readIntoClause(void)
READ_STRING_FIELD(tableSpaceName);
READ_NODE_FIELD(viewQuery);
READ_BOOL_FIELD(skipData);
+ READ_BOOL_FIELD(ivm);
READ_DONE();
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index e3068a374e..46a812e550 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -439,6 +439,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -668,7 +669,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4264,30 +4265,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4304,9 +4307,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15606,6 +15614,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
@@ -16158,6 +16167,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 433437643e..7ea80c7ded 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f836acf876..2cafb4e7fe 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -205,6 +205,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.17.1
--Multipart=_Thu__23_Sep_2021_04_57_30_+0900_b5pmgR1N8oMaMz.U
Content-Type: text/x-diff;
name="v24-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v24-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v24 01/15] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
7 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index 82464c9889..5769a10586 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -1423,6 +1423,7 @@ _copyIntoClause(const IntoClause *from)
COPY_STRING_FIELD(tableSpaceName);
COPY_NODE_FIELD(viewQuery);
COPY_SCALAR_FIELD(skipData);
+ COPY_SCALAR_FIELD(ivm);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f537d3eb96..80497dc240 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -155,6 +155,7 @@ _equalIntoClause(const IntoClause *a, const IntoClause *b)
COMPARE_STRING_FIELD(tableSpaceName);
COMPARE_NODE_FIELD(viewQuery);
COMPARE_SCALAR_FIELD(skipData);
+ COMPARE_SCALAR_FIELD(ivm);
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 2e5ed77e18..f6166f7859 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1116,6 +1116,7 @@ _outIntoClause(StringInfo str, const IntoClause *node)
WRITE_STRING_FIELD(tableSpaceName);
WRITE_NODE_FIELD(viewQuery);
WRITE_BOOL_FIELD(skipData);
+ WRITE_BOOL_FIELD(ivm);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index abf08b7a2f..1e55a58f69 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -565,6 +565,7 @@ _readIntoClause(void)
READ_STRING_FIELD(tableSpaceName);
READ_NODE_FIELD(viewQuery);
READ_BOOL_FIELD(skipData);
+ READ_BOOL_FIELD(ivm);
READ_DONE();
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index d0eb80e69c..78e17a655b 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -444,6 +444,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -675,7 +676,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4271,30 +4272,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4311,9 +4314,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15664,6 +15672,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
@@ -16216,6 +16225,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 433437643e..7ea80c7ded 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f836acf876..2cafb4e7fe 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -205,6 +205,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.17.1
--Multipart=_Fri__29_Oct_2021_18_16_28_+0900_jlYRKjywLqhZ7oyk
Content-Type: text/x-diff;
name="v24-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v24-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v26 01/10] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/nodes/copyfuncs.c | 1 +
src/backend/nodes/equalfuncs.c | 1 +
src/backend/nodes/outfuncs.c | 1 +
src/backend/nodes/readfuncs.c | 1 +
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
7 files changed, 27 insertions(+), 11 deletions(-)
diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c
index d4f8455a2b..eb28657791 100644
--- a/src/backend/nodes/copyfuncs.c
+++ b/src/backend/nodes/copyfuncs.c
@@ -1426,6 +1426,7 @@ _copyIntoClause(const IntoClause *from)
COPY_STRING_FIELD(tableSpaceName);
COPY_NODE_FIELD(viewQuery);
COPY_SCALAR_FIELD(skipData);
+ COPY_SCALAR_FIELD(ivm);
return newnode;
}
diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c
index f1002afe7a..521a87a8ea 100644
--- a/src/backend/nodes/equalfuncs.c
+++ b/src/backend/nodes/equalfuncs.c
@@ -155,6 +155,7 @@ _equalIntoClause(const IntoClause *a, const IntoClause *b)
COMPARE_STRING_FIELD(tableSpaceName);
COMPARE_NODE_FIELD(viewQuery);
COMPARE_SCALAR_FIELD(skipData);
+ COMPARE_SCALAR_FIELD(ivm);
return true;
}
diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c
index 6bdad462c7..801c41b978 100644
--- a/src/backend/nodes/outfuncs.c
+++ b/src/backend/nodes/outfuncs.c
@@ -1119,6 +1119,7 @@ _outIntoClause(StringInfo str, const IntoClause *node)
WRITE_STRING_FIELD(tableSpaceName);
WRITE_NODE_FIELD(viewQuery);
WRITE_BOOL_FIELD(skipData);
+ WRITE_BOOL_FIELD(ivm);
}
static void
diff --git a/src/backend/nodes/readfuncs.c b/src/backend/nodes/readfuncs.c
index 3f68f7c18d..cc6dcb7220 100644
--- a/src/backend/nodes/readfuncs.c
+++ b/src/backend/nodes/readfuncs.c
@@ -563,6 +563,7 @@ _readIntoClause(void)
READ_STRING_FIELD(tableSpaceName);
READ_NODE_FIELD(viewQuery);
READ_BOOL_FIELD(skipData);
+ READ_BOOL_FIELD(ivm);
READ_DONE();
}
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index a03b33b53b..e80ce50f82 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -459,6 +459,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -693,7 +694,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4368,30 +4369,32 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4408,9 +4411,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -15787,6 +15795,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
@@ -16339,6 +16348,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDEX
| INDEXES
| INHERIT
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index dab5c4ff5d..bc1bcbda13 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -117,6 +117,7 @@ typedef struct IntoClause
char *tableSpaceName; /* table space to use, or NULL */
Node *viewQuery; /* materialized view's SELECT query */
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index bcef7eed2f..a43af4a597 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -205,6 +205,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.17.1
--Multipart=_Mon__14_Mar_2022_19_12_17_+0900_E9HVp8j5QFkIIek.--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v30 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 130f7fc7c3..7ba972ae5d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -466,6 +466,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -719,7 +720,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4696,32 +4697,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4738,9 +4741,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17252,6 +17260,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -17820,6 +17829,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 4a154606d2..5d61964f37 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -154,6 +154,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 2331acac09..ab92b54642 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -207,6 +207,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v30-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v30 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 130f7fc7c3..7ba972ae5d 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -466,6 +466,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -719,7 +720,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4696,32 +4697,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4738,9 +4741,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17252,6 +17260,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -17820,6 +17829,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 4a154606d2..5d61964f37 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -154,6 +154,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 2331acac09..ab92b54642 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -207,6 +207,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Mon__4_Mar_2024_11_58_46_+0900_UaponF/qQhQrVCFt
Content-Type: text/x-diff;
name="v30-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v30-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v28 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 39ab7eac0d..3534b37f81 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -465,6 +465,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -721,7 +722,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4637,32 +4638,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4679,9 +4682,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17055,6 +17063,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -17621,6 +17630,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 792a743f72..e87a105707 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -146,6 +146,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index f5b2e61ca5..a3fbfedbf4 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -207,6 +207,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Thu__1_Jun_2023_23_59_09_+0900_/G5+8nG46.f1T42K
Content-Type: text/x-diff;
name="v28-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v28-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v39 1/8] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index bb05aeebee4..da35b15cc27 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v38 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index bb05aeebee4..da35b15cc27 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v37 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 7977ee24783..213adeec2e2 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Fri__29_May_2026_23_14_17_+0900_Te0o73X2VqYK57Gd--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v29 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 7d2032885e..33c647b0c7 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -465,6 +465,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -718,7 +719,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4652,32 +4653,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4694,9 +4697,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17141,6 +17149,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -17709,6 +17718,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 60d72a876b..cecb968b36 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -146,6 +146,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa4b..d60eb98d65 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -207,6 +207,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Mon__28_Aug_2023_16_05_30_+0900_b1OvQD_3A3ZMTGvj
Content-Type: text/x-diff;
name="v29-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v29-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v29 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 7d2032885e..33c647b0c7 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -465,6 +465,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -718,7 +719,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4652,32 +4653,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4694,9 +4697,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17141,6 +17149,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -17709,6 +17718,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 60d72a876b..cecb968b36 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -146,6 +146,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa4b..d60eb98d65 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -207,6 +207,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v29-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v39 1/8] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index bb05aeebee4..da35b15cc27 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Fri__3_Jul_2026_19_11_16_+0900_CskxSIu_iGcDMEyM--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v38 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index ff4e1388c55..9a353550c37 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -473,6 +473,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -776,7 +777,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P IGNORE_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -5039,32 +5040,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -5081,9 +5084,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -18948,6 +18956,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -19554,6 +19563,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index bb05aeebee4..da35b15cc27 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -170,6 +170,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
struct Query *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 51ead54f015..7849ee7f960 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -216,6 +216,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.43.0
--Multipart=_Wed__1_Jul_2026_00_04_01_+0900_OVSy2WWK_9aByzDJ--
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v29 01/11] Add a syntax to create Incrementally Maintainable Materialized Views
@ 2019-12-20 01:05 Yugo Nagata <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Yugo Nagata @ 2019-12-20 01:05 UTC (permalink / raw)
Allow to create Incrementally Maintainable Materialized View (IMMV)
by using INCREMENTAL option in CREATE MATERIALIZED VIEW command
as follow:
CREATE [INCREMANTAL] MATERIALIZED VIEW xxxxx AS SELECT ....;
---
src/backend/parser/gram.y | 32 +++++++++++++++++++++-----------
src/include/nodes/primnodes.h | 1 +
src/include/parser/kwlist.h | 1 +
3 files changed, 23 insertions(+), 11 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 7d2032885e..33c647b0c7 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -465,6 +465,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> OptTempTableName
%type <into> into_clause create_as_target create_mv_target
+%type <boolean> incremental
%type <defelt> createfunc_opt_item common_func_opt_item dostmt_opt_item
%type <fun_param> func_arg func_arg_with_default table_func_column aggr_arg
@@ -718,7 +719,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INCREMENTAL INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -4652,32 +4653,34 @@ opt_with_data:
*****************************************************************************/
CreateMatViewStmt:
- CREATE OptNoLog MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
+ CREATE OptNoLog incremental MATERIALIZED VIEW create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $7;
- ctas->into = $5;
+ ctas->query = $8;
+ ctas->into = $6;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = false;
/* cram additional flags into the IntoClause */
- $5->rel->relpersistence = $2;
- $5->skipData = !($8);
+ $6->rel->relpersistence = $2;
+ $6->skipData = !($9);
+ $6->ivm = $3;
$$ = (Node *) ctas;
}
- | CREATE OptNoLog MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
+ | CREATE OptNoLog incremental MATERIALIZED VIEW IF_P NOT EXISTS create_mv_target AS SelectStmt opt_with_data
{
CreateTableAsStmt *ctas = makeNode(CreateTableAsStmt);
- ctas->query = $10;
- ctas->into = $8;
+ ctas->query = $11;
+ ctas->into = $9;
ctas->objtype = OBJECT_MATVIEW;
ctas->is_select_into = false;
ctas->if_not_exists = true;
/* cram additional flags into the IntoClause */
- $8->rel->relpersistence = $2;
- $8->skipData = !($11);
+ $9->rel->relpersistence = $2;
+ $9->skipData = !($12);
+ $9->ivm = $3;
$$ = (Node *) ctas;
}
;
@@ -4694,9 +4697,14 @@ create_mv_target:
$$->tableSpaceName = $5;
$$->viewQuery = NULL; /* filled at analysis time */
$$->skipData = false; /* might get changed later */
+ $$->ivm = false;
}
;
+incremental: INCREMENTAL { $$ = true; }
+ | /*EMPTY*/ { $$ = false; }
+ ;
+
OptNoLog: UNLOGGED { $$ = RELPERSISTENCE_UNLOGGED; }
| /*EMPTY*/ { $$ = RELPERSISTENCE_PERMANENT; }
;
@@ -17141,6 +17149,7 @@ unreserved_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
@@ -17709,6 +17718,7 @@ bare_label_keyword:
| INCLUDE
| INCLUDING
| INCREMENT
+ | INCREMENTAL
| INDENT
| INDEX
| INDEXES
diff --git a/src/include/nodes/primnodes.h b/src/include/nodes/primnodes.h
index 60d72a876b..cecb968b36 100644
--- a/src/include/nodes/primnodes.h
+++ b/src/include/nodes/primnodes.h
@@ -146,6 +146,7 @@ typedef struct IntoClause
/* materialized view's SELECT query */
Node *viewQuery pg_node_attr(query_jumble_ignore);
bool skipData; /* true for WITH NO DATA */
+ bool ivm; /* true for WITH IVM */
} IntoClause;
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa4b..d60eb98d65 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -207,6 +207,7 @@ PG_KEYWORD("in", IN_P, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("include", INCLUDE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("including", INCLUDING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("increment", INCREMENT, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("incremental", INCREMENTAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indent", INDENT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
--
2.25.1
--Multipart=_Mon__28_Aug_2023_11_52_52_+0900_hj6L5h176QaSGtg7
Content-Type: text/x-diff;
name="v29-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Disposition: attachment;
filename="v29-0002-Add-relisivm-column-to-pg_class-system-catalog.patch"
Content-Transfer-Encoding: 7bit
^ permalink raw reply [nested|flat] 86+ messages in thread
* ResourceOwner refactoring
@ 2020-11-17 14:21 Heikki Linnakangas <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Heikki Linnakangas @ 2020-11-17 14:21 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Michael Paquier <[email protected]>; Kyotaro Horiguchi <[email protected]>
Two recent patches that I have reviewed have reminded me that the
ResourceOwner interface is a bit clunky. There are two issues:
1. Performance. It's fast enough in most use, but when I was testing
Kyotaro's catcache patches [1], the Resowner functions to track catcache
references nevertheless showed up, accounting for about 20% of the CPU
time used by catcache lookups.
2. It's difficult for extensions to use. There is a callback mechanism
for extensions, but it's much less convenient to use than the built-in
functions. The pgcrypto code uses the callbacks currently, and Michael's
patch [2] would move that support for tracking OpenSSL contexts to the
core, which makes it a lot more convenient for pgcrypto. Wouldn't it be
nice if extensions could have the same ergonomics as built-in code, if
they need to track external resources?
Attached patch refactors the ResourceOwner internals to do that.
The current code in HEAD has a separate array for each "kind" of object
that can be tracked. The number of different kinds of objects has grown
over the years, currently there is support for tracking buffers, files,
catcache, relcache and plancache references, tupledescs, snapshots, DSM
segments and LLVM JIT contexts. And locks, which are a bit special.
In the patch, I'm using the same array to hold all kinds of objects, and
carry another field with each tracked object, to tell what kind of an
object it is. An extension can define a new object kind, by defining
Release and PrintLeakWarning callback functions for it. The code in
resowner.c is now much more generic, as it doesn't need to know about
all the different object kinds anymore (with a couple of exceptions). In
the new scheme, there is one small fixed-size array to hold a few most
recently-added references, that is linear-searched, and older entries
are moved to a hash table.
I haven't done thorough performance testing of this, but with some quick
testing with Kyotaro's "catcachebench" to stress-test syscache lookups,
this performs a little better. In that test, all the activity happens in
the small array portion, but I believe that's true for most use cases.
Thoughts? Can anyone suggest test scenarios to verify the performance of
this?
[1]
https://www.postgresql.org/message-id/20201106.172958.1103727352904717607.horikyota.ntt%40gmail.com
[2] https://www.postgresql.org/message-id/[email protected]
- Heikki
Attachments:
[text/x-patch] v1-resowner-refactor.patch (76.2K, ../../[email protected]/2-v1-resowner-refactor.patch)
download | inline diff:
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 30c30cf3a2e..c099f04f551 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -29,9 +29,21 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
+/* ResourceOwner callbacks to hold tupledesc references */
+static void ResOwnerReleaseTupleDesc(Datum res);
+static void ResOwnerPrintTupleDescLeakWarning(Datum res);
+
+static ResourceOwnerFuncs tupdesc_resowner_funcs =
+{
+ /* relcache references */
+ .name = "tupdesc reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseTupleDesc,
+ .PrintLeakWarning = ResOwnerPrintTupleDescLeakWarning
+};
/*
* CreateTemplateTupleDesc
@@ -376,9 +388,10 @@ IncrTupleDescRefCount(TupleDesc tupdesc)
{
Assert(tupdesc->tdrefcount >= 0);
- ResourceOwnerEnlargeTupleDescs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
tupdesc->tdrefcount++;
- ResourceOwnerRememberTupleDesc(CurrentResourceOwner, tupdesc);
+ ResourceOwnerRemember(CurrentResourceOwner, PointerGetDatum(tupdesc),
+ &tupdesc_resowner_funcs);
}
/*
@@ -394,7 +407,8 @@ DecrTupleDescRefCount(TupleDesc tupdesc)
{
Assert(tupdesc->tdrefcount > 0);
- ResourceOwnerForgetTupleDesc(CurrentResourceOwner, tupdesc);
+ ResourceOwnerForget(CurrentResourceOwner, PointerGetDatum(tupdesc),
+ &tupdesc_resowner_funcs);
if (--tupdesc->tdrefcount == 0)
FreeTupleDesc(tupdesc);
}
@@ -925,3 +939,23 @@ BuildDescFromLists(List *names, List *types, List *typmods, List *collations)
return desc;
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseTupleDesc(Datum res)
+{
+ DecrTupleDescRefCount((TupleDesc) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintTupleDescLeakWarning(Datum res)
+{
+ TupleDesc tupdesc = (TupleDesc) DatumGetPointer(res);
+
+ elog(WARNING,
+ "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
+ tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
+}
diff --git a/src/backend/jit/jit.c b/src/backend/jit/jit.c
index 5ca3f922fed..c44080b9742 100644
--- a/src/backend/jit/jit.c
+++ b/src/backend/jit/jit.c
@@ -26,7 +26,6 @@
#include "jit/jit.h"
#include "miscadmin.h"
#include "utils/fmgrprotos.h"
-#include "utils/resowner_private.h"
/* GUCs */
bool jit_enabled = true;
@@ -140,7 +139,6 @@ jit_release_context(JitContext *context)
if (provider_successfully_loaded)
provider.release_context(context);
- ResourceOwnerForgetJIT(context->resowner, PointerGetDatum(context));
pfree(context);
}
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index 40a439326c6..edea0388c04 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -40,7 +40,7 @@
#include "portability/instr_time.h"
#include "storage/ipc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Handle of a module emitted via ORC JIT */
typedef struct LLVMJitHandle
@@ -121,8 +121,20 @@ static LLVMOrcLLJITRef llvm_create_jit_instance(LLVMTargetMachineRef tm);
static char *llvm_error_message(LLVMErrorRef error);
#endif /* LLVM_VERSION_MAJOR > 11 */
-PG_MODULE_MAGIC;
+/* ResourceOwner callbacks to hold JitContexts */
+static void ResOwnerReleaseJitContext(Datum res);
+static void ResOwnerPrintJitContextLeakWarning(Datum res);
+
+static ResourceOwnerFuncs jit_funcs =
+{
+ /* relcache references */
+ .name = "LLVM JIT context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseJitContext,
+ .PrintLeakWarning = ResOwnerPrintJitContextLeakWarning
+};
+PG_MODULE_MAGIC;
/*
* Initialize LLVM JIT provider.
@@ -151,7 +163,7 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
- ResourceOwnerEnlargeJIT(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
context = MemoryContextAllocZero(TopMemoryContext,
sizeof(LLVMJitContext));
@@ -159,7 +171,7 @@ llvm_create_context(int jitFlags)
/* ensure cleanup */
context->base.resowner = CurrentResourceOwner;
- ResourceOwnerRememberJIT(CurrentResourceOwner, PointerGetDatum(context));
+ ResourceOwnerRemember(CurrentResourceOwner, PointerGetDatum(context), &jit_funcs);
return context;
}
@@ -221,6 +233,8 @@ llvm_release_context(JitContext *context)
pfree(jit_handle);
}
+
+ ResourceOwnerForget(context->resowner, PointerGetDatum(context), &jit_funcs);
}
/*
@@ -1210,3 +1224,21 @@ llvm_error_message(LLVMErrorRef error)
}
#endif /* LLVM_VERSION_MAJOR > 11 */
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseJitContext(Datum res)
+{
+ jit_release_context((JitContext *) PointerGetDatum(res));
+}
+
+static void
+ResOwnerPrintJitContextLeakWarning(Datum res)
+{
+ /* XXX: We used to not print these. Was that intentional? */
+ JitContext *context = (JitContext *) PointerGetDatum(res);
+
+ elog(WARNING, "JIT context leak: context %p still referenced", context);
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index ad0d1a9abc0..11a42cbdea6 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,7 +52,7 @@
#include "utils/memdebug.h"
#include "utils/ps_status.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/timestamp.h"
@@ -198,6 +198,18 @@ static PrivateRefCountEntry *GetPrivateRefCountEntry(Buffer buffer, bool do_move
static inline int32 GetPrivateRefCount(Buffer buffer);
static void ForgetPrivateRefCountEntry(PrivateRefCountEntry *ref);
+/* ResourceOwner callbacks to hold buffer pins */
+static void ResOwnerReleaseBuffer(Datum res);
+static void ResOwnerPrintBufferLeakWarning(Datum res);
+
+ResourceOwnerFuncs buffer_resowner_funcs =
+{
+ .name = "buffer",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseBuffer,
+ .PrintLeakWarning = ResOwnerPrintBufferLeakWarning
+};
+
/*
* Ensure that the PrivateRefCountArray has sufficient space to store one more
* entry. This has to be called before using NewPrivateRefCountEntry() to fill
@@ -727,7 +739,7 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
/* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
isExtend = (blockNum == P_NEW);
@@ -1846,7 +1858,7 @@ BufferSync(int flags)
WritebackContext wb_context;
/* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
@@ -2323,7 +2335,7 @@ BgBufferSync(WritebackContext *wb_context)
*/
/* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
num_to_scan = bufs_to_lap;
num_written = 0;
@@ -3301,7 +3313,7 @@ FlushRelationBuffers(Relation rel)
}
/* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
for (i = 0; i < NBuffers; i++)
{
@@ -3374,7 +3386,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
/* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
for (i = 0; i < NBuffers; i++)
{
@@ -3453,7 +3465,7 @@ FlushDatabaseBuffers(Oid dbid)
BufferDesc *bufHdr;
/* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
for (i = 0; i < NBuffers; i++)
{
@@ -3551,7 +3563,7 @@ void
IncrBufferRefCount(Buffer buffer)
{
Assert(BufferIsPinned(buffer));
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
if (BufferIsLocal(buffer))
LocalRefCount[-buffer - 1]++;
else
@@ -4585,3 +4597,18 @@ TestForOldSnapshot_impl(Snapshot snapshot, Relation relation)
(errcode(ERRCODE_SNAPSHOT_TOO_OLD),
errmsg("snapshot too old")));
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseBuffer(Datum res)
+{
+ ReleaseBuffer(DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintBufferLeakWarning(Datum res)
+{
+ PrintBufferLeakWarning(DatumGetInt32(res));
+}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 6ffd7b33062..cbf9819cba5 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -22,7 +22,7 @@
#include "storage/bufmgr.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/*#define LBDEBUG*/
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 05abcf72d68..ba6262c83da 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -96,7 +96,7 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Define PG_FLUSH_DATA_WORKS if we have an implementation for pg_flush_data */
#if defined(HAVE_SYNC_FILE_RANGE)
@@ -339,6 +339,24 @@ static void unlink_if_exists_fname(const char *fname, bool isdir, int elevel);
static int fsync_parent_path(const char *fname, int elevel);
+/* ResourceOwner callbacks to hold virtual file descriptors */
+static void ResOwnerReleaseFile(Datum res);
+static void ResOwnerPrintFileLeakWarning(Datum res);
+
+static ResourceOwnerFuncs file_resowner_funcs =
+{
+ .name = "File",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseFile,
+ .PrintLeakWarning = ResOwnerPrintFileLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberFile(owner, file) \
+ ResourceOwnerRemember(owner, Int32GetDatum(file), &file_resowner_funcs)
+#define ResourceOwnerForgetFile(owner, file) \
+ ResourceOwnerForget(owner, Int32GetDatum(file), &file_resowner_funcs)
+
/*
* pg_fsync --- do fsync with or without writethrough
*/
@@ -1402,7 +1420,7 @@ ReportTemporaryFileUsage(const char *path, off_t size)
/*
* Called to register a temporary file for automatic close.
- * ResourceOwnerEnlargeFiles(CurrentResourceOwner) must have been called
+ * ResourceOwnerEnlarge(CurrentResourceOwner) must have been called
* before the file was opened.
*/
static void
@@ -1584,7 +1602,7 @@ OpenTemporaryFile(bool interXact)
* open it, if we'll be registering it below.
*/
if (!interXact)
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* If some temp tablespace(s) have been given to us, try to use the next
@@ -1714,7 +1732,7 @@ PathNameCreateTemporaryFile(const char *path, bool error_on_failure)
{
File file;
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Open the file. Note: we don't use O_EXCL, in case there is an orphaned
@@ -1752,7 +1770,7 @@ PathNameOpenTemporaryFile(const char *path, int mode)
{
File file;
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
file = PathNameOpenFile(path, mode | PG_BINARY);
@@ -3604,3 +3622,19 @@ data_sync_elevel(int elevel)
{
return data_sync_retry ? elevel : PANIC;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseFile(Datum res)
+{
+ FileClose((File) DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintFileLeakWarning(Datum res)
+{
+ elog(WARNING, "temporary file leak: File %d still referenced",
+ DatumGetInt32(res));
+}
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index dffbd8e82a2..c665bc6cecf 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -37,13 +37,15 @@
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "storage/dsm.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
+#include "storage/shmem.h"
#include "utils/freepage.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#define PG_DYNSHMEM_CONTROL_MAGIC 0x9a503d32
@@ -139,6 +141,25 @@ static dsm_control_header *dsm_control;
static Size dsm_control_mapped_size = 0;
static void *dsm_control_impl_private = NULL;
+
+/* ResourceOwner callbacks to hold DSM segments */
+static void ResOwnerReleaseDSM(Datum res);
+static void ResOwnerPrintDSMLeakWarning(Datum res);
+
+static ResourceOwnerFuncs dsm_resowner_funcs =
+{
+ .name = "dynamic shared memory segment",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseDSM,
+ .PrintLeakWarning = ResOwnerPrintDSMLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberDSM(owner, seg) \
+ ResourceOwnerRemember(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+#define ResourceOwnerForgetDSM(owner, seg) \
+ ResourceOwnerForget(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+
/*
* Start up the dynamic shared memory system.
*
@@ -895,7 +916,7 @@ void
dsm_unpin_mapping(dsm_segment *seg)
{
Assert(seg->resowner == NULL);
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg->resowner = CurrentResourceOwner;
ResourceOwnerRememberDSM(seg->resowner, seg);
}
@@ -1162,7 +1183,7 @@ dsm_create_descriptor(void)
dsm_segment *seg;
if (CurrentResourceOwner)
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg = MemoryContextAlloc(TopMemoryContext, sizeof(dsm_segment));
dlist_push_head(&dsm_segment_list, &seg->node);
@@ -1241,3 +1262,20 @@ is_main_region_dsm_handle(dsm_handle handle)
{
return handle & 1;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseDSM(Datum res)
+{
+ dsm_detach((dsm_segment *) DatumGetPointer(res));
+}
+static void
+ResOwnerPrintDSMLeakWarning(Datum res)
+{
+ dsm_segment *seg = (dsm_segment *) res;
+
+ elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
+ dsm_segment_handle(seg));
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index d86566f4554..98b4725c406 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -47,7 +47,7 @@
#include "storage/standby.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* This configuration variable is used to set the lock table size */
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 3613ae5f44d..c9ebf9effe1 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -31,12 +31,13 @@
#endif
#include "storage/lmgr.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
@@ -104,6 +105,66 @@ static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
* internal support functions
*/
+/* ResourceOwner callbacks to hold catcache references */
+
+static void ResOwnerReleaseCatCache(Datum res);
+static void ResOwnerPrintCatCacheLeakWarning(Datum res);
+static void ResOwnerReleaseCatCacheList(Datum res);
+static void ResOwnerPrintCatCacheListLeakWarning(Datum res);
+
+static ResourceOwnerFuncs catcache_funcs =
+{
+ /* catcache references */
+ .name = "catcache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCache,
+ .PrintLeakWarning = ResOwnerPrintCatCacheLeakWarning
+};
+
+static ResourceOwnerFuncs catlistref_funcs =
+{
+ /* catcache-list pins */
+ .name = "catcache list reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCacheList,
+ .PrintLeakWarning = ResOwnerPrintCatCacheListLeakWarning
+};
+
+/* support for catcache refcount management */
+static inline void
+ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner)
+{
+ ResourceOwnerEnlarge(owner);
+}
+static inline void
+ResourceOwnerRememberCatCacheRef(ResourceOwner owner, HeapTuple tuple)
+{
+ ResourceOwnerRemember(owner, PointerGetDatum(tuple), &catcache_funcs);
+}
+static inline void
+ResourceOwnerForgetCatCacheRef(ResourceOwner owner, HeapTuple tuple)
+{
+ ResourceOwnerForget(owner, PointerGetDatum(tuple), &catcache_funcs);
+}
+
+static inline void
+ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner)
+{
+ ResourceOwnerEnlarge(owner);
+}
+static inline void
+ResourceOwnerRememberCatCacheListRef(ResourceOwner owner, CatCList *list)
+{
+ ResourceOwnerRemember(owner, PointerGetDatum(list), &catlistref_funcs);
+}
+
+static inline void
+ResourceOwnerForgetCatCacheListRef(ResourceOwner owner, CatCList *list)
+{
+ ResourceOwnerForget(owner, PointerGetDatum(list), &catlistref_funcs);
+}
+
+
/*
* Hash and equality functions for system types that are used as cache key
* fields. In some cases, we just call the regular SQL-callable functions for
@@ -1270,7 +1331,7 @@ SearchCatCacheInternal(CatCache *cache,
*/
if (!ct->negative)
{
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
@@ -1371,7 +1432,7 @@ SearchCatCacheMiss(CatCache *cache,
hashValue, hashIndex,
false);
/* immediately set the refcount to 1 */
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
break; /* assume only one match */
@@ -1583,7 +1644,7 @@ SearchCatCacheList(CatCache *cache,
dlist_move_head(&cache->cc_lists, &cl->cache_elem);
/* Bump the list's refcount and return it */
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
cl->refcount++;
ResourceOwnerRememberCatCacheListRef(CurrentResourceOwner, cl);
@@ -1607,7 +1668,7 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ctlist = NIL;
@@ -2062,14 +2123,19 @@ PrepareToInvalidateCacheTuple(Relation relation,
}
}
-
/*
- * Subroutines for warning about reference leaks. These are exported so
- * that resowner.c can call them.
+ * ResourceOwner callbacks
*/
-void
-PrintCatCacheLeakWarning(HeapTuple tuple)
+static void
+ResOwnerReleaseCatCache(Datum res)
{
+ ReleaseCatCache((HeapTuple) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheLeakWarning(Datum res)
+{
+ HeapTuple tuple = (HeapTuple) DatumGetPointer(res);
CatCTup *ct = (CatCTup *) (((char *) tuple) -
offsetof(CatCTup, tuple));
@@ -2083,9 +2149,17 @@ PrintCatCacheLeakWarning(HeapTuple tuple)
ct->refcount);
}
-void
-PrintCatCacheListLeakWarning(CatCList *list)
+static void
+ResOwnerReleaseCatCacheList(Datum res)
+{
+ ReleaseCatCacheList((CatCList *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheListLeakWarning(Datum res)
{
+ CatCList *list = (CatCList *) DatumGetPointer(res);
+
elog(WARNING, "cache reference leak: cache %s (%d), list %p has count %d",
list->my_cache->cc_relname, list->my_cache->id,
list, list->refcount);
diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c
index 50d6ad28b4c..8bce9d16ed5 100644
--- a/src/backend/utils/cache/plancache.c
+++ b/src/backend/utils/cache/plancache.c
@@ -69,7 +69,7 @@
#include "tcop/utility.h"
#include "utils/inval.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -115,6 +115,31 @@ static void PlanCacheRelCallback(Datum arg, Oid relid);
static void PlanCacheObjectCallback(Datum arg, int cacheid, uint32 hashvalue);
static void PlanCacheSysCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* ResourceOwner callbacks to track plancache references */
+static void ResOwnerReleaseCachedPlan(Datum res);
+static void ResOwnerPrintPlanCacheLeakWarning(Datum res);
+
+/* this is exported for ResourceOwnerReleaseAllPlanCacheRefs() */
+ResourceOwnerFuncs planref_resowner_funcs =
+{
+ .name = "plancache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCachedPlan,
+ .PrintLeakWarning = ResOwnerPrintPlanCacheLeakWarning
+};
+
+static inline void
+ResourceOwnerRememberPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
+{
+ ResourceOwnerRemember(owner, PointerGetDatum(plan), &planref_resowner_funcs);
+}
+static inline void
+ResourceOwnerForgetPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
+{
+ ResourceOwnerForget(owner, PointerGetDatum(plan), &planref_resowner_funcs);
+}
+
+
/* GUC parameter */
int plan_cache_mode;
@@ -1229,7 +1254,7 @@ GetCachedPlan(CachedPlanSource *plansource, ParamListInfo boundParams,
/* Flag the plan as in use by caller */
if (useResOwner)
- ResourceOwnerEnlargePlanCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
plan->refcount++;
if (useResOwner)
ResourceOwnerRememberPlanCacheRef(CurrentResourceOwner, plan);
@@ -1392,7 +1417,7 @@ CachedPlanAllowsSimpleValidityCheck(CachedPlanSource *plansource,
/* Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -1451,7 +1476,7 @@ CachedPlanIsSimplyValid(CachedPlanSource *plansource, CachedPlan *plan,
/* It's still good. Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -2205,3 +2230,20 @@ ResetPlanCache(void)
cexpr->is_valid = false;
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+
+static void
+ResOwnerReleaseCachedPlan(Datum res)
+{
+ ReleaseCachedPlan((CachedPlan *) DatumGetPointer(res), true);
+}
+
+static void
+ResOwnerPrintPlanCacheLeakWarning(Datum res)
+{
+ elog(WARNING, "plancache reference leak: plan %p not closed",
+ DatumGetPointer(res));
+}
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 66393becfb6..efff30e9238 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -78,13 +78,14 @@
#include "storage/smgr.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/relmapper.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -2066,6 +2067,18 @@ RelationIdGetRelation(Oid relationId)
* ----------------------------------------------------------------
*/
+/* ResourceOwner callbacks to track relcache references */
+static void ResOwnerReleaseRelation(Datum res);
+static void ResOwnerPrintRelCacheLeakWarning(Datum res);
+
+static ResourceOwnerFuncs relref_resowner_funcs =
+{
+ .name = "relcache reference",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseRelation,
+ .PrintLeakWarning = ResOwnerPrintRelCacheLeakWarning
+};
+
/*
* RelationIncrementReferenceCount
* Increments relation reference count.
@@ -2077,10 +2090,10 @@ RelationIdGetRelation(Oid relationId)
void
RelationIncrementReferenceCount(Relation rel)
{
- ResourceOwnerEnlargeRelationRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
rel->rd_refcnt += 1;
if (!IsBootstrapProcessingMode())
- ResourceOwnerRememberRelationRef(CurrentResourceOwner, rel);
+ ResourceOwnerRemember(CurrentResourceOwner, PointerGetDatum(rel), &relref_resowner_funcs);
}
/*
@@ -2093,7 +2106,7 @@ RelationDecrementReferenceCount(Relation rel)
Assert(rel->rd_refcnt > 0);
rel->rd_refcnt -= 1;
if (!IsBootstrapProcessingMode())
- ResourceOwnerForgetRelationRef(CurrentResourceOwner, rel);
+ ResourceOwnerForget(CurrentResourceOwner, PointerGetDatum(rel), &relref_resowner_funcs);
}
/*
@@ -6406,3 +6419,21 @@ unlink_initfile(const char *initfilename, int elevel)
initfilename)));
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerPrintRelCacheLeakWarning(Datum res)
+{
+ Relation rel = (Relation) res;
+
+ elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
+ RelationGetRelationName(rel));
+}
+
+static void
+ResOwnerReleaseRelation(Datum res)
+{
+ RelationClose((Relation) res);
+}
diff --git a/src/backend/utils/cache/syscache.c b/src/backend/utils/cache/syscache.c
index 809b27a038f..a3c4063e655 100644
--- a/src/backend/utils/cache/syscache.c
+++ b/src/backend/utils/cache/syscache.c
@@ -982,7 +982,7 @@ static const struct cachedesc cacheinfo[] = {
}
};
-static CatCache *SysCache[SysCacheSize];
+ CatCache *SysCache[SysCacheSize];
static bool CacheInitialized = false;
diff --git a/src/backend/utils/resowner/README b/src/backend/utils/resowner/README
index 2998f6bb362..890db7d1e66 100644
--- a/src/backend/utils/resowner/README
+++ b/src/backend/utils/resowner/README
@@ -60,13 +60,18 @@ subtransaction or portal. Therefore, the "release" operation on a child
ResourceOwner transfers lock ownership to the parent instead of actually
releasing the lock, if isCommit is true.
-Currently, ResourceOwners contain direct support for recording ownership of
-buffer pins, lmgr locks, and catcache, relcache, plancache, tupdesc, and
-snapshot references. Other objects can be associated with a ResourceOwner by
-recording the address of the owning ResourceOwner in such an object. There is
-an API for other modules to get control during ResourceOwner release, so that
-they can scan their own data structures to find the objects that need to be
-deleted.
+ResourceOwner can record ownership of many different kinds of objects.
+As of this writing, it's used internally for buffer pins, lmgr locks, and
+catcache, relcache, plancache, tupdesc, snapshot, DSM and JIT context references.
+ResourceOwner treates all objects the same, but to register a new kind of
+an object with it, you need to fill in a few callback functions, see
+ResourceOwnerFuncs.
+
+There is also an API for other modules to get control during ResourceOwner
+release, so that they can scan their own data structures to find the objects
+that need to be deleted. This used to be the only way to register new kinds
+of objects with a resource owner; nowadays it easier to write custom
+ResourceOwnerFuncs callabacks.
Whenever we are inside a transaction, the global variable
CurrentResourceOwner shows which resource owner should be assigned
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 8bc2c4e9ea3..2e1ae4f8e0c 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -21,73 +21,44 @@
#include "postgres.h"
#include "common/hashfn.h"
-#include "jit/jit.h"
-#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
#include "utils/memutils.h"
-#include "utils/rel.h"
-#include "utils/resowner_private.h"
-#include "utils/snapmgr.h"
-
+#include "utils/plancache.h"
+#include "utils/resowner.h"
/*
- * All resource IDs managed by this code are required to fit into a Datum,
- * which is fine since they are generally pointers or integers.
- *
- * Provide Datum conversion macros for a couple of things that are really
- * just "int".
- */
-#define FileGetDatum(file) Int32GetDatum(file)
-#define DatumGetFile(datum) ((File) DatumGetInt32(datum))
-#define BufferGetDatum(buffer) Int32GetDatum(buffer)
-#define DatumGetBuffer(datum) ((Buffer) DatumGetInt32(datum))
-
-/*
- * ResourceArray is a common structure for storing all types of resource IDs.
- *
- * We manage small sets of resource IDs by keeping them in a simple array:
- * itemsarr[k] holds an ID, for 0 <= k < nitems <= maxitems = capacity.
+ * ResourceElem represents a reference associated with a resource owner.
*
- * If a set grows large, we switch over to using open-addressing hashing.
- * Then, itemsarr[] is a hash table of "capacity" slots, with each
- * slot holding either an ID or "invalidval". nitems is the number of valid
- * items present; if it would exceed maxitems, we enlarge the array and
- * re-hash. In this mode, maxitems should be rather less than capacity so
- * that we don't waste too much time searching for empty slots.
- *
- * In either mode, lastidx remembers the location of the last item inserted
- * or returned by GetAny; this speeds up searches in ResourceArrayRemove.
+ * All objects managed by this code are required to fit into a Datum,
+ * which is fine since they are generally pointers or integers.
*/
-typedef struct ResourceArray
+typedef struct ResourceElem
{
- Datum *itemsarr; /* buffer for storing values */
- Datum invalidval; /* value that is considered invalid */
- uint32 capacity; /* allocated length of itemsarr[] */
- uint32 nitems; /* how many items are stored in items array */
- uint32 maxitems; /* current limit on nitems before enlarging */
- uint32 lastidx; /* index of last item returned by GetAny */
-} ResourceArray;
+ Datum item;
+ ResourceOwnerFuncs *kind;
+} ResourceElem;
/*
- * Initially allocated size of a ResourceArray. Must be power of two since
- * we'll use (arraysize - 1) as mask for hashing.
+ * Size of the small fixed-size array to hold most-recently remembered resources.
*/
-#define RESARRAY_INIT_SIZE 16
+#define RESOWNER_ARRAY_SIZE 8
/*
- * When to switch to hashing vs. simple array logic in a ResourceArray.
+ * Initially allocated size of a ResourceOwner's hash. Must be power of two since
+ * we'll use (capacity - 1) as mask for hashing.
*/
-#define RESARRAY_MAX_ARRAY 64
-#define RESARRAY_IS_ARRAY(resarr) ((resarr)->capacity <= RESARRAY_MAX_ARRAY)
+#define RESOWNER_HASH_INIT_SIZE 32
/*
- * How many items may be stored in a resource array of given capacity.
+ * How many items may be stored in a hash of given capacity.
* When this number is reached, we must resize.
*/
-#define RESARRAY_MAX_ITEMS(capacity) \
- ((capacity) <= RESARRAY_MAX_ARRAY ? (capacity) : (capacity)/4 * 3)
+#define RESOWNER_HASH_MAX_ITEMS(capacity) ((capacity)/4 * 3)
+
+StaticAssertDecl(RESOWNER_HASH_MAX_ITEMS(RESOWNER_HASH_INIT_SIZE) > RESOWNER_ARRAY_SIZE,
+ "initial hash size too small compared to array size");
/*
* To speed up bulk releasing or reassigning locks from a resource owner to
@@ -117,22 +88,33 @@ typedef struct ResourceOwnerData
ResourceOwner nextchild; /* next child of same parent */
const char *name; /* name (just for debugging) */
- /* We have built-in support for remembering: */
- ResourceArray bufferarr; /* owned buffers */
- ResourceArray catrefarr; /* catcache references */
- ResourceArray catlistrefarr; /* catcache-list pins */
- ResourceArray relrefarr; /* relcache references */
- ResourceArray planrefarr; /* plancache references */
- ResourceArray tupdescarr; /* tupdesc references */
- ResourceArray snapshotarr; /* snapshot references */
- ResourceArray filearr; /* open temporary files */
- ResourceArray dsmarr; /* dynamic shmem segments */
- ResourceArray jitarr; /* JIT contexts */
+ /*
+ * These structs keep track of the objects registered with this owner.
+ *
+ * We manage a small set of references by keeping them in a simple
+ * array. When the array gets full, all the elements in the array are
+ * moved to a hash table. This way, the array always contains a few
+ * most recently remembered references. To find a particular reference,
+ * you need to search both the array and the hash table.
+ */
+ ResourceElem arr[RESOWNER_ARRAY_SIZE];
+ uint32 narr; /* how many items are stored in the array */
+
+ /*
+ * The hash table. Uses open-addressing. 'nhash' is the number of items
+ * present; if it would exceed 'grow_at', we enlarge it and re-hash.
+ * 'grow_at' should be rather less than 'capacity' so that we don't waste
+ * too much time searching for empty slots.
+ */
+ ResourceElem *hash;
+ uint32 nhash; /* how many items are stored in the hash */
+ uint32 capacity; /* allocated length of hash[] */
+ uint32 grow_at; /* grow hash when reach this */
/* We can remember up to MAX_RESOWNER_LOCKS references to local locks. */
int nlocks; /* number of owned locks */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
-} ResourceOwnerData;
+} ResourceOwnerData;
/*****************************************************************************
@@ -144,6 +126,18 @@ ResourceOwner CurTransactionResourceOwner = NULL;
ResourceOwner TopTransactionResourceOwner = NULL;
ResourceOwner AuxProcessResourceOwner = NULL;
+/* #define RESOWNER_STATS */
+/* #define RESOWNER_TRACE */
+
+#ifdef RESOWNER_STATS
+static int narray_lookups = 0;
+static int nhash_lookups = 0;
+#endif
+
+#ifdef RESOWNER_TRACE
+static int resowner_trace_counter = 0;
+#endif
+
/*
* List of add-on callbacks for resource releasing
*/
@@ -158,44 +152,36 @@ static ResourceReleaseCallbackItem *ResourceRelease_callbacks = NULL;
/* Internal routines */
-static void ResourceArrayInit(ResourceArray *resarr, Datum invalidval);
-static void ResourceArrayEnlarge(ResourceArray *resarr);
-static void ResourceArrayAdd(ResourceArray *resarr, Datum value);
-static bool ResourceArrayRemove(ResourceArray *resarr, Datum value);
-static bool ResourceArrayGetAny(ResourceArray *resarr, Datum *value);
-static void ResourceArrayFree(ResourceArray *resarr);
static void ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
-static void PrintRelCacheLeakWarning(Relation rel);
-static void PrintPlanCacheLeakWarning(CachedPlan *plan);
-static void PrintTupleDescLeakWarning(TupleDesc tupdesc);
-static void PrintSnapshotLeakWarning(Snapshot snapshot);
-static void PrintFileLeakWarning(File file);
-static void PrintDSMLeakWarning(dsm_segment *seg);
+
+
/*****************************************************************************
* INTERNAL ROUTINES *
*****************************************************************************/
-
-/*
- * Initialize a ResourceArray
- */
static void
-ResourceArrayInit(ResourceArray *resarr, Datum invalidval)
+ResourceArrayAddToHash(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- /* Assert it's empty */
- Assert(resarr->itemsarr == NULL);
- Assert(resarr->capacity == 0);
- Assert(resarr->nitems == 0);
- Assert(resarr->maxitems == 0);
- /* Remember the appropriate "invalid" value */
- resarr->invalidval = invalidval;
- /* We don't allocate any storage until needed */
+ /* Insert into first free slot at or after hash location. */
+ uint32 mask = owner->capacity - 1;
+ uint32 idx;
+
+ idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
+ for (;;)
+ {
+ if (owner->hash[idx].kind == NULL)
+ break;
+ idx = (idx + 1) & mask;
+ }
+ owner->hash[idx].item = value;
+ owner->hash[idx].kind = kind;
+ owner->nhash++;
}
/*
@@ -204,205 +190,222 @@ ResourceArrayInit(ResourceArray *resarr, Datum invalidval)
* This is separate from actually inserting a resource because if we run out
* of memory, it's critical to do so *before* acquiring the resource.
*/
-static void
-ResourceArrayEnlarge(ResourceArray *resarr)
+void
+ResourceOwnerEnlarge(ResourceOwner owner)
{
- uint32 i,
- oldcap,
- newcap;
- Datum *olditemsarr;
- Datum *newitemsarr;
-
- if (resarr->nitems < resarr->maxitems)
+ if (owner->narr < RESOWNER_ARRAY_SIZE)
return; /* no work needed */
- olditemsarr = resarr->itemsarr;
- oldcap = resarr->capacity;
+ /* Is there space in the hash? If not, enlarge it. */
/* Double the capacity of the array (capacity must stay a power of 2!) */
- newcap = (oldcap > 0) ? oldcap * 2 : RESARRAY_INIT_SIZE;
- newitemsarr = (Datum *) MemoryContextAlloc(TopMemoryContext,
- newcap * sizeof(Datum));
- for (i = 0; i < newcap; i++)
- newitemsarr[i] = resarr->invalidval;
-
- /* We assume we can't fail below this point, so OK to scribble on resarr */
- resarr->itemsarr = newitemsarr;
- resarr->capacity = newcap;
- resarr->maxitems = RESARRAY_MAX_ITEMS(newcap);
- resarr->nitems = 0;
-
- if (olditemsarr != NULL)
+ if (owner->narr + owner->nhash >= owner->grow_at)
{
- /*
- * Transfer any pre-existing entries into the new array; they don't
- * necessarily go where they were before, so this simple logic is the
- * best way. Note that if we were managing the set as a simple array,
- * the entries after nitems are garbage, but that shouldn't matter
- * because we won't get here unless nitems was equal to oldcap.
- */
- for (i = 0; i < oldcap; i++)
+ uint32 i,
+ oldcap,
+ newcap;
+ ResourceElem *oldhash;
+ ResourceElem *newhash;
+
+ oldhash = owner->hash;
+ oldcap = owner->capacity;
+
+ newcap = (oldcap > 0) ? oldcap * 2 : RESOWNER_HASH_INIT_SIZE;
+ newhash = (ResourceElem *) MemoryContextAllocZero(TopMemoryContext,
+ newcap * sizeof(ResourceElem));
+
+ /* We assume we can't fail below this point, so OK to scribble on FIXME */
+ owner->hash = newhash;
+ owner->capacity = newcap;
+ owner->grow_at = RESOWNER_HASH_MAX_ITEMS(newcap);
+ owner->nhash = 0;
+
+ if (oldhash != NULL)
{
- if (olditemsarr[i] != resarr->invalidval)
- ResourceArrayAdd(resarr, olditemsarr[i]);
+ /*
+ * Transfer any pre-existing entries into the new array; they don't
+ * necessarily go where they were before, so this simple logic is the
+ * best way.
+ */
+ for (i = 0; i < oldcap; i++)
+ {
+ if (oldhash[i].kind != NULL)
+ ResourceArrayAddToHash(owner, oldhash[i].item, oldhash[i].kind);
+ }
+
+ /* And release old array. */
+ pfree(oldhash);
}
+ }
- /* And release old array. */
- pfree(olditemsarr);
+ /* Move items from the array to the hash */
+ for (int i = 0; i < owner->narr; i++)
+ {
+ ResourceArrayAddToHash(owner, owner->arr[i].item, owner->arr[i].kind);
}
+ owner->narr = 0;
- Assert(resarr->nitems < resarr->maxitems);
+ Assert(owner->nhash < owner->grow_at);
}
/*
- * Add a resource to ResourceArray
+ * Remember that an object is owner by a ReourceOwner
*
- * Caller must have previously done ResourceArrayEnlarge()
+ * Caller must have previously done ResourceOwnerEnlarge()
*/
-static void
-ResourceArrayAdd(ResourceArray *resarr, Datum value)
+void
+ResourceOwnerRemember(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
uint32 idx;
- Assert(value != resarr->invalidval);
- Assert(resarr->nitems < resarr->maxitems);
+#ifdef RESOWNER_TRACE
+ elog(LOG, "REMEMBER %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
- if (RESARRAY_IS_ARRAY(resarr))
- {
- /* Append to linear array. */
- idx = resarr->nitems;
- }
- else
- {
- /* Insert into first free slot at or after hash location. */
- uint32 mask = resarr->capacity - 1;
+ Assert(owner->narr < RESOWNER_ARRAY_SIZE);
- idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (;;)
- {
- if (resarr->itemsarr[idx] == resarr->invalidval)
- break;
- idx = (idx + 1) & mask;
- }
- }
- resarr->lastidx = idx;
- resarr->itemsarr[idx] = value;
- resarr->nitems++;
+ /* Append to linear array. */
+ idx = owner->narr;
+ owner->arr[idx].item = value;
+ owner->arr[idx].kind = kind;
+ owner->narr++;
}
/*
- * Remove a resource from ResourceArray
+ * Forget that an object is owned by a ResourceOwner
*
- * Returns true on success, false if resource was not found.
+ * Returns true on success. If the resource was not found, returns false,
+ * and calls kind->ForgetError callback.
*
- * Note: if same resource ID appears more than once, one instance is removed.
+ * Note: if same resource ID is associated with the ResourceOwner more than once,
+ * one instance is removed.
*/
-static bool
-ResourceArrayRemove(ResourceArray *resarr, Datum value)
+void
+ResourceOwnerForget(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
uint32 i,
- idx,
- lastidx = resarr->lastidx;
+ idx;
- Assert(value != resarr->invalidval);
+#ifdef RESOWNER_TRACE
+ elog(LOG, "FORGET %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
- /* Search through all items, but try lastidx first. */
- if (RESARRAY_IS_ARRAY(resarr))
+ /* Search through all items, but check the array first. */
+ for (i = 0; i < owner->narr; i++)
{
- if (lastidx < resarr->nitems &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
- }
- for (i = 0; i < resarr->nitems; i++)
+ if (owner->arr[i].item == value &&
+ owner->arr[i].kind == kind)
{
- if (resarr->itemsarr[i] == value)
- {
- resarr->itemsarr[i] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
- }
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
+
+#ifdef RESOWNER_STATS
+ narray_lookups++;
+#endif
+
+ return;
}
}
- else
+
+ /* Search hash */
+ if (owner->nhash > 0)
{
- uint32 mask = resarr->capacity - 1;
+ uint32 mask = owner->capacity - 1;
- if (lastidx < resarr->capacity &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->invalidval;
- resarr->nitems--;
- return true;
- }
idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (i = 0; i < resarr->capacity; i++)
+ for (i = 0; i < owner->capacity; i++)
{
- if (resarr->itemsarr[idx] == value)
+ if (owner->hash[idx].item == value &&
+ owner->hash[idx].kind == kind)
{
- resarr->itemsarr[idx] = resarr->invalidval;
- resarr->nitems--;
- return true;
+ owner->hash[idx].item = (Datum) 0;
+ owner->hash[idx].kind = NULL;
+ owner->nhash--;
+
+#ifdef RESOWNER_STATS
+ nhash_lookups++;
+#endif
+ return;
}
idx = (idx + 1) & mask;
}
}
- return false;
+ /*
+ * Use %p to print the reference, since most objects tracked by a resource owner
+ * are pointers. It's a bit misleading if it's not a pointer, but this is a
+ * programmer error, anyway.
+ */
+ elog(ERROR, "%s %p is not owned by resource owner %s",
+ kind->name, DatumGetPointer(value), ResourceOwnerGetName(owner));
}
/*
- * Get any convenient entry in a ResourceArray.
- *
- * "Convenient" is defined as "easy for ResourceArrayRemove to remove";
- * we help that along by setting lastidx to match. This avoids O(N^2) cost
- * when removing all ResourceArray items during ResourceOwner destruction.
- *
- * Returns true if we found an element, or false if the array is empty.
+ * Call the ReleaseResource callback on entries with given 'phase'.
*/
-static bool
-ResourceArrayGetAny(ResourceArray *resarr, Datum *value)
+static void
+ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
+ bool printLeakWarnings)
{
- if (resarr->nitems == 0)
- return false;
+ bool found;
- if (RESARRAY_IS_ARRAY(resarr))
- {
- /* Linear array: just return the first element. */
- resarr->lastidx = 0;
- }
- else
+ /* First handle all the entries in the array. */
+ do
{
- /* Hash: search forward from wherever we were last. */
- uint32 mask = resarr->capacity - 1;
+ found = false;
- for (;;)
+ for (int i = 0; i < owner->narr; i++)
{
- resarr->lastidx &= mask;
- if (resarr->itemsarr[resarr->lastidx] != resarr->invalidval)
- break;
- resarr->lastidx++;
+ if (owner->arr[i].kind->phase == phase)
+ {
+ Datum value = owner->arr[i].item;
+ ResourceOwnerFuncs *kind = owner->arr[i].kind;
+
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+ found = true;
+ }
}
- }
- *value = resarr->itemsarr[resarr->lastidx];
- return true;
-}
+ /*
+ * If any resources were released, check again because some of the
+ * elements might have moved by the callbacks. We don't want to
+ * miss them.
+ */
+ } while (found && owner->narr > 0);
-/*
- * Trash a ResourceArray (we don't care about its state after this)
- */
-static void
-ResourceArrayFree(ResourceArray *resarr)
-{
- if (resarr->itemsarr)
- pfree(resarr->itemsarr);
+ /* Ok, the array has now been handled. Then the hash */
+ do
+ {
+ found = false;
+
+ for (int idx = 0; idx < owner->capacity; idx++)
+ {
+ if (owner->hash[idx].kind != NULL &&
+ owner->hash[idx].kind->phase == phase)
+ {
+ /* found one */
+ Datum value = owner->hash[idx].item;
+ ResourceOwnerFuncs *kind = owner->hash[idx].kind;
+
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+ found = true;
+ }
+ }
+ /*
+ * Like with the array, we must check again after we reach the
+ * end, if any callbacks were called. XXX: We could probably
+ * stipulate that the callbacks may not do certain thing, like
+ * remember more references in the same resource owner, to avoid
+ * that.
+ */
+ }
+ while (found && owner->nhash > 0);
}
@@ -434,16 +437,10 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
parent->firstchild = owner;
}
- ResourceArrayInit(&(owner->bufferarr), BufferGetDatum(InvalidBuffer));
- ResourceArrayInit(&(owner->catrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->catlistrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->relrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->planrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->tupdescarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->snapshotarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->filearr), FileGetDatum(-1));
- ResourceArrayInit(&(owner->dsmarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->jitarr), PointerGetDatum(NULL));
+#ifdef RESOWNER_TRACE
+ elog(LOG, "CREATE %d: %p %s",
+ resowner_trace_counter++, owner, name);
+#endif
return owner;
}
@@ -482,6 +479,15 @@ ResourceOwnerRelease(ResourceOwner owner,
{
/* There's not currently any setup needed before recursing */
ResourceOwnerReleaseInternal(owner, phase, isCommit, isTopLevel);
+
+#ifdef RESOWNER_STATS
+ if (isTopLevel)
+ {
+ elog(LOG, "RESOWNER STATS: lookups: array %d, hash %d", narray_lookups, nhash_lookups);
+ narray_lookups = 0;
+ nhash_lookups = 0;
+ }
+#endif
}
static void
@@ -493,7 +499,6 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceOwner child;
ResourceOwner save;
ResourceReleaseCallbackItem *item;
- Datum foundres;
/* Recurse to handle descendants */
for (child = owner->firstchild; child != NULL; child = child->nextchild)
@@ -509,50 +514,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
if (phase == RESOURCE_RELEASE_BEFORE_LOCKS)
{
/*
- * Release buffer pins. Note that ReleaseBuffer will remove the
- * buffer entry from our array, so we just have to iterate till there
- * are none.
+ * Release all references that need to be released before the locks.
*
- * During a commit, there shouldn't be any remaining pins --- that
+ * During a commit, there shouldn't be any remaining references --- that
* would indicate failure to clean up the executor correctly --- so
* issue warnings. In the abort case, just clean up quietly.
*/
- while (ResourceArrayGetAny(&(owner->bufferarr), &foundres))
- {
- Buffer res = DatumGetBuffer(foundres);
-
- if (isCommit)
- PrintBufferLeakWarning(res);
- ReleaseBuffer(res);
- }
-
- /* Ditto for relcache references */
- while (ResourceArrayGetAny(&(owner->relrefarr), &foundres))
- {
- Relation res = (Relation) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintRelCacheLeakWarning(res);
- RelationClose(res);
- }
-
- /* Ditto for dynamic shared memory segments */
- while (ResourceArrayGetAny(&(owner->dsmarr), &foundres))
- {
- dsm_segment *res = (dsm_segment *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintDSMLeakWarning(res);
- dsm_detach(res);
- }
-
- /* Ditto for JIT contexts */
- while (ResourceArrayGetAny(&(owner->jitarr), &foundres))
- {
- JitContext *context = (JitContext *) PointerGetDatum(foundres);
-
- jit_release_context(context);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -561,7 +529,7 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
/*
* For a top-level xact we are going to release all locks (or at
* least all non-session locks), so just do a single lmgr call at
- * the top of the recursion.
+ * the top of the recursion
*/
if (owner == TopTransactionResourceOwner)
{
@@ -605,70 +573,9 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
else if (phase == RESOURCE_RELEASE_AFTER_LOCKS)
{
/*
- * Release catcache references. Note that ReleaseCatCache will remove
- * the catref entry from our array, so we just have to iterate till
- * there are none.
- *
- * As with buffer pins, warn if any are left at commit time.
+ * Release all references that need to be released after the locks.
*/
- while (ResourceArrayGetAny(&(owner->catrefarr), &foundres))
- {
- HeapTuple res = (HeapTuple) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheLeakWarning(res);
- ReleaseCatCache(res);
- }
-
- /* Ditto for catcache lists */
- while (ResourceArrayGetAny(&(owner->catlistrefarr), &foundres))
- {
- CatCList *res = (CatCList *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheListLeakWarning(res);
- ReleaseCatCacheList(res);
- }
-
- /* Ditto for plancache references */
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
- {
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintPlanCacheLeakWarning(res);
- ReleaseCachedPlan(res, true);
- }
-
- /* Ditto for tupdesc references */
- while (ResourceArrayGetAny(&(owner->tupdescarr), &foundres))
- {
- TupleDesc res = (TupleDesc) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintTupleDescLeakWarning(res);
- DecrTupleDescRefCount(res);
- }
-
- /* Ditto for snapshot references */
- while (ResourceArrayGetAny(&(owner->snapshotarr), &foundres))
- {
- Snapshot res = (Snapshot) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintSnapshotLeakWarning(res);
- UnregisterSnapshot(res);
- }
-
- /* Ditto for temporary files */
- while (ResourceArrayGetAny(&(owner->filearr), &foundres))
- {
- File res = DatumGetFile(foundres);
-
- if (isCommit)
- PrintFileLeakWarning(res);
- FileClose(res);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
/* Let add-on modules get a chance too */
@@ -689,16 +596,42 @@ void
ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner)
{
ResourceOwner save;
- Datum foundres;
save = CurrentResourceOwner;
CurrentResourceOwner = owner;
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
+
+ /* array first */
+ for (int i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->arr[i].item);
+
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
+ i--;
+
+ /* pass 'false' because we already removed the entry from the resowner */
+ ReleaseCachedPlan(planref, false);
+ }
+ }
+
+ /* Then hash */
+ for (int i = 0; i < owner->capacity; i++)
{
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
+ if (owner->hash[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->hash[i].item);
+
+ owner->hash[i].item = (Datum) 0;
+ owner->hash[i].kind = NULL;
+ owner->nhash--;
- ReleaseCachedPlan(res, true);
+ /* pass 'false' because we already removed the entry from the resowner */
+ ReleaseCachedPlan(planref, false);
+ }
}
+
CurrentResourceOwner = save;
}
@@ -715,18 +648,15 @@ ResourceOwnerDelete(ResourceOwner owner)
Assert(owner != CurrentResourceOwner);
/* And it better not own any resources, either */
- Assert(owner->bufferarr.nitems == 0);
- Assert(owner->catrefarr.nitems == 0);
- Assert(owner->catlistrefarr.nitems == 0);
- Assert(owner->relrefarr.nitems == 0);
- Assert(owner->planrefarr.nitems == 0);
- Assert(owner->tupdescarr.nitems == 0);
- Assert(owner->snapshotarr.nitems == 0);
- Assert(owner->filearr.nitems == 0);
- Assert(owner->dsmarr.nitems == 0);
- Assert(owner->jitarr.nitems == 0);
+ Assert(owner->narr == 0);
+ Assert(owner->nhash == 0);
Assert(owner->nlocks == 0 || owner->nlocks == MAX_RESOWNER_LOCKS + 1);
+#ifdef RESOWNER_TRACE
+ elog(LOG, "DELETE %d: %p %s",
+ resowner_trace_counter++, owner, owner->name);
+#endif
+
/*
* Delete children. The recursive call will delink the child from me, so
* just iterate as long as there is a child.
@@ -742,17 +672,8 @@ ResourceOwnerDelete(ResourceOwner owner)
ResourceOwnerNewParent(owner, NULL);
/* And free the object. */
- ResourceArrayFree(&(owner->bufferarr));
- ResourceArrayFree(&(owner->catrefarr));
- ResourceArrayFree(&(owner->catlistrefarr));
- ResourceArrayFree(&(owner->relrefarr));
- ResourceArrayFree(&(owner->planrefarr));
- ResourceArrayFree(&(owner->tupdescarr));
- ResourceArrayFree(&(owner->snapshotarr));
- ResourceArrayFree(&(owner->filearr));
- ResourceArrayFree(&(owner->dsmarr));
- ResourceArrayFree(&(owner->jitarr));
-
+ if (owner->hash)
+ pfree(owner->hash);
pfree(owner);
}
@@ -807,6 +728,15 @@ ResourceOwnerNewParent(ResourceOwner owner,
}
}
+/*
+ * Get the name of a resource owner, for debugging purposes
+ */
+const char *
+ResourceOwnerGetName(ResourceOwner owner)
+{
+ return owner->name;
+}
+
/*
* Register or deregister callback functions for resource cleanup
*
@@ -905,44 +835,6 @@ ReleaseAuxProcessResourcesCallback(int code, Datum arg)
ReleaseAuxProcessResources(isCommit);
}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * buffer array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeBuffers(ResourceOwner owner)
-{
- /* We used to allow pinning buffers without a resowner, but no more */
- Assert(owner != NULL);
- ResourceArrayEnlarge(&(owner->bufferarr));
-}
-
-/*
- * Remember that a buffer pin is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeBuffers()
- */
-void
-ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer)
-{
- ResourceArrayAdd(&(owner->bufferarr), BufferGetDatum(buffer));
-}
-
-/*
- * Forget that a buffer pin is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer)
-{
- if (!ResourceArrayRemove(&(owner->bufferarr), BufferGetDatum(buffer)))
- elog(ERROR, "buffer %d is not owned by resource owner %s",
- buffer, owner->name);
-}
-
/*
* Remember that a Local Lock is owned by a ResourceOwner
*
@@ -994,379 +886,3 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catrefarr));
-}
-
-/*
- * Remember that a catcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheRefs()
- */
-void
-ResourceOwnerRememberCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- ResourceArrayAdd(&(owner->catrefarr), PointerGetDatum(tuple));
-}
-
-/*
- * Forget that a catcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- if (!ResourceArrayRemove(&(owner->catrefarr), PointerGetDatum(tuple)))
- elog(ERROR, "catcache reference %p is not owned by resource owner %s",
- tuple, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache-list reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catlistrefarr));
-}
-
-/*
- * Remember that a catcache-list reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheListRefs()
- */
-void
-ResourceOwnerRememberCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- ResourceArrayAdd(&(owner->catlistrefarr), PointerGetDatum(list));
-}
-
-/*
- * Forget that a catcache-list reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- if (!ResourceArrayRemove(&(owner->catlistrefarr), PointerGetDatum(list)))
- elog(ERROR, "catcache list reference %p is not owned by resource owner %s",
- list, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * relcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeRelationRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->relrefarr));
-}
-
-/*
- * Remember that a relcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeRelationRefs()
- */
-void
-ResourceOwnerRememberRelationRef(ResourceOwner owner, Relation rel)
-{
- ResourceArrayAdd(&(owner->relrefarr), PointerGetDatum(rel));
-}
-
-/*
- * Forget that a relcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetRelationRef(ResourceOwner owner, Relation rel)
-{
- if (!ResourceArrayRemove(&(owner->relrefarr), PointerGetDatum(rel)))
- elog(ERROR, "relcache reference %s is not owned by resource owner %s",
- RelationGetRelationName(rel), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintRelCacheLeakWarning(Relation rel)
-{
- elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
- RelationGetRelationName(rel));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * plancache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->planrefarr));
-}
-
-/*
- * Remember that a plancache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargePlanCacheRefs()
- */
-void
-ResourceOwnerRememberPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- ResourceArrayAdd(&(owner->planrefarr), PointerGetDatum(plan));
-}
-
-/*
- * Forget that a plancache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- if (!ResourceArrayRemove(&(owner->planrefarr), PointerGetDatum(plan)))
- elog(ERROR, "plancache reference %p is not owned by resource owner %s",
- plan, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintPlanCacheLeakWarning(CachedPlan *plan)
-{
- elog(WARNING, "plancache reference leak: plan %p not closed", plan);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * tupdesc reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeTupleDescs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->tupdescarr));
-}
-
-/*
- * Remember that a tupdesc reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeTupleDescs()
- */
-void
-ResourceOwnerRememberTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- ResourceArrayAdd(&(owner->tupdescarr), PointerGetDatum(tupdesc));
-}
-
-/*
- * Forget that a tupdesc reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- if (!ResourceArrayRemove(&(owner->tupdescarr), PointerGetDatum(tupdesc)))
- elog(ERROR, "tupdesc reference %p is not owned by resource owner %s",
- tupdesc, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintTupleDescLeakWarning(TupleDesc tupdesc)
-{
- elog(WARNING,
- "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
- tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * snapshot reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeSnapshots(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->snapshotarr));
-}
-
-/*
- * Remember that a snapshot reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeSnapshots()
- */
-void
-ResourceOwnerRememberSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- ResourceArrayAdd(&(owner->snapshotarr), PointerGetDatum(snapshot));
-}
-
-/*
- * Forget that a snapshot reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- if (!ResourceArrayRemove(&(owner->snapshotarr), PointerGetDatum(snapshot)))
- elog(ERROR, "snapshot reference %p is not owned by resource owner %s",
- snapshot, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintSnapshotLeakWarning(Snapshot snapshot)
-{
- elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
- snapshot);
-}
-
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * files reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeFiles(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->filearr));
-}
-
-/*
- * Remember that a temporary file is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeFiles()
- */
-void
-ResourceOwnerRememberFile(ResourceOwner owner, File file)
-{
- ResourceArrayAdd(&(owner->filearr), FileGetDatum(file));
-}
-
-/*
- * Forget that a temporary file is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetFile(ResourceOwner owner, File file)
-{
- if (!ResourceArrayRemove(&(owner->filearr), FileGetDatum(file)))
- elog(ERROR, "temporary file %d is not owned by resource owner %s",
- file, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintFileLeakWarning(File file)
-{
- elog(WARNING, "temporary file leak: File %d still referenced",
- file);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * dynamic shmem segment reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeDSMs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->dsmarr));
-}
-
-/*
- * Remember that a dynamic shmem segment is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeDSMs()
- */
-void
-ResourceOwnerRememberDSM(ResourceOwner owner, dsm_segment *seg)
-{
- ResourceArrayAdd(&(owner->dsmarr), PointerGetDatum(seg));
-}
-
-/*
- * Forget that a dynamic shmem segment is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetDSM(ResourceOwner owner, dsm_segment *seg)
-{
- if (!ResourceArrayRemove(&(owner->dsmarr), PointerGetDatum(seg)))
- elog(ERROR, "dynamic shared memory segment %u is not owned by resource owner %s",
- dsm_segment_handle(seg), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintDSMLeakWarning(dsm_segment *seg)
-{
- elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
- dsm_segment_handle(seg));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * JIT context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeJIT(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->jitarr));
-}
-
-/*
- * Remember that a JIT context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeJIT()
- */
-void
-ResourceOwnerRememberJIT(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->jitarr), handle);
-}
-
-/*
- * Forget that a JIT context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetJIT(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->jitarr), handle))
- elog(ERROR, "JIT context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 8c41483e87c..b3ea3ce7650 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -66,7 +66,7 @@
#include "utils/memutils.h"
#include "utils/old_snapshot.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
#include "utils/timestamp.h"
@@ -174,6 +174,18 @@ static Snapshot CopySnapshot(Snapshot snapshot);
static void FreeSnapshot(Snapshot snapshot);
static void SnapshotResetXmin(void);
+/* ResourceOwner callbacks to track snapshot references */
+static void ResOwnerReleaseSnapshot(Datum res);
+static void ResOwnerPrintSnapshotLeakWarning(Datum res);
+
+static ResourceOwnerFuncs snapshot_resowner_funcs =
+{
+ .name = "snapshot reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseSnapshot,
+ .PrintLeakWarning = ResOwnerPrintSnapshotLeakWarning
+};
+
/*
* Snapshot fields to be serialized.
*
@@ -831,9 +843,10 @@ RegisterSnapshotOnOwner(Snapshot snapshot, ResourceOwner owner)
snap = snapshot->copied ? snapshot : CopySnapshot(snapshot);
/* and tell resowner.c about it */
- ResourceOwnerEnlargeSnapshots(owner);
+ ResourceOwnerEnlarge(owner);
snap->regd_count++;
- ResourceOwnerRememberSnapshot(owner, snap);
+ ResourceOwnerRemember(owner, PointerGetDatum(snap),
+ &snapshot_resowner_funcs);
if (snap->regd_count == 1)
pairingheap_add(&RegisteredSnapshots, &snap->ph_node);
@@ -870,7 +883,8 @@ UnregisterSnapshotFromOwner(Snapshot snapshot, ResourceOwner owner)
Assert(snapshot->regd_count > 0);
Assert(!pairingheap_is_empty(&RegisteredSnapshots));
- ResourceOwnerForgetSnapshot(owner, snapshot);
+ ResourceOwnerForget(owner, PointerGetDatum(snapshot),
+ &snapshot_resowner_funcs);
snapshot->regd_count--;
if (snapshot->regd_count == 0)
@@ -2345,3 +2359,19 @@ XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot)
return false;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseSnapshot(Datum res)
+{
+ UnregisterSnapshot((Snapshot) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintSnapshotLeakWarning(Datum res)
+{
+ elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
+ DatumGetPointer(res));
+}
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 3377fa56768..c4aa7d7e31e 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -296,6 +296,21 @@ typedef struct CkptSortItem
extern CkptSortItem *CkptBufferIds;
+/* ResourceOwner callbacks to hold buffer pins */
+extern ResourceOwnerFuncs buffer_resowner_funcs;
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+static inline void
+ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer)
+{
+ ResourceOwnerRemember(owner, Int32GetDatum(buffer), &buffer_resowner_funcs);
+}
+static inline void
+ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer)
+{
+ ResourceOwnerForget(owner, Int32GetDatum(buffer), &buffer_resowner_funcs);
+}
+
/*
* Internal buffer management routines
*/
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index f4aa316604e..dbb10dfdd16 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -225,7 +225,4 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
HeapTuple newtuple,
void (*function) (int, uint32, Oid));
-extern void PrintCatCacheLeakWarning(HeapTuple tuple);
-extern void PrintCatCacheListLeakWarning(CatCList *list);
-
#endif /* CATCACHE_H */
diff --git a/src/include/utils/plancache.h b/src/include/utils/plancache.h
index 4901568553c..2e0d5006719 100644
--- a/src/include/utils/plancache.h
+++ b/src/include/utils/plancache.h
@@ -233,4 +233,6 @@ extern bool CachedPlanIsSimplyValid(CachedPlanSource *plansource,
extern CachedExpression *GetCachedExpression(Node *expr);
extern void FreeCachedExpression(CachedExpression *cexpr);
+extern ResourceOwnerFuncs planref_resowner_funcs;
+
#endif /* PLANCACHE_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index 878f39ccf14..87a2bf0558e 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -50,6 +50,32 @@ typedef enum
RESOURCE_RELEASE_AFTER_LOCKS
} ResourceReleasePhase;
+/*
+ * In order to track an object, resowner.c needs a few callbacks for it.
+ * The callbacks for an object of a specific kind are encapsulated in
+ * ResourceOwnerFuncs.
+ */
+typedef struct ResourceOwnerFuncs
+{
+ const char *name; /* name for the object kind, for debugging */
+ ResourceReleasePhase phase; /* when are these objects released? */
+
+ /*
+ * Release resource.
+ *
+ * NOTE: this must call ResourceOwnerForget to disassociate it with the
+ * resource owner.
+ */
+ void (*ReleaseResource)(Datum res);
+
+ /*
+ * Print a warning, when a resource has not been properly released before
+ * commit.
+ */
+ void (*PrintLeakWarning)(Datum res);
+
+} ResourceOwnerFuncs;
+
/*
* Dynamically loaded modules can get control during ResourceOwnerRelease
* by providing a callback of this form.
@@ -71,16 +97,30 @@ extern void ResourceOwnerRelease(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
-extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
extern void ResourceOwnerDelete(ResourceOwner owner);
extern ResourceOwner ResourceOwnerGetParent(ResourceOwner owner);
extern void ResourceOwnerNewParent(ResourceOwner owner,
ResourceOwner newparent);
+extern const char *ResourceOwnerGetName(ResourceOwner owner);
+
+extern void ResourceOwnerEnlarge(ResourceOwner owner);
+extern void ResourceOwnerRemember(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+extern void ResourceOwnerForget(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+
extern void RegisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
extern void UnregisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
+
extern void CreateAuxProcessResourceOwner(void);
extern void ReleaseAuxProcessResources(bool isCommit);
+/* special support for local lock management */
+struct LOCALLOCK;
+extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+
+/* special function to relase all plancache references */
+extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
+
#endif /* RESOWNER_H */
diff --git a/src/include/utils/resowner_private.h b/src/include/utils/resowner_private.h
deleted file mode 100644
index a781a7a2aa6..00000000000
--- a/src/include/utils/resowner_private.h
+++ /dev/null
@@ -1,98 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * resowner_private.h
- * POSTGRES resource owner private definitions.
- *
- * See utils/resowner/README for more info.
- *
- *
- * Portions Copyright (c) 1996-2020, PostgreSQL Global Development Group
- * Portions Copyright (c) 1994, Regents of the University of California
- *
- * src/include/utils/resowner_private.h
- *
- *-------------------------------------------------------------------------
- */
-#ifndef RESOWNER_PRIVATE_H
-#define RESOWNER_PRIVATE_H
-
-#include "storage/dsm.h"
-#include "storage/fd.h"
-#include "storage/lock.h"
-#include "utils/catcache.h"
-#include "utils/plancache.h"
-#include "utils/resowner.h"
-#include "utils/snapshot.h"
-
-
-/* support for buffer refcount management */
-extern void ResourceOwnerEnlargeBuffers(ResourceOwner owner);
-extern void ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer);
-extern void ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer);
-
-/* support for local lock management */
-extern void ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock);
-extern void ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock);
-
-/* support for catcache refcount management */
-extern void ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerForgetCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-extern void ResourceOwnerForgetCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-
-/* support for relcache refcount management */
-extern void ResourceOwnerEnlargeRelationRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberRelationRef(ResourceOwner owner,
- Relation rel);
-extern void ResourceOwnerForgetRelationRef(ResourceOwner owner,
- Relation rel);
-
-/* support for plancache refcount management */
-extern void ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-extern void ResourceOwnerForgetPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-
-/* support for tupledesc refcount management */
-extern void ResourceOwnerEnlargeTupleDescs(ResourceOwner owner);
-extern void ResourceOwnerRememberTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-extern void ResourceOwnerForgetTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-
-/* support for snapshot refcount management */
-extern void ResourceOwnerEnlargeSnapshots(ResourceOwner owner);
-extern void ResourceOwnerRememberSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-extern void ResourceOwnerForgetSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-
-/* support for temporary file management */
-extern void ResourceOwnerEnlargeFiles(ResourceOwner owner);
-extern void ResourceOwnerRememberFile(ResourceOwner owner,
- File file);
-extern void ResourceOwnerForgetFile(ResourceOwner owner,
- File file);
-
-/* support for dynamic shared memory management */
-extern void ResourceOwnerEnlargeDSMs(ResourceOwner owner);
-extern void ResourceOwnerRememberDSM(ResourceOwner owner,
- dsm_segment *);
-extern void ResourceOwnerForgetDSM(ResourceOwner owner,
- dsm_segment *);
-
-/* support for JITContext management */
-extern void ResourceOwnerEnlargeJIT(ResourceOwner owner);
-extern void ResourceOwnerRememberJIT(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetJIT(ResourceOwner owner,
- Datum handle);
-
-#endif /* RESOWNER_PRIVATE_H */
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-11-18 08:06 Michael Paquier <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2020-11-18 08:06 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers; Kyotaro Horiguchi <[email protected]>; Andres Freund <[email protected]>
On Tue, Nov 17, 2020 at 04:21:29PM +0200, Heikki Linnakangas wrote:
> 2. It's difficult for extensions to use. There is a callback mechanism for
> extensions, but it's much less convenient to use than the built-in
> functions. The pgcrypto code uses the callbacks currently, and Michael's
> patch [2] would move that support for tracking OpenSSL contexts to the core,
> which makes it a lot more convenient for pgcrypto. Wouldn't it be nice if
> extensions could have the same ergonomics as built-in code, if they need to
> track external resources?
+1. True that the current interface is a bit hard to grasp, one can
easily miss that showing reference leaks is very important if an
allocation happens out of the in-core palloc machinery at commit time.
(The patch you are referring to is not really popular, but that does
not prevent this thread to move on on its own.)
> Attached patch refactors the ResourceOwner internals to do that.
+ * Size of the small fixed-size array to hold most-recently remembered resources.
*/
-#define RESARRAY_INIT_SIZE 16
+#define RESOWNER_ARRAY_SIZE 8
Why did you choose this size for the initial array?
+extern const char *ResourceOwnerGetName(ResourceOwner owner);
This is only used in resowner.c, at one place.
@@ -140,7 +139,6 @@ jit_release_context(JitContext *context)
if (provider_successfully_loaded)
provider.release_context(context);
- ResourceOwnerForgetJIT(context->resowner, PointerGetDatum(context));
[...]
+ ResourceOwnerForget(context->resowner, PointerGetDatum(context), &jit_funcs);
This moves the JIT context release from jit.c to llvm.c. I think
that's indeed more consistent, and correct. It would be better to
check this one with Andres, though.
> I haven't done thorough performance testing of this, but with some quick
> testing with Kyotaro's "catcachebench" to stress-test syscache lookups, this
> performs a little better. In that test, all the activity happens in the
> small array portion, but I believe that's true for most use cases.
> Thoughts? Can anyone suggest test scenarios to verify the performance of
> this?
Playing with catcache.c is the first thing that came to my mind.
After that some micro-benchmarking with DSM or snapshots? I am not
sure if we would notice a difference in a real-life scenario, say even
a pgbench -S with prepared queries.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-11-18 08:50 Heikki Linnakangas <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2020-11-18 08:50 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: pgsql-hackers; Kyotaro Horiguchi <[email protected]>; Andres Freund <[email protected]>
On 18/11/2020 10:06, Michael Paquier wrote:
> On Tue, Nov 17, 2020 at 04:21:29PM +0200, Heikki Linnakangas wrote:
>> Attached patch refactors the ResourceOwner internals to do that.
>
> + * Size of the small fixed-size array to hold most-recently remembered resources.
> */
> -#define RESARRAY_INIT_SIZE 16
> +#define RESOWNER_ARRAY_SIZE 8
> Why did you choose this size for the initial array?
Just a guess. The old init size was 16 Datums. The entries in the new
array are twice as large, Datum+pointer.
The "RESOWNER_STATS" #ifdef blocks can be enabled to check how many
lookups fit in the array. With pgbench, RESOWNER_ARRAY_SIZE 8:
RESOWNER STATS: lookups: array 235, hash 32
If RESOWNER_ARRAY_STATS is increased to 16, all the lookups fit in the
array. But I haven't done any benchmarking to see which is faster.
BTW, I think there would be an easy win in the hashing codepath, by
changing to a cheaper hash function. Currently, with or without this
patch, we use hash_any(). Changing that to murmurhash32() or something
similar would be a drop-in replacement.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-11-19 02:16 Michael Paquier <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2020-11-19 02:16 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers; Kyotaro Horiguchi <[email protected]>; Andres Freund <[email protected]>
On Wed, Nov 18, 2020 at 10:50:08AM +0200, Heikki Linnakangas wrote:
> If RESOWNER_ARRAY_STATS is increased to 16, all the lookups fit in the
> array. But I haven't done any benchmarking to see which is faster.
My gut tells me that your guess is right, but it would be better to be
sure.
> BTW, I think there would be an easy win in the hashing codepath, by changing
> to a cheaper hash function. Currently, with or without this patch, we use
> hash_any(). Changing that to murmurhash32() or something similar would be a
> drop-in replacement.
Good idea.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-11-19 09:27 Julien Rouhaud <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Julien Rouhaud @ 2020-11-19 09:27 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; pgsql-hackers; Kyotaro Horiguchi <[email protected]>; Andres Freund <[email protected]>
On Thu, Nov 19, 2020 at 10:16 AM Michael Paquier <[email protected]> wrote:
>
> On Wed, Nov 18, 2020 at 10:50:08AM +0200, Heikki Linnakangas wrote:
> > If RESOWNER_ARRAY_STATS is increased to 16, all the lookups fit in the
> > array. But I haven't done any benchmarking to see which is faster.
>
> My gut tells me that your guess is right, but it would be better to be
> sure.
>
> > BTW, I think there would be an easy win in the hashing codepath, by changing
> > to a cheaper hash function. Currently, with or without this patch, we use
> > hash_any(). Changing that to murmurhash32() or something similar would be a
> > drop-in replacement.
>
> Good idea.
+1, and +1 for this refactoring.
I just saw a minor issue in a comment while reviewing the patch:
[...]
+ /* Is there space in the hash? If not, enlarge it. */
/* Double the capacity of the array (capacity must stay a power of 2!) */
- newcap = (oldcap > 0) ? oldcap * 2 : RESARRAY_INIT_SIZE;
[...]
The existing comment is kept as-is, but it should mention that it's
now the hash capacity that is increased.
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-11-19 10:40 Craig Ringer <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Craig Ringer @ 2020-11-19 10:40 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers; Michael Paquier <[email protected]>; Kyotaro Horiguchi <[email protected]>
[off-list for now]
On Tue, Nov 17, 2020 at 10:21 PM Heikki Linnakangas <[email protected]> wrote:
>
> 2. It's difficult for extensions to use. There is a callback mechanism
> for extensions, but it's much less convenient to use than the built-in
> functions. The pgcrypto code uses the callbacks currently, and Michael's
> patch [2] would move that support for tracking OpenSSL contexts to the
> core, which makes it a lot more convenient for pgcrypto. Wouldn't it be
> nice if extensions could have the same ergonomics as built-in code, if
> they need to track external resources?
>
Yes, in general I'm a big fan of making life easier for extensions (see
postscript), and this looks helpful. It's certainly fairly clear to read.
I'm in-principle in favour, though I haven't read the old resowner code
closely enough to have a strong opinion.
I haven't done thorough performance testing of this, but with some quick
> testing with Kyotaro's "catcachebench" to stress-test syscache lookups,
> this performs a little better. In that test, all the activity happens in
> the small array portion, but I believe that's true for most use cases.
>
> Thoughts? Can anyone suggest test scenarios to verify the performance of
> this?
>
I didn't think of much really.
I have tossed together a patch on top of yours that adds some
systemtap/dtrace tracepoints and provides a demo systemtap script that
shows some basic stats collection done using them. My intention was to make
it easier to see where the work in the resource owner code was being done.
But after I did the initial version I realised that in this case it
probably doesn't add a lot of value over using targeted profiling of only
functions in resowner.c and their callees which is easy enough using perf
and regular DWARF debuginfo. So I didn't land up polishing it and adding
stats for the other counters. It's attached anyway, in case it's
interesting or useful to anyone.
P.S. At some point I want to tackle some things similar to what you've done
here for other kinds of backend resources. In particular, to allow
extensions to register their own heavyweight lock types - with the ability
to handle lock timeouts, and add callbacks in the deadlock detector to
allow extensions to register dependency edges that are not natively visible
to the deadlock detector and to choose victim priorities. Also some
enhancements to how pg_depend tracking can be used by extensions. And I
want to help get the proposed custom ProcSignal changes in too. I think
there's a whole lot to be done to make extensions easier and safer to
author in general, like providing a simple way to do error trapping and
recovery in an extension mainloop without copy/pasting a bunch of
PostgresMain code, better default signal handlers, startup/shutdown that
shares more with user backends, etc. Right now it's quite tricky to get
bgworkers to behave well. </wildhandwaving>
Attachments:
[text/x-patch] 0001-Poc-of-systemtap-probes-for-resowner-timings.patch (14.5K, ../../CAGRY4nx+RrrcC+qxs10PAbnpeKC4we8Bf+E1qz+c-L3G6UkdUQ@mail.gmail.com/3-0001-Poc-of-systemtap-probes-for-resowner-timings.patch)
download | inline diff:
From 9f3e6978b1bbfeb8ca3f1cac42e86c4731143404 Mon Sep 17 00:00:00 2001
From: Craig Ringer <[email protected]>
Date: Wed, 18 Nov 2020 14:30:08 +0800
Subject: [PATCH] Poc of systemtap probes for resowner timings
---
resowner.stp | 106 ++++++++++++++++++++++++++
src/backend/utils/probes.d | 22 ++++++
src/backend/utils/resowner/resowner.c | 96 +++++++++++++++++++++--
3 files changed, 216 insertions(+), 8 deletions(-)
create mode 100644 resowner.stp
diff --git a/resowner.stp b/resowner.stp
new file mode 100644
index 0000000000..24b37b39e6
--- /dev/null
+++ b/resowner.stp
@@ -0,0 +1,106 @@
+# PATH="$(dirname $(realpath $(find build/ -name postgres -type f))):$PATH" /usr/local/systemtap/bin/stap -v ./resowner.stp
+
+/*
+.mark("resowner__created") $arg1:long $arg2:long
+.mark("resowner__delete__done") $arg1:long $arg2:long
+.mark("resowner__delete__start") $arg1:long $arg2:long
+.mark("resowner__enlarge__done") $arg1:long $arg2:long $arg3:long
+.mark("resowner__enlarge__skipped") $arg1:long $arg2:long
+.mark("resowner__enlarge__start") $arg1:long $arg2:long $arg3:long $arg4:long
+.mark("resowner__forget__done") $arg1:long $arg2:long $arg3:long $arg4:long $arg5:long
+.mark("resowner__forget__start") $arg1:long $arg2:long $arg3:long $arg4:long
+.mark("resowner__new__parent__done") $arg1:long
+.mark("resowner__new__parent__start") $arg1:long $arg2:long $arg3:long
+.mark("resowner__release__all__done") $arg1:long $arg2:long $arg3:long
+.mark("resowner__release__all__hash")
+.mark("resowner__release__all__start") $arg1:long $arg2:long $arg3:long $arg4:long $arg5:long
+.mark("resowner__release__done") $arg1:long $arg2:long
+.mark("resowner__release__plancacherefs__done") $arg1:long $arg2:long
+.mark("resowner__release__plancacherefs__start") $arg1:long $arg2:long
+.mark("resowner__release__start") $arg1:long $arg2:long $arg3:long $arg4:long $arg5:long
+.mark("resowner__remembered") $arg1:long $arg2:long $arg3:long $arg4:long
+*/
+
+@define probe_prefix %( sprintf("[%6d] %20s %20s %8x %20s", pid(), application_names[pid()], $$name, owner, owner_name) %)
+
+probe pg = process("postgres") {}
+
+# stats
+global operation_stats;
+
+# backend info
+private application_names;
+
+# in-progress operations
+private track_resowner_deletes;
+
+probe pg.begin {
+ if (@defined(@var("[email protected]","postgres"))) {
+ application_names[pid()] = user_string(@var("[email protected]","postgres"),"<?>")
+ } else {
+ application_names[pid()] = "<?>"
+ }
+ printf("[%6d] started %s\n", pid(), application_names[pid()])
+}
+
+probe pg.end {
+ delete application_names[pid()];
+ delete track_resowner_deletes[pid(),*];
+}
+
+probe pg.mark("resowner__created") {
+ owner = $arg1
+ owner_name = user_string($arg2)
+ printf("%s\n", @probe_prefix)
+}
+
+probe pg.mark("resowner__delete__start") {
+ owner = $arg1
+ owner_name = user_string($arg2)
+ track_resowner_deletes[pid(),owner] = gettimeofday_us()
+}
+
+probe pg.mark("resowner__delete__done") {
+ owner = $arg1
+ owner_name = user_string($arg2)
+ starttime_us = track_resowner_deletes[pid(),owner]
+ if (starttime_us != 0) {
+ elapsed = gettimeofday_us() - starttime_us
+ operation_stats["resowner_delete"] <<< elapsed
+ printf("%s: %s\n", @probe_prefix, usecs_to_string(elapsed));
+ }
+ delete track_resowner_deletes[pid(),owner]
+}
+
+function print_stat(statname) {
+ count = @count(operation_stats[statname])
+ if (count > 0) {
+ min = @min(operation_stats[statname])
+ max = @max(operation_stats[statname])
+ printf("-- %s:\n"
+ "-- count: %6d\n"
+ "-- mean: %6d\n"
+ "-- stddev: %6d\n"
+ "-- min: %6d\n"
+ "-- max: %6d\n",
+ statname,
+ count,
+ @avg(operation_stats[statname]),
+ @variance(operation_stats[statname])/count,
+ min, max
+ )
+ println(@hist_log(operation_stats[statname]))
+ }
+}
+
+function print_stats() {
+ print_stat("resowner_delete")
+}
+
+probe timer.ms(5000) {
+ print_stats()
+}
+
+probe end {
+ print_stats()
+ }
diff --git a/src/backend/utils/probes.d b/src/backend/utils/probes.d
index a0b0458108..7be815f9de 100644
--- a/src/backend/utils/probes.d
+++ b/src/backend/utils/probes.d
@@ -21,6 +21,9 @@
#define Oid unsigned int
#define ForkNumber int
#define bool unsigned char
+#define Datum long int
+#define ResourceOwnerData void
+#define ResourceReleasePhase int
provider postgresql {
@@ -91,4 +94,23 @@ provider postgresql {
probe wal__switch();
probe wal__buffer__write__dirty__start();
probe wal__buffer__write__dirty__done();
+
+ probe resowner__created(ResourceOwnerData*, const char *);
+ probe resowner__remembered(ResourceOwnerData*, const char *, Datum, const char *);
+ probe resowner__enlarge__skipped(ResourceOwnerData*, const char*);
+ probe resowner__enlarge__start(ResourceOwnerData*, const char *, int, int);
+ probe resowner__enlarge__done(ResourceOwnerData*, const char *, int);
+ probe resowner__forget__start(ResourceOwnerData*, const char *, Datum, const char *);
+ probe resowner__forget__done(ResourceOwnerData*, const char *, Datum, int, int);
+ probe resowner__release__all__start(ResourceOwnerData*, const char *, ResourceReleasePhase, int, int);
+ probe resowner__release__all__hash();
+ probe resowner__release__all__done(ResourceOwnerData*, const char *, ResourceReleasePhase);
+ probe resowner__release__start(ResourceOwnerData*, const char *, int, bool, bool);
+ probe resowner__release__done(ResourceOwnerData*, const char *);
+ probe resowner__delete__start(ResourceOwnerData*, const char *);
+ probe resowner__delete__done(long int, const char *);
+ probe resowner__release__plancacherefs__start(ResourceOwnerData*, const char *);
+ probe resowner__release__plancacherefs__done(ResourceOwnerData*, const char *);
+ probe resowner__new__parent__start(ResourceOwnerData*, ResourceOwnerData*, ResourceOwnerData*);
+ probe resowner__new__parent__done(ResourceOwnerData*);
};
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 2e1ae4f8e0..fcd2c6f61d 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -26,6 +26,7 @@
#include "storage/proc.h"
#include "utils/memutils.h"
#include "utils/plancache.h"
+#include "utils/probes.h"
#include "utils/resowner.h"
/*
@@ -152,10 +153,18 @@ static ResourceReleaseCallbackItem *ResourceRelease_callbacks = NULL;
/* Internal routines */
-static void ResourceOwnerReleaseInternal(ResourceOwner owner,
+static void ResourceOwnerReleaseRecurse(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
+static void ResourceOwnerReleaseSelf(ResourceOwner owner,
+ ResourceReleasePhase phase,
+ bool isCommit,
+ bool isTopLevel);
+static void ResourceOwnerReleaseCallbacks(ResourceOwner owner,
+ ResourceReleasePhase phase,
+ bool isCommit,
+ bool isTopLevel);
static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
@@ -194,9 +203,13 @@ void
ResourceOwnerEnlarge(ResourceOwner owner)
{
if (owner->narr < RESOWNER_ARRAY_SIZE)
+ {
+ TRACE_POSTGRESQL_RESOWNER_ENLARGE_SKIPPED(owner, owner->name);
return; /* no work needed */
+ }
/* Is there space in the hash? If not, enlarge it. */
+ TRACE_POSTGRESQL_RESOWNER_ENLARGE_START(owner, owner->name, owner->nhash, owner->narr);
/* Double the capacity of the array (capacity must stay a power of 2!) */
if (owner->narr + owner->nhash >= owner->grow_at)
@@ -246,6 +259,8 @@ ResourceOwnerEnlarge(ResourceOwner owner)
owner->narr = 0;
Assert(owner->nhash < owner->grow_at);
+
+ TRACE_POSTGRESQL_RESOWNER_ENLARGE_DONE(owner, owner->name, owner->nhash);
}
/*
@@ -270,6 +285,8 @@ ResourceOwnerRemember(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind
owner->arr[idx].item = value;
owner->arr[idx].kind = kind;
owner->narr++;
+
+ TRACE_POSTGRESQL_RESOWNER_REMEMBERED(owner, owner->name, value, kind->name);
}
/*
@@ -292,6 +309,8 @@ ResourceOwnerForget(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
#endif
+ TRACE_POSTGRESQL_RESOWNER_FORGET_START(owner, owner->name, value, kind->name);
+
/* Search through all items, but check the array first. */
for (i = 0; i < owner->narr; i++)
{
@@ -305,6 +324,7 @@ ResourceOwnerForget(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
narray_lookups++;
#endif
+ TRACE_POSTGRESQL_RESOWNER_FORGET_DONE(owner, owner->name, value, i, 0);
return;
}
}
@@ -327,6 +347,7 @@ ResourceOwnerForget(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
#ifdef RESOWNER_STATS
nhash_lookups++;
#endif
+ TRACE_POSTGRESQL_RESOWNER_FORGET_DONE(owner, owner->name, value, owner->narr, i);
return;
}
idx = (idx + 1) & mask;
@@ -351,6 +372,9 @@ ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
{
bool found;
+ TRACE_POSTGRESQL_RESOWNER_RELEASE_ALL_START(owner, owner->name, phase,
+ owner->narr, owner->nhash);
+
/* First handle all the entries in the array. */
do
{
@@ -377,6 +401,9 @@ ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
*/
} while (found && owner->narr > 0);
+ /* For probe/trace tools to time array and hash release separately */
+ TRACE_POSTGRESQL_RESOWNER_RELEASE_ALL_HASH();
+
/* Ok, the array has now been handled. Then the hash */
do
{
@@ -406,6 +433,8 @@ ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
*/
}
while (found && owner->nhash > 0);
+
+ TRACE_POSTGRESQL_RESOWNER_RELEASE_ALL_DONE(owner, owner->name, phase);
}
@@ -420,6 +449,9 @@ ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
*
* All ResourceOwner objects are kept in TopMemoryContext, since they should
* only be freed explicitly.
+ *
+ * The owner name is not copied. It should generally be a string constant. Otherwise
+ * the
*/
ResourceOwner
ResourceOwnerCreate(ResourceOwner parent, const char *name)
@@ -442,6 +474,8 @@ ResourceOwnerCreate(ResourceOwner parent, const char *name)
resowner_trace_counter++, owner, name);
#endif
+ TRACE_POSTGRESQL_RESOWNER_CREATED(owner, owner->name);
+
return owner;
}
@@ -478,7 +512,7 @@ ResourceOwnerRelease(ResourceOwner owner,
bool isTopLevel)
{
/* There's not currently any setup needed before recursing */
- ResourceOwnerReleaseInternal(owner, phase, isCommit, isTopLevel);
+ ResourceOwnerReleaseRecurse(owner, phase, isCommit, isTopLevel);
#ifdef RESOWNER_STATS
if (isTopLevel)
@@ -491,18 +525,19 @@ ResourceOwnerRelease(ResourceOwner owner,
}
static void
-ResourceOwnerReleaseInternal(ResourceOwner owner,
+ResourceOwnerReleaseRecurse(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel)
{
ResourceOwner child;
ResourceOwner save;
- ResourceReleaseCallbackItem *item;
+
+ TRACE_POSTGRESQL_RESOWNER_RELEASE_START(owner, owner->name, phase, isCommit, isTopLevel);
/* Recurse to handle descendants */
for (child = owner->firstchild; child != NULL; child = child->nextchild)
- ResourceOwnerReleaseInternal(child, phase, isCommit, isTopLevel);
+ ResourceOwnerReleaseRecurse(child, phase, isCommit, isTopLevel);
/*
* Make CurrentResourceOwner point to me, so that ReleaseBuffer etc don't
@@ -511,6 +546,27 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
save = CurrentResourceOwner;
CurrentResourceOwner = owner;
+ /* Release directly managed resources */
+ ResourceOwnerReleaseSelf(owner, phase, isCommit, isTopLevel);
+
+ /* And run any release callbacks */
+ ResourceOwnerReleaseCallbacks(owner, phase, isCommit, isTopLevel);
+
+ CurrentResourceOwner = save;
+
+ TRACE_POSTGRESQL_RESOWNER_RELEASE_DONE(owner, owner->name);
+}
+
+/*
+ * Non-recursive part of ResourceOwnerRelease for a single resowner's
+ * resources.
+ */
+static void
+ResourceOwnerReleaseSelf(ResourceOwner owner,
+ ResourceReleasePhase phase,
+ bool isCommit,
+ bool isTopLevel)
+{
if (phase == RESOURCE_RELEASE_BEFORE_LOCKS)
{
/*
@@ -577,12 +633,18 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
*/
ResourceOwnerReleaseAll(owner, phase, isCommit);
}
+}
+
+static void
+ResourceOwnerReleaseCallbacks(ResourceOwner owner,
+ ResourceReleasePhase phase,
+ bool isCommit,
+ bool isTopLevel)
+{
+ ResourceReleaseCallbackItem *item;
- /* Let add-on modules get a chance too */
for (item = ResourceRelease_callbacks; item; item = item->next)
item->callback(phase, isCommit, isTopLevel, item->arg);
-
- CurrentResourceOwner = save;
}
/*
@@ -600,6 +662,8 @@ ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner)
save = CurrentResourceOwner;
CurrentResourceOwner = owner;
+ TRACE_POSTGRESQL_RESOWNER_RELEASE_PLANCACHEREFS_START(owner, owner->name);
+
/* array first */
for (int i = 0; i < owner->narr; i++)
{
@@ -633,6 +697,8 @@ ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner)
}
CurrentResourceOwner = save;
+
+ TRACE_POSTGRESQL_RESOWNER_RELEASE_PLANCACHEREFS_DONE(owner, owner->name);
}
/*
@@ -644,6 +710,8 @@ ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner)
void
ResourceOwnerDelete(ResourceOwner owner)
{
+ const char * const name = owner->name;
+
/* We had better not be deleting CurrentResourceOwner ... */
Assert(owner != CurrentResourceOwner);
@@ -657,6 +725,8 @@ ResourceOwnerDelete(ResourceOwner owner)
resowner_trace_counter++, owner, owner->name);
#endif
+ TRACE_POSTGRESQL_RESOWNER_DELETE_START(owner, owner->name);
+
/*
* Delete children. The recursive call will delink the child from me, so
* just iterate as long as there is a child.
@@ -675,6 +745,12 @@ ResourceOwnerDelete(ResourceOwner owner)
if (owner->hash)
pfree(owner->hash);
pfree(owner);
+
+ /*
+ * Passing the free'd owner pointer here is deliberate, it serves to
+ * identify the freed owner when concurrently tracing many backends.
+ */
+ TRACE_POSTGRESQL_RESOWNER_DELETE_DONE((long int)owner, name);
}
/*
@@ -695,6 +771,8 @@ ResourceOwnerNewParent(ResourceOwner owner,
{
ResourceOwner oldparent = owner->parent;
+ TRACE_POSTGRESQL_RESOWNER_NEW_PARENT_START(owner, oldparent, newparent);
+
if (oldparent)
{
if (owner == oldparent->firstchild)
@@ -726,6 +804,8 @@ ResourceOwnerNewParent(ResourceOwner owner,
owner->parent = NULL;
owner->nextchild = NULL;
}
+
+ TRACE_POSTGRESQL_RESOWNER_NEW_PARENT_DONE(owner);
}
/*
--
2.26.2
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-11-26 07:29 Michael Paquier <[email protected]>
parent: Craig Ringer <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2020-11-26 07:29 UTC (permalink / raw)
To: Craig Ringer <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; pgsql-hackers; Kyotaro Horiguchi <[email protected]>
On Thu, Nov 19, 2020 at 06:40:19PM +0800, Craig Ringer wrote:
> [off-list for now]
Looks like you have missed something here.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../X79Za8w6t%[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-11-26 08:52 Kyotaro Horiguchi <[email protected]>
parent: Julien Rouhaud <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Kyotaro Horiguchi @ 2020-11-26 08:52 UTC (permalink / raw)
To: [email protected]; +Cc: [email protected]; [email protected]; pgsql-hackers; [email protected]
At Thu, 19 Nov 2020 17:27:18 +0800, Julien Rouhaud <[email protected]> wrote in
> On Thu, Nov 19, 2020 at 10:16 AM Michael Paquier <[email protected]> wrote:
> >
> > On Wed, Nov 18, 2020 at 10:50:08AM +0200, Heikki Linnakangas wrote:
> > > If RESOWNER_ARRAY_STATS is increased to 16, all the lookups fit in the
> > > array. But I haven't done any benchmarking to see which is faster.
> >
> > My gut tells me that your guess is right, but it would be better to be
> > sure.
> >
> > > BTW, I think there would be an easy win in the hashing codepath, by changing
> > > to a cheaper hash function. Currently, with or without this patch, we use
> > > hash_any(). Changing that to murmurhash32() or something similar would be a
> > > drop-in replacement.
> >
> > Good idea.
>
> +1, and +1 for this refactoring.
+1 for making the interface more generic. I thought a similar thing
when I added an resowner array for WaitEventSet (not committed yet).
About performance, though I'm not sure about, there's no reason not to
do this as far as the resowner mechanism doesn't get slower, and +2 if
gets faster.
> I just saw a minor issue in a comment while reviewing the patch:
>
> [...]
> + /* Is there space in the hash? If not, enlarge it. */
>
> /* Double the capacity of the array (capacity must stay a power of 2!) */
> - newcap = (oldcap > 0) ? oldcap * 2 : RESARRAY_INIT_SIZE;
> [...]
>
> The existing comment is kept as-is, but it should mention that it's
> now the hash capacity that is increased.
+ /* And release old array. */
+ pfree(oldhash);
:p
+ for (int i = 0; i < owner->narr; i++)
{
+ if (owner->arr[i].kind->phase == phase)
+ {
+ Datum value = owner->arr[i].item;
+ ResourceOwnerFuncs *kind = owner->arr[i].kind;
+
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+ found = true;
+ }
+ /*
+ * If any resources were released, check again because some of the
+ * elements might have moved by the callbacks. We don't want to
+ * miss them.
+ */
+ } while (found && owner->narr > 0);
Coundn't that missing be avoided by just not incrementing i if found?
+ /*
+ * Like with the array, we must check again after we reach the
+ * end, if any callbacks were called. XXX: We could probably
+ * stipulate that the callbacks may not do certain thing, like
+ * remember more references in the same resource owner, to avoid
+ * that.
+ */
If I read this patch correctly, ResourceOwnerForget doesn't seem to do
such a thing for hash?
regards.
--
Kyotaro Horiguchi
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2020-12-16 15:46 Heikki Linnakangas <[email protected]>
parent: Kyotaro Horiguchi <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2020-12-16 15:46 UTC (permalink / raw)
To: Kyotaro Horiguchi <[email protected]>; [email protected]; +Cc: [email protected]; pgsql-hackers; [email protected]
On 26/11/2020 10:52, Kyotaro Horiguchi wrote:
> + for (int i = 0; i < owner->narr; i++)
> {
> + if (owner->arr[i].kind->phase == phase)
> + {
> + Datum value = owner->arr[i].item;
> + ResourceOwnerFuncs *kind = owner->arr[i].kind;
> +
> + if (printLeakWarnings)
> + kind->PrintLeakWarning(value);
> + kind->ReleaseResource(value);
> + found = true;
> + }
> + /*
> + * If any resources were released, check again because some of the
> + * elements might have moved by the callbacks. We don't want to
> + * miss them.
> + */
> + } while (found && owner->narr > 0);
>
> Coundn't that missing be avoided by just not incrementing i if found?
Hmm, perhaps. ResourceOwnerForget() can move an entry from the end of
the array to the beginning, but if it's removing the entry that we're
just looking at, it probably can't move anything before that entry. I'm
not very comfortable with that, though. What if the callback releases
something else as a side effect?
This isn't super-critical for performance, and given that the array is
very small, it's very cheap to loop through it. So I'm inclined to keep
it safe.
> + /*
> + * Like with the array, we must check again after we reach the
> + * end, if any callbacks were called. XXX: We could probably
> + * stipulate that the callbacks may not do certain thing, like
> + * remember more references in the same resource owner, to avoid
> + * that.
> + */
>
> If I read this patch correctly, ResourceOwnerForget doesn't seem to do
> such a thing for hash?
Hmm, true. I tried removing the loop and hit another issue, however: if
the same resource has been remembered twice in the same resource owner,
the callback might remove different reference to it than what we're
looking at. So we need to loop anyway to handle that. Also, what if the
callback remembers some other resource in the same resowner, causing the
hash to grow? I'm not sure if any of the callbacks currently do that,
but better safe than sorry. I updated the code and the comments accordingly.
Here's an updated version of this patch. I fixed the bit rot, and
addressed all the other comment issues and such that people pointed out
(thanks!).
TODO:
1. Performance testing. We discussed this a little bit, but I haven't
done any more testing.
2. Before this patch, the ResourceOwnerEnlarge() calls enlarged the
array for the particular "kind" of resource, but now they are all stored
in the same structure. That could lead to trouble if we do something
like this:
ResourceOwnerEnlargeAAA()
ResourceOwnerEnlargeBBB()
ResourceOwnerRememberAAA()
ResourceOwnerRememberBBB()
Previously, this was OK, because resources AAA and BBB were kept in
separate arrays. But after this patch, it's not guaranteed that the
ResourceOwnerRememberBBB() will find an empty slot.
I don't think we do that currently, but to be sure, I'm planning to grep
ResourceOwnerRemember and look at each call carefullly. And perhaps we
can add an assertion for this, although I'm not sure where.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* RE: ResourceOwner refactoring
@ 2021-01-13 01:55 [email protected] <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: [email protected] @ 2021-01-13 01:55 UTC (permalink / raw)
To: 'Heikki Linnakangas' <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
Dear Heikki,
I'm also interested in this patch, but it cannot be applied to the current HEAD...
$ git apply ~/v2-0001-Make-resowners-more-easily-extensible.patch
error: patch failed: src/common/cryptohash_openssl.c:57
error: src/common/cryptohash_openssl.c: patch does not apply
error: patch failed: src/include/utils/resowner_private.h:1
error: src/include/utils/resowner_private.h: patch does not apply
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-13 07:18 Heikki Linnakangas <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-13 07:18 UTC (permalink / raw)
To: [email protected] <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
On 13/01/2021 03:55, [email protected] wrote:
> Dear Heikki,
>
> I'm also interested in this patch, but it cannot be applied to the current HEAD...
>
> $ git apply ~/v2-0001-Make-resowners-more-easily-extensible.patch
> error: patch failed: src/common/cryptohash_openssl.c:57
> error: src/common/cryptohash_openssl.c: patch does not apply
> error: patch failed: src/include/utils/resowner_private.h:1
> error: src/include/utils/resowner_private.h: patch does not apply
Here's a rebased version. Thanks!
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-13 07:22 Michael Paquier <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2021-01-13 07:22 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: [email protected] <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
On Wed, Jan 13, 2021 at 09:18:57AM +0200, Heikki Linnakangas wrote:
> --- a/src/common/cryptohash_openssl.c
> +++ b/src/common/cryptohash_openssl.c
> +static ResourceOwnerFuncs cryptohash_funcs =
> +{
> + /* relcache references */
> + .name = "LLVM JIT context",
> + .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
> + .ReleaseResource = ResOwnerReleaseCryptoHash,
> + .PrintLeakWarning = ResOwnerPrintCryptoHashLeakWarning,
> +};
> +#endif
Looks like a copy-paste error here.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../X%2F6fvGM%[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* RE: ResourceOwner refactoring
@ 2021-01-13 07:39 [email protected] <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: [email protected] @ 2021-01-13 07:39 UTC (permalink / raw)
To: 'Heikki Linnakangas' <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
Dear Heikki,
Thank you for rebasing it, I confirmed it can be applied.
I will check the source.
Now I put the very elementary comment.
ResourceOwnerEnlarge(), ResourceOwnerRemember(), and ResourceOwnerForget()
are exported routines.
They should put below L418.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 86+ messages in thread
* RE: ResourceOwner refactoring
@ 2021-01-14 10:15 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: [email protected] @ 2021-01-14 10:15 UTC (permalink / raw)
To: 'Heikki Linnakangas' <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>
Hi,
I put some comments.
Throughout, some components don’t have helper functions.
(e.g. catcache has ResOwnerReleaseCatCache, but tupdesc doesn't.)
I think it should be unified.
[catcache.c]
> +/* support for catcache refcount management */
> +static inline void
> +ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner)
> +{
> + ResourceOwnerEnlarge(owner);
> +}
This function is not needed.
[syscache.c]
> -static CatCache *SysCache[SysCacheSize];
> + CatCache *SysCache[SysCacheSize];
Is it right? Compilation is done even if this variable is static...
[fd.c, dsm.c]
In these files helper functions are implemented as the define directive.
Could you explain the reason? For the performance?
> Previously, this was OK, because resources AAA and BBB were kept in
> separate arrays. But after this patch, it's not guaranteed that the
> ResourceOwnerRememberBBB() will find an empty slot.
>
> I don't think we do that currently, but to be sure, I'm planning to grep
> ResourceOwnerRemember and look at each call carefullly. And perhaps we
> can add an assertion for this, although I'm not sure where.
Indeed, but I think this line works well, isn't it?
> Assert(owner->narr < RESOWNER_ARRAY_SIZE);
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-14 13:34 Heikki Linnakangas <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-14 13:34 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>
On 14/01/2021 12:15, [email protected] wrote:
> I put some comments.
Thanks for the review!
> Throughout, some components don’t have helper functions.
> (e.g. catcache has ResOwnerReleaseCatCache, but tupdesc doesn't.)
> I think it should be unified.
Hmm. ResOwnerReleaseTupleDesc() does exist, those functions are needed
for the callbacks. I think you meant the wrappers around
ResourceOwnerRemember and ResourceOwnerForget, like
ResourceOwnerRememberCatCacheRef(). I admit those are not fully
consistent: I didn't bother with the wrapper functions when there is
only one caller.
> [catcache.c]
>> +/* support for catcache refcount management */
>> +static inline void
>> +ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner)
>> +{
>> + ResourceOwnerEnlarge(owner);
>> +}
>
> This function is not needed.
Removed.
> [syscache.c]
>> -static CatCache *SysCache[SysCacheSize];
>> + CatCache *SysCache[SysCacheSize];
>
> Is it right? Compilation is done even if this variable is static...
Fixed. (It was a leftover from when I was playing with Kyotaro's
"catcachebench" tool from another thread).
> [fd.c, dsm.c]
> In these files helper functions are implemented as the define directive.
> Could you explain the reason? For the performance?
No particular reason. I turned them all into macros for consistency.
>> Previously, this was OK, because resources AAA and BBB were kept in
>> separate arrays. But after this patch, it's not guaranteed that the
>> ResourceOwnerRememberBBB() will find an empty slot.
>>
>> I don't think we do that currently, but to be sure, I'm planning to grep
>> ResourceOwnerRemember and look at each call carefullly. And perhaps we
>> can add an assertion for this, although I'm not sure where.
>
> Indeed, but I think this line works well, isn't it?
>
>> Assert(owner->narr < RESOWNER_ARRAY_SIZE);
That catches cases where you actually overrun the array, but it doesn't
catch unsafe patterns when there happens to be enough space left in the
array. For example, if you have code like this:
/* Make sure there's room for one more entry, but remember *two* things */
ResourceOwnerEnlarge();
ResourceOwnerRemember(foo);
ResourceOwnerRemember(bar);
That is not safe, but it would only fail the assertion if the first
ResourceOwnerRemember() call happens to consume the last remaining slot
in the array.
I checked all the callers of ResourceOwnerEnlarge() to see if they're
safe. A couple of places seemed a bit suspicious. I fixed them by moving
the ResourceOwnerEnlarge() calls closer to the ResourceOwnerRemember()
calls, so that it's now easier to see that they are correct. See first
attached patch.
The second patch is an updated version of the main patch, fixing all the
little things you and Michael pointed out since the last patch version.
I've been working on performance testing too. I'll post more numbers
later, but preliminary result from some micro-benchmarking suggests that
the new code is somewhat slower, except in the common case that the
object to remember and forget fits in the array. When running the
regression test suite, about 96% of ResourceOwnerForget() calls fit in
the array. I think that's acceptable.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* RE: ResourceOwner refactoring
@ 2021-01-15 09:10 [email protected] <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: [email protected] @ 2021-01-15 09:10 UTC (permalink / raw)
To: 'Heikki Linnakangas' <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; Kyotaro Horiguchi <[email protected]>; [email protected] <[email protected]>
Dear Heikki,
> Hmm. ResOwnerReleaseTupleDesc() does exist, those functions are needed
> for the callbacks. I think you meant the wrappers around
> ResourceOwnerRemember and ResourceOwnerForget, like
> ResourceOwnerRememberCatCacheRef(). I admit those are not fully
> consistent: I didn't bother with the wrapper functions when there is
> only one caller.
Yeah, I meant it. And I prefer your policy.
> Hmm. ResOwnerReleaseTupleDesc() does exist, those functions are needed
> for the callbacks. I think you meant the wrappers around
> ResourceOwnerRemember and ResourceOwnerForget, like
> ResourceOwnerRememberCatCacheRef(). I admit those are not fully
> consistent: I didn't bother with the wrapper functions when there is
> only one caller.
Good job. I confirmed your fixes, and I confirmed it looks fine.
I will check another ResourceOwnerEnlarge() if I have a time.
> I've been working on performance testing too.
I'm looking forward to seeing it.
Best regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 86+ messages in thread
* RE: ResourceOwner refactoring
@ 2021-01-18 07:49 [email protected] <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: [email protected] @ 2021-01-18 07:49 UTC (permalink / raw)
To: 'Heikki Linnakangas' <[email protected]>; +Cc: pgsql-hackers
Dear Heikki,
I apologize for sending again.
> I will check another ResourceOwnerEnlarge() if I have a time.
I checked all ResourceOwnerEnlarge() types after applying patch 0001.
(searched by "grep -rI ResourceOwnerEnlarge")
No problem was found.
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-18 12:22 Heikki Linnakangas <[email protected]>
parent: [email protected] <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-18 12:22 UTC (permalink / raw)
To: [email protected] <[email protected]>; +Cc: pgsql-hackers
On 18/01/2021 09:49, [email protected] wrote:
> Dear Heikki,
>
> I apologize for sending again.
>
>> I will check another ResourceOwnerEnlarge() if I have a time.
>
> I checked all ResourceOwnerEnlarge() types after applying patch 0001.
> (searched by "grep -rI ResourceOwnerEnlarge")
> No problem was found.
Great, thanks!
Here are more details on the performance testing I did:
Unpatched
----------
postgres=# \i snaptest.sql
numkeep | numsnaps | lifo_time_ns | fifo_time_ns
---------+----------+--------------+--------------
0 | 1 | 38.2 | 34.8
0 | 5 | 35.7 | 35.4
0 | 10 | 37.6 | 37.6
0 | 60 | 35.4 | 39.2
0 | 70 | 55.0 | 53.8
0 | 100 | 48.6 | 48.9
0 | 1000 | 54.8 | 57.0
0 | 10000 | 63.9 | 67.1
9 | 10 | 39.3 | 38.8
9 | 100 | 44.0 | 42.5
9 | 1000 | 45.8 | 47.1
9 | 10000 | 64.2 | 66.8
65 | 70 | 45.7 | 43.7
65 | 100 | 42.9 | 43.7
65 | 1000 | 46.9 | 45.7
65 | 10000 | 65.0 | 64.5
(16 rows)
Patched
--------
postgres=# \i snaptest.sql
numkeep | numsnaps | lifo_time_ns | fifo_time_ns
---------+----------+--------------+--------------
0 | 1 | 35.3 | 33.8
0 | 5 | 34.8 | 34.6
0 | 10 | 49.8 | 51.4
0 | 60 | 53.1 | 57.2
0 | 70 | 53.2 | 59.6
0 | 100 | 53.1 | 58.2
0 | 1000 | 63.1 | 69.7
0 | 10000 | 83.3 | 83.5
9 | 10 | 35.0 | 35.1
9 | 100 | 55.4 | 54.1
9 | 1000 | 56.8 | 60.3
9 | 10000 | 88.6 | 82.0
65 | 70 | 36.4 | 35.1
65 | 100 | 52.4 | 53.0
65 | 1000 | 55.8 | 59.4
65 | 10000 | 79.3 | 80.3
(16 rows)
About the test:
Each test call Register/UnregisterSnapshot in a loop. numsnaps is the
number of snapshots that are registered in each iteration. For exmaple,
on the second line with numkeep=0 and numnaps=5, each iteration in the
LIFO test does essentially:
rs[0] = RegisterSnapshot()
rs[1] = RegisterSnapshot()
rs[2] = RegisterSnapshot()
rs[3] = RegisterSnapshot()
rs[4] = RegisterSnapshot()
UnregisterSnapshot(rs[4]);
UnregisterSnapshot(rs[3]);
UnregisterSnapshot(rs[2]);
UnregisterSnapshot(rs[1]);
UnregisterSnapshot(rs[0]);
In the FIFO tests, the Unregister calls are made in reverse order.
When numkeep is non zero, that many snapshots are registered once at the
beginning of the test, and released only after the test loop. So this
simulates the scenario that you remember a bunch of resources and hold
them for a long time, and while holding them, you do many more
remember/forget calls.
Based on this test, this patch makes things slightly slower overall.
However, *not* in the cases that matter the most I believe. That's the
cases where the (numsnaps - numkeep) is small, so that the hot action
happens in the array and the hashing is not required. In my testing
earlier, about 95% of the ResourceOwnerRemember calls in the regression
tests fall into that category.
There are a few simple things we could do speed this up, if needed. A
different hash function might be cheaper, for example. And we could
inline the fast paths of the ResourceOwnerEnlarge and
ResourceOwnerRemember() into the callers. But I didn't do those things
as part of this patch, because it wouldn't be a fair comparison; we
could do those with the old implementation too. And unless we really
need the speed, it's more readable this way.
I'm OK with these results. Any objections?
Attached are the patches again. Same as previous version, except for a
couple of little comment changes. And a third patch containing the
source for the performance test.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-18 14:34 Alvaro Herrera <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 2 replies; 86+ messages in thread
From: Alvaro Herrera @ 2021-01-18 14:34 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers
On 2021-Jan-18, Heikki Linnakangas wrote:
> +static ResourceOwnerFuncs jit_funcs =
> +{
> + /* relcache references */
> + .name = "LLVM JIT context",
> + .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
> + .ReleaseResource = ResOwnerReleaseJitContext,
> + .PrintLeakWarning = ResOwnerPrintJitContextLeakWarning
> +};
I think you mean jit_resowner_funcs here; "jit_funcs" is a bit
excessively vague. Also, why did you choose not to define
ResourceOwnerRememberJIT? You do that in other modules and it seems
better.
> +static ResourceOwnerFuncs catcache_funcs =
> +{
> + /* catcache references */
> + .name = "catcache reference",
> + .phase = RESOURCE_RELEASE_AFTER_LOCKS,
> + .ReleaseResource = ResOwnerReleaseCatCache,
> + .PrintLeakWarning = ResOwnerPrintCatCacheLeakWarning
> +};
> +
> +static ResourceOwnerFuncs catlistref_funcs =
> +{
> + /* catcache-list pins */
> + .name = "catcache list reference",
> + .phase = RESOURCE_RELEASE_AFTER_LOCKS,
> + .ReleaseResource = ResOwnerReleaseCatCacheList,
> + .PrintLeakWarning = ResOwnerPrintCatCacheListLeakWarning
> +};
Similar naming issue here, I say.
> diff --git a/src/common/cryptohash_openssl.c b/src/common/cryptohash_openssl.c
> index 551ec392b60..642e72d8c0f 100644
> --- a/src/common/cryptohash_openssl.c
> +++ b/src/common/cryptohash_openssl.c
> @@ -60,6 +59,21 @@ struct pg_cryptohash_ctx
> #endif
> };
>
> +/* ResourceOwner callbacks to hold JitContexts */
> +#ifndef FRONTEND
> +static void ResOwnerReleaseCryptoHash(Datum res);
> +static void ResOwnerPrintCryptoHashLeakWarning(Datum res);
The comment is wrong -- "Crypohashes", not "JitContexts".
So according to your performance benchmark, we're willing to accept a
30% performance loss on an allegedly common operation -- numkeep=0
numsnaps=10 becomes 49.8ns from 37.6ns. That seems a bit shocking.
Maybe you can claim that these operations aren't exactly hot spots, and
so the fact that we remain in the same power-of-ten is sufficient. Is
that the argument?
--
Álvaro Herrera 39°49'30"S 73°17'W
"The Postgresql hackers have what I call a "NASA space shot" mentality.
Quite refreshing in a world of "weekend drag racer" developers."
(Scott Marlowe)
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-18 15:19 Heikki Linnakangas <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-18 15:19 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers
On 18/01/2021 16:34, Alvaro Herrera wrote:
> So according to your performance benchmark, we're willing to accept a
> 30% performance loss on an allegedly common operation -- numkeep=0
> numsnaps=10 becomes 49.8ns from 37.6ns. That seems a bit shocking.
> Maybe you can claim that these operations aren't exactly hot spots, and
> so the fact that we remain in the same power-of-ten is sufficient. Is
> that the argument?
That's right. The fast path is fast, and that's important. The slow path
becomes 30% slower, but that's acceptable.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-18 16:11 Robert Haas <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Robert Haas @ 2021-01-18 16:11 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Mon, Jan 18, 2021 at 10:19 AM Heikki Linnakangas <[email protected]> wrote:
>
> On 18/01/2021 16:34, Alvaro Herrera wrote:
> > So according to your performance benchmark, we're willing to accept a
> > 30% performance loss on an allegedly common operation -- numkeep=0
> > numsnaps=10 becomes 49.8ns from 37.6ns. That seems a bit shocking.
> > Maybe you can claim that these operations aren't exactly hot spots, and
> > so the fact that we remain in the same power-of-ten is sufficient. Is
> > that the argument?
>
> That's right. The fast path is fast, and that's important. The slow path
> becomes 30% slower, but that's acceptable.
>
> - Heikki
>
>
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-18 16:11 Robert Haas <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Robert Haas @ 2021-01-18 16:11 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Mon, Jan 18, 2021 at 11:11 AM Robert Haas <[email protected]> wrote:
> On Mon, Jan 18, 2021 at 10:19 AM Heikki Linnakangas <[email protected]> wrote:
> > On 18/01/2021 16:34, Alvaro Herrera wrote:
> > > So according to your performance benchmark, we're willing to accept a
> > > 30% performance loss on an allegedly common operation -- numkeep=0
> > > numsnaps=10 becomes 49.8ns from 37.6ns. That seems a bit shocking.
> > > Maybe you can claim that these operations aren't exactly hot spots, and
> > > so the fact that we remain in the same power-of-ten is sufficient. Is
> > > that the argument?
> >
> > That's right. The fast path is fast, and that's important. The slow path
> > becomes 30% slower, but that's acceptable.
Sorry for the empty message.
I don't know whether a 30% slowdown will hurt anybody, but it seems
like kind of a lot, and I'm not sure I understand what corresponding
benefit we're getting.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-19 06:48 Michael Paquier <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2021-01-19 06:48 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers
On Mon, Jan 18, 2021 at 02:22:33PM +0200, Heikki Linnakangas wrote:
> diff --git a/src/common/cryptohash_openssl.c b/src/common/cryptohash_openssl.c
> index 551ec392b60..642e72d8c0f 100644
> --- a/src/common/cryptohash_openssl.c
> +++ b/src/common/cryptohash_openssl.c
[...]
> +/* ResourceOwner callbacks to hold JitContexts */
Slight copy-paste error here.
> /*
> * Ensure, while the spinlock's not yet held, that there's a free
> - * refcount entry.
> + * refcount entry and that the resoure owner has room to remember the
> + * pin.
s/resoure/resource/.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-19 09:09 Heikki Linnakangas <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-19 09:09 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On 18/01/2021 18:11, Robert Haas wrote:
> On Mon, Jan 18, 2021 at 11:11 AM Robert Haas <[email protected]> wrote:
>> On Mon, Jan 18, 2021 at 10:19 AM Heikki Linnakangas <[email protected]> wrote:
>>> On 18/01/2021 16:34, Alvaro Herrera wrote:
>>>> So according to your performance benchmark, we're willing to accept a
>>>> 30% performance loss on an allegedly common operation -- numkeep=0
>>>> numsnaps=10 becomes 49.8ns from 37.6ns. That seems a bit shocking.
>>>> Maybe you can claim that these operations aren't exactly hot spots, and
>>>> so the fact that we remain in the same power-of-ten is sufficient. Is
>>>> that the argument?
>>>
>>> That's right. The fast path is fast, and that's important. The slow path
>>> becomes 30% slower, but that's acceptable.
>
> I don't know whether a 30% slowdown will hurt anybody, but it seems
> like kind of a lot, and I'm not sure I understand what corresponding
> benefit we're getting.
The benefit is to make it easy for extensions to use resource owners to
track whatever resources they need to track. And arguably, the new
mechanism is nicer for built-in code, too.
I'll see if I can optimize the slow paths, to make it more palatable.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-19 14:45 Heikki Linnakangas <[email protected]>
parent: Alvaro Herrera <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-19 14:45 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; +Cc: [email protected] <[email protected]>; pgsql-hackers
On 18/01/2021 16:34, Alvaro Herrera wrote:
> On 2021-Jan-18, Heikki Linnakangas wrote:
>
>> +static ResourceOwnerFuncs jit_funcs =
>> +{
>> + /* relcache references */
>> + .name = "LLVM JIT context",
>> + .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
>> + .ReleaseResource = ResOwnerReleaseJitContext,
>> + .PrintLeakWarning = ResOwnerPrintJitContextLeakWarning
>> +};
>
> I think you mean jit_resowner_funcs here; "jit_funcs" is a bit
> excessively vague. Also, why did you choose not to define
> ResourceOwnerRememberJIT? You do that in other modules and it seems
> better.
I did it in modules that had more than one ResourceOwnerRemeber/Forget
call. Didn't seem worth it in functions like IncrTupleDescRefCount(),
for example.
Hayato Kuroda also pointed that out, though. So perhaps it's better to
be consistent, to avoid the confusion. I'll add the missing wrappers.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-20 22:11 Heikki Linnakangas <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-20 22:11 UTC (permalink / raw)
To: Robert Haas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On 19/01/2021 11:09, Heikki Linnakangas wrote:
> On 18/01/2021 18:11, Robert Haas wrote:
>> On Mon, Jan 18, 2021 at 11:11 AM Robert Haas <[email protected]> wrote:
>>> On Mon, Jan 18, 2021 at 10:19 AM Heikki Linnakangas <[email protected]> wrote:
>>>> On 18/01/2021 16:34, Alvaro Herrera wrote:
>>>>> So according to your performance benchmark, we're willing to accept a
>>>>> 30% performance loss on an allegedly common operation -- numkeep=0
>>>>> numsnaps=10 becomes 49.8ns from 37.6ns. That seems a bit shocking.
>>>>> Maybe you can claim that these operations aren't exactly hot spots, and
>>>>> so the fact that we remain in the same power-of-ten is sufficient. Is
>>>>> that the argument?
>>>>
>>>> That's right. The fast path is fast, and that's important. The slow path
>>>> becomes 30% slower, but that's acceptable.
>>
>> I don't know whether a 30% slowdown will hurt anybody, but it seems
>> like kind of a lot, and I'm not sure I understand what corresponding
>> benefit we're getting.
>
> The benefit is to make it easy for extensions to use resource owners to
> track whatever resources they need to track. And arguably, the new
> mechanism is nicer for built-in code, too.
>
> I'll see if I can optimize the slow paths, to make it more palatable.
Ok, here's a new set of patches, and new test results. I replaced the
hash function with a cheaper one. I also added the missing wrappers that
Alvaro and Hayato Kuroda commented on, and fixed the typos that Michael
Paquier pointed out.
In the test script, I increased the number of iterations used in the
tests, to make them run longer and produce more stable results. There is
still a fair amount of jitter in the results, so take any particular
number with a grain of salt, but the overall trend is repeatable.
The results now look like this:
Unpatched
---------
postgres=# \i snaptest.sql
numkeep | numsnaps | lifo_time_ns | fifo_time_ns
---------+----------+--------------+--------------
0 | 1 | 32.7 | 32.3
0 | 5 | 33.0 | 32.9
0 | 10 | 34.1 | 35.5
0 | 60 | 32.5 | 38.3
0 | 70 | 47.6 | 48.9
0 | 100 | 47.9 | 49.3
0 | 1000 | 52.9 | 52.7
0 | 10000 | 61.7 | 62.4
9 | 10 | 38.4 | 37.6
9 | 100 | 42.3 | 42.3
9 | 1000 | 43.9 | 45.0
9 | 10000 | 62.2 | 62.5
65 | 70 | 42.4 | 42.9
65 | 100 | 43.2 | 43.9
65 | 1000 | 44.0 | 45.1
65 | 10000 | 62.4 | 62.6
(16 rows)
Patched
-------
postgres=# \i snaptest.sql
numkeep | numsnaps | lifo_time_ns | fifo_time_ns
---------+----------+--------------+--------------
0 | 1 | 32.8 | 34.2
0 | 5 | 33.8 | 32.2
0 | 10 | 37.2 | 40.2
0 | 60 | 40.2 | 45.1
0 | 70 | 40.9 | 48.4
0 | 100 | 41.9 | 45.2
0 | 1000 | 49.0 | 55.6
0 | 10000 | 62.4 | 67.2
9 | 10 | 33.6 | 33.0
9 | 100 | 39.6 | 39.7
9 | 1000 | 42.2 | 45.0
9 | 10000 | 60.7 | 63.9
65 | 70 | 32.5 | 33.6
65 | 100 | 37.5 | 39.7
65 | 1000 | 42.3 | 46.3
65 | 10000 | 61.9 | 64.8
(16 rows)
For easier side-by-side comparison, here are the same numbers with the
patched and unpatched results side by side, and their ratio (ratio < 1
means the patched version is faster):
LIFO tests:
numkeep | numsnaps | unpatched | patched | ratio
---------+----------+-----------+---------+-------
0 | 1 | 32.7 | 32.8 | 1.00
0 | 5 | 33.0 | 33.8 | 1.02
0 | 10 | 34.1 | 37.2 | 1.09
0 | 60 | 32.5 | 40.2 | 1.24
0 | 70 | 47.6 | 40.9 | 0.86
0 | 100 | 47.9 | 41.9 | 0.87
0 | 1000 | 52.9 | 49.0 | 0.93
0 | 10000 | 61.7 | 62.4 | 1.01
9 | 10 | 38.4 | 33.6 | 0.88
9 | 100 | 42.3 | 39.6 | 0.94
9 | 1000 | 43.9 | 42.2 | 0.96
9 | 10000 | 62.2 | 60.7 | 0.98
65 | 70 | 42.4 | 32.5 | 0.77
65 | 100 | 43.2 | 37.5 | 0.87
65 | 1000 | 44.0 | 42.3 | 0.96
65 | 10000 | 62.4 | 61.9 | 0.99
(16 rows)
FIFO tests:
numkeep | numsnaps | unpatched | patched | ratio
---------+----------+-----------+---------+-------
0 | 1 | 32.3 | 34.2 | 1.06
0 | 5 | 32.9 | 32.2 | 0.98
0 | 10 | 35.5 | 40.2 | 1.13
0 | 60 | 38.3 | 45.1 | 1.18
0 | 70 | 48.9 | 48.4 | 0.99
0 | 100 | 49.3 | 45.2 | 0.92
0 | 1000 | 52.7 | 55.6 | 1.06
0 | 10000 | 62.4 | 67.2 | 1.08
9 | 10 | 37.6 | 33.0 | 0.88
9 | 100 | 42.3 | 39.7 | 0.94
9 | 1000 | 45.0 | 45.0 | 1.00
9 | 10000 | 62.5 | 63.9 | 1.02
65 | 70 | 42.9 | 33.6 | 0.78
65 | 100 | 43.9 | 39.7 | 0.90
65 | 1000 | 45.1 | 46.3 | 1.03
65 | 10000 | 62.6 | 64.8 | 1.04
(16 rows)
Summary: In the the worst scenario, the patched version is still 24%
slower than unpatched. But many other scenarios are now faster with the
patch.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* RE: ResourceOwner refactoring
@ 2021-01-21 02:22 [email protected] <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: [email protected] @ 2021-01-21 02:22 UTC (permalink / raw)
To: 'Heikki Linnakangas' <[email protected]>; Robert Haas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; pgsql-hackers
Dear Heikki,
I tested in the same situation, and I confirmed that almost same results are returned.
(results are at the end of the e-mail)
You think that these results are acceptable
because resowners own many resources(more than 64) in general
and it's mainly faster in such a situation, isn't it?
I cannot distinguish correctly, but sounds good.
====test result====
unpatched
numkeep | numsnaps | lifo_time_ns | fifo_time_ns
---------+----------+--------------+--------------
0 | 1 | 68.5 | 69.9
0 | 5 | 73.2 | 76.8
0 | 10 | 70.6 | 74.7
0 | 60 | 68.0 | 75.6
0 | 70 | 91.3 | 94.8
0 | 100 | 89.0 | 89.1
0 | 1000 | 97.9 | 98.9
0 | 10000 | 116.0 | 115.9
9 | 10 | 74.7 | 76.6
9 | 100 | 80.8 | 80.1
9 | 1000 | 86.0 | 86.2
9 | 10000 | 116.1 | 116.8
65 | 70 | 84.7 | 85.3
65 | 100 | 80.5 | 80.3
65 | 1000 | 86.3 | 86.2
65 | 10000 | 115.4 | 115.9
(16 rows)
patched
numkeep | numsnaps | lifo_time_ns | fifo_time_ns
---------+----------+--------------+--------------
0 | 1 | 62.4 | 62.6
0 | 5 | 68.0 | 66.9
0 | 10 | 73.6 | 78.1
0 | 60 | 82.3 | 87.2
0 | 70 | 83.0 | 89.1
0 | 100 | 82.8 | 87.9
0 | 1000 | 88.2 | 96.6
0 | 10000 | 119.6 | 124.5
9 | 10 | 62.0 | 62.8
9 | 100 | 75.3 | 78.0
9 | 1000 | 82.6 | 89.3
9 | 10000 | 116.6 | 122.6
65 | 70 | 66.7 | 66.4
65 | 100 | 74.6 | 77.2
65 | 1000 | 82.1 | 88.2
65 | 10000 | 118.0 | 124.1
(16 rows)
Best Regards,
Hayato Kuroda
FUJITSU LIMITED
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-21 04:14 Michael Paquier <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Michael Paquier @ 2021-01-21 04:14 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thu, Jan 21, 2021 at 12:11:37AM +0200, Heikki Linnakangas wrote:
> Summary: In the the worst scenario, the patched version is still 24% slower
> than unpatched. But many other scenarios are now faster with the patch.
Is there a reason explaining the sudden drop for numsnaps within the
[10,60] range? The gap looks deeper with a low numkeep.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../YAj%[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-21 10:14 Heikki Linnakangas <[email protected]>
parent: Michael Paquier <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-01-21 10:14 UTC (permalink / raw)
To: Michael Paquier <[email protected]>; +Cc: Robert Haas <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On 21/01/2021 06:14, Michael Paquier wrote:
> On Thu, Jan 21, 2021 at 12:11:37AM +0200, Heikki Linnakangas wrote:
>> Summary: In the the worst scenario, the patched version is still 24% slower
>> than unpatched. But many other scenarios are now faster with the patch.
>
> Is there a reason explaining the sudden drop for numsnaps within the
> [10,60] range? The gap looks deeper with a low numkeep.
It's the switch from array to hash table. With the patch, the array
holds 8 entries. Without the patch, it's 64 entries. So you see a drop
around those points. I added more data points in that range to get a
better picture:
LIFO tests:
numkeep | numsnaps | unpatched | patched | ratio
---------+----------+-----------+---------+-------
0 | 1 | 32.8 | 32.9 | 1.00
0 | 2 | 31.6 | 32.8 | 1.04
0 | 4 | 32.7 | 32.0 | 0.98
0 | 6 | 34.1 | 33.9 | 0.99
0 | 8 | 35.1 | 32.4 | 0.92
0 | 10 | 34.0 | 37.1 | 1.09
0 | 15 | 33.1 | 35.9 | 1.08
0 | 20 | 33.0 | 38.8 | 1.18
0 | 25 | 32.9 | 42.3 | 1.29
0 | 30 | 32.9 | 40.5 | 1.23
0 | 35 | 33.1 | 39.9 | 1.21
0 | 40 | 33.0 | 39.0 | 1.18
0 | 45 | 35.3 | 41.1 | 1.16
0 | 50 | 33.0 | 40.8 | 1.24
0 | 55 | 32.8 | 40.6 | 1.24
0 | 58 | 33.0 | 41.5 | 1.26
0 | 60 | 33.1 | 41.6 | 1.26
0 | 62 | 32.8 | 41.7 | 1.27
0 | 64 | 46.8 | 40.9 | 0.87
0 | 66 | 47.0 | 42.5 | 0.90
0 | 68 | 47.1 | 41.8 | 0.89
0 | 70 | 47.8 | 41.7 | 0.87
(22 rows)
FIFO tests:
numkeep | numsnaps | unpatched | patched | ratio
---------+----------+-----------+---------+-------
0 | 1 | 32.3 | 32.1 | 0.99
0 | 2 | 33.4 | 31.6 | 0.95
0 | 4 | 34.0 | 31.4 | 0.92
0 | 6 | 35.4 | 33.2 | 0.94
0 | 8 | 34.8 | 31.9 | 0.92
0 | 10 | 35.4 | 40.2 | 1.14
0 | 15 | 35.4 | 40.3 | 1.14
0 | 20 | 35.6 | 43.8 | 1.23
0 | 25 | 35.4 | 42.4 | 1.20
0 | 30 | 36.0 | 43.3 | 1.20
0 | 35 | 36.4 | 45.1 | 1.24
0 | 40 | 36.9 | 46.6 | 1.26
0 | 45 | 37.7 | 45.3 | 1.20
0 | 50 | 37.2 | 43.9 | 1.18
0 | 55 | 38.4 | 46.8 | 1.22
0 | 58 | 37.6 | 45.0 | 1.20
0 | 60 | 37.7 | 46.6 | 1.24
0 | 62 | 38.4 | 46.5 | 1.21
0 | 64 | 48.7 | 47.6 | 0.98
0 | 66 | 48.2 | 45.8 | 0.95
0 | 68 | 48.5 | 47.5 | 0.98
0 | 70 | 48.4 | 47.3 | 0.98
(22 rows)
Let's recap the behavior:
Without patch
-------------
For each different kind of resource, there's an array that holds up to
64 entries. In ResourceOwnerForget(), the array is scanned linearly. If
the array fills up, we replace the array with a hash table. After
switching, all operations use the hash table.
With patch
----------
There is one array that holds up to 8 entries. It is shared by all
resources. In ResourceOwnerForget(), the array is always scanned.
If the array fills up, all the entries are moved to a hash table,
freeing up space in the array, and the new entry is added to the array.
So the array is used together with the hash table, like an LRU cache of
the most recently remembered entries.
Why this change? I was afraid that now that all different resources
share the same data structure, remembering e.g. a lot of locks at the
beginning of a transaction would cause the switch to the hash table,
making all subsequent remember/forget operations, like for buffer pins,
slower. That kind of interference seems bad. By continuing to use the
array for the recently-remembered entries, we avoid that problem. The
[numkeep, numsnaps] = [65, 70] test is in that regime, and the patched
version was significantly faster.
Because the array is now always scanned, I felt that it needs to be
small, to avoid wasting much time scanning for entries that have already
been moved to the hash table. That's why I made it just 8 entries.
Perhaps 8 entries is too small, after all? Linearly scanning an array is
very fast. To test that, I bumped up RESOWNER_ARRAY_SIZE to 64, and ran
the test again:
numkeep | numsnaps | lifo_time_ns | fifo_time_ns
---------+----------+--------------+--------------
0 | 1 | 35.4 | 31.5
0 | 2 | 32.3 | 32.3
0 | 4 | 32.8 | 31.0
0 | 6 | 34.5 | 33.7
0 | 8 | 33.9 | 32.7
0 | 10 | 33.7 | 33.0
0 | 15 | 34.8 | 33.1
0 | 20 | 35.0 | 32.6
0 | 25 | 36.9 | 33.0
0 | 30 | 38.7 | 33.2
0 | 35 | 39.9 | 34.5
0 | 40 | 40.8 | 35.5
0 | 45 | 42.6 | 36.4
0 | 50 | 44.9 | 37.8
0 | 55 | 45.4 | 38.5
0 | 58 | 47.7 | 39.6
0 | 60 | 45.9 | 40.2
0 | 62 | 47.6 | 40.9
0 | 64 | 51.4 | 41.2
0 | 66 | 38.2 | 39.8
0 | 68 | 39.7 | 40.3
0 | 70 | 38.1 | 40.9
(22 rows)
Here you can see that as numsnaps increases, the test becomes slower,
but then it becomes faster again at 64-66, when it switches to the hash
table. So 64 seems too much. 32 seems to be the sweet spot here, that's
where scanning the hash and scanning the array are about the same speed.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-01-25 17:14 Robert Haas <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Robert Haas @ 2021-01-25 17:14 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers
On Thu, Jan 21, 2021 at 5:14 AM Heikki Linnakangas <[email protected]> wrote:
> Here you can see that as numsnaps increases, the test becomes slower,
> but then it becomes faster again at 64-66, when it switches to the hash
> table. So 64 seems too much. 32 seems to be the sweet spot here, that's
> where scanning the hash and scanning the array are about the same speed.
That sounds good. I mean, it could be that not all hardware behaves
the same here. But trying to get it into the right ballpark makes
sense.
I also like the fact that this now has some cases where it wins by a
significant margin. That's pretty cool; thanks for working on it!
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-03-08 16:47 Ibrar Ahmed <[email protected]>
parent: Robert Haas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Ibrar Ahmed @ 2021-03-08 16:47 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Michael Paquier <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On Mon, Jan 25, 2021 at 10:15 PM Robert Haas <[email protected]> wrote:
> On Thu, Jan 21, 2021 at 5:14 AM Heikki Linnakangas <[email protected]>
> wrote:
> > Here you can see that as numsnaps increases, the test becomes slower,
> > but then it becomes faster again at 64-66, when it switches to the hash
> > table. So 64 seems too much. 32 seems to be the sweet spot here, that's
> > where scanning the hash and scanning the array are about the same speed.
>
> That sounds good. I mean, it could be that not all hardware behaves
> the same here. But trying to get it into the right ballpark makes
> sense.
>
> I also like the fact that this now has some cases where it wins by a
> significant margin. That's pretty cool; thanks for working on it!
>
> --
> Robert Haas
> EDB: http://www.enterprisedb.com
>
>
>
The patchset does not apply successfully, there are some hunk failures.
http://cfbot.cputube.org/patch_32_2834.log
v6-0002-Make-resowners-more-easily-extensible.patch
1 out of 6 hunks FAILED -- saving rejects to file
src/backend/utils/cache/plancache.c.rej
2 out of 15 hunks FAILED -- saving rejects to file
src/backend/utils/resowner/resowner.c.rej
Can we get a rebase?
I am marking the patch "Waiting on Author"
--
Ibrar Ahmed
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-03-09 12:40 Heikki Linnakangas <[email protected]>
parent: Ibrar Ahmed <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-03-09 12:40 UTC (permalink / raw)
To: Ibrar Ahmed <[email protected]>; +Cc: Michael Paquier <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On 08/03/2021 18:47, Ibrar Ahmed wrote:
> The patchset does not apply successfully, there are some hunk failures.
>
> http://cfbot.cputube.org/patch_32_2834.log
> <http://cfbot.cputube.org/patch_32_2834.log;
>
> v6-0002-Make-resowners-more-easily-extensible.patch
>
> 1 out of 6 hunks FAILED -- saving rejects to file
> src/backend/utils/cache/plancache.c.rej
> 2 out of 15 hunks FAILED -- saving rejects to file
> src/backend/utils/resowner/resowner.c.rej
>
>
> Can we get a rebase?
Here you go.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-07-14 12:14 vignesh C <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: vignesh C @ 2021-07-14 12:14 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Ibrar Ahmed <[email protected]>; Michael Paquier <[email protected]>; Alvaro Herrera <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On Tue, Mar 9, 2021 at 6:10 PM Heikki Linnakangas <[email protected]> wrote:
>
> On 08/03/2021 18:47, Ibrar Ahmed wrote:
> > The patchset does not apply successfully, there are some hunk failures.
> >
> > http://cfbot.cputube.org/patch_32_2834.log
> > <http://cfbot.cputube.org/patch_32_2834.log;
> >
> > v6-0002-Make-resowners-more-easily-extensible.patch
> >
> > 1 out of 6 hunks FAILED -- saving rejects to file
> > src/backend/utils/cache/plancache.c.rej
> > 2 out of 15 hunks FAILED -- saving rejects to file
> > src/backend/utils/resowner/resowner.c.rej
> >
> >
> > Can we get a rebase?
>
> Here you go.
The patch does not apply on Head anymore, could you rebase and post a
patch. I'm changing the status to "Waiting for Author".
Regards,
Vignesh
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-07-14 14:07 Alvaro Herrera <[email protected]>
parent: vignesh C <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Alvaro Herrera @ 2021-07-14 14:07 UTC (permalink / raw)
To: vignesh C <[email protected]>; +Cc: Heikki Linnakangas <[email protected]>; Ibrar Ahmed <[email protected]>; Michael Paquier <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On 2021-Jul-14, vignesh C wrote:
> On Tue, Mar 9, 2021 at 6:10 PM Heikki Linnakangas <[email protected]> wrote:
> > Here you go.
>
> The patch does not apply on Head anymore, could you rebase and post a
> patch. I'm changing the status to "Waiting for Author".
Support for hmac was added by e6bdfd9700eb so the rebase is not trivial.
--
Álvaro Herrera Valdivia, Chile — https://www.EnterpriseDB.com/
"Ellos andaban todos desnudos como su madre los parió, y también las mujeres,
aunque no vi más que una, harto moza, y todos los que yo vi eran todos
mancebos, que ninguno vi de edad de más de XXX años" (Cristóbal Colón)
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-07-14 14:40 Heikki Linnakangas <[email protected]>
parent: Alvaro Herrera <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-07-14 14:40 UTC (permalink / raw)
To: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; +Cc: Ibrar Ahmed <[email protected]>; Michael Paquier <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On 14/07/2021 17:07, Alvaro Herrera wrote:
> On 2021-Jul-14, vignesh C wrote:
>
>> On Tue, Mar 9, 2021 at 6:10 PM Heikki Linnakangas <[email protected]> wrote:
>
>>> Here you go.
>>
>> The patch does not apply on Head anymore, could you rebase and post a
>> patch. I'm changing the status to "Waiting for Author".
>
> Support for hmac was added by e6bdfd9700eb so the rebase is not trivial.
Yeah, needed some manual fixing, but here you go.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-07-14 15:18 Zhihong Yu <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Zhihong Yu @ 2021-07-14 15:18 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Ibrar Ahmed <[email protected]>; Michael Paquier <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On Wed, Jul 14, 2021 at 7:40 AM Heikki Linnakangas <[email protected]> wrote:
> On 14/07/2021 17:07, Alvaro Herrera wrote:
> > On 2021-Jul-14, vignesh C wrote:
> >
> >> On Tue, Mar 9, 2021 at 6:10 PM Heikki Linnakangas <[email protected]>
> wrote:
> >
> >>> Here you go.
> >>
> >> The patch does not apply on Head anymore, could you rebase and post a
> >> patch. I'm changing the status to "Waiting for Author".
> >
> > Support for hmac was added by e6bdfd9700eb so the rebase is not trivial.
>
> Yeah, needed some manual fixing, but here you go.
>
> - Heikki
>
Hi,
For the loop over the hash:
+ for (int idx = 0; idx < capacity; idx++)
{
- if (olditemsarr[i] != resarr->invalidval)
- ResourceArrayAdd(resarr, olditemsarr[i]);
+ while (owner->hash[idx].kind != NULL &&
+ owner->hash[idx].kind->phase == phase)
...
+ } while (capacity != owner->capacity);
Since the phase variable doesn't seem to change for the while loop, I
wonder what benefit the while loop has (since the release is governed by
phase).
Cheers
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-07-14 15:26 Heikki Linnakangas <[email protected]>
parent: Zhihong Yu <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-07-14 15:26 UTC (permalink / raw)
To: Zhihong Yu <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Ibrar Ahmed <[email protected]>; Michael Paquier <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
Thanks for having a look!
On 14/07/2021 18:18, Zhihong Yu wrote:
> For the loop over the hash:
>
> + for (int idx = 0; idx < capacity; idx++)
> {
> - if (olditemsarr[i] != resarr->invalidval)
> - ResourceArrayAdd(resarr, olditemsarr[i]);
> + while (owner->hash[idx].kind != NULL &&
> + owner->hash[idx].kind->phase == phase)
> ...
> + } while (capacity != owner->capacity);
>
> Since the phase variable doesn't seem to change for the while loop, I
> wonder what benefit the while loop has (since the release is governed by
> phase).
Hmm, the phase variable doesn't change, but could the element at
'owner->hash[idx]' change? I'm not sure about that. The loop is supposed
to handle the case that the hash table grows; could that replace the
element at 'owner->hash[idx]' with something else, with different phase?
The check is very cheap, so I'm inclined to keep it to be sure.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-07-14 15:41 Zhihong Yu <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Zhihong Yu @ 2021-07-14 15:41 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Alvaro Herrera <[email protected]>; vignesh C <[email protected]>; Ibrar Ahmed <[email protected]>; Michael Paquier <[email protected]>; [email protected] <[email protected]>; pgsql-hackers; Robert Haas <[email protected]>
On Wed, Jul 14, 2021 at 8:26 AM Heikki Linnakangas <[email protected]> wrote:
> Thanks for having a look!
>
> On 14/07/2021 18:18, Zhihong Yu wrote:
> > For the loop over the hash:
> >
> > + for (int idx = 0; idx < capacity; idx++)
> > {
> > - if (olditemsarr[i] != resarr->invalidval)
> > - ResourceArrayAdd(resarr, olditemsarr[i]);
> > + while (owner->hash[idx].kind != NULL &&
> > + owner->hash[idx].kind->phase == phase)
> > ...
> > + } while (capacity != owner->capacity);
> >
> > Since the phase variable doesn't seem to change for the while loop, I
> > wonder what benefit the while loop has (since the release is governed by
> > phase).
>
> Hmm, the phase variable doesn't change, but could the element at
> 'owner->hash[idx]' change? I'm not sure about that. The loop is supposed
> to handle the case that the hash table grows; could that replace the
> element at 'owner->hash[idx]' with something else, with different phase?
> The check is very cheap, so I'm inclined to keep it to be sure.
>
> - Heikki
>
Hi,
Agreed that ```owner->hash[idx].kind->phase == phase``` can be kept.
I just wonder if we should put limit on the number of iterations for the
while loop.
Maybe add a bool variable indicating that kind->ReleaseResource(value) is
called in the inner loop.
If there is no releasing when the inner loop finishes, we can come out of
the while loop.
Cheers
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-09-08 10:19 Aleksander Alekseev <[email protected]>
parent: Zhihong Yu <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Aleksander Alekseev @ 2021-09-08 10:19 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Heikki Linnakangas <[email protected]>
Hi Heikki,
> Yeah, needed some manual fixing, but here you go.
Thanks for working on this!
v8-0002 didn't apply to the current master, so I rebased it. See
attached v9-* patches. I also included v9-0004 with some minor tweaks
from me. I have several notes regarding the code.
1. Not sure if I understand this code of ResourceOwnerReleaseAll():
```
/* First handle all the entries in the array. */
do
{
found = false;
for (int i = 0; i < owner->narr; i++)
{
if (owner->arr[i].kind->phase == phase)
{
Datum value = owner->arr[i].item;
ResourceOwnerFuncs *kind = owner->arr[i].kind;
if (printLeakWarnings)
kind->PrintLeakWarning(value);
kind->ReleaseResource(value);
found = true;
}
}
/*
* If any resources were released, check again because some of the
* elements might have been moved by the callbacks. We don't want to
* miss them.
*/
} while (found && owner->narr > 0);
```
Shouldn't we mark the resource as released and/or decrease narr and/or
save the last processed i? Why this will not call ReleaseResource()
endlessly on the same resource (array item)? Same question for the
following code that iterates over the hash table.
2. Just an idea/observation. It's possible that the performance of
ResourceOwnerEnlarge() can be slightly improved. Since the size of the
hash table is always a power of 2 and the code always doubles the size
of the hash table, (idx & mask) value will get one extra bit, which
can be 0 or 1. If it's 0, the value is already in its place,
otherwise, it should be moved on the known distance. In other words,
it's possible to copy `oldhash` to `newhash` and then move only half
of the items. I don't claim that this code necessarily should be
faster, or that this should be checked in the scope of this work.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v9-0003-Optimize-hash-function.patch (2.3K, ../../CAJ7c6TP8v=2Ym5S1C4YenaCMgwu5FrFTGi40Nn5TyDS4ig-gRA@mail.gmail.com/2-v9-0003-Optimize-hash-function.patch)
download | inline diff:
From 3a92705934fc91cd9f0389e08df1fd90e4f862ee Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Thu, 21 Jan 2021 00:06:58 +0200
Subject: [PATCH v8 3/3] Optimize hash function
---
src/backend/utils/resowner/resowner.c | 24 ++++++++++++++++++------
src/include/common/hashfn.h | 15 +++++++++++++++
2 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index d62b1495dd3..6189d051072 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -163,15 +163,27 @@ static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
* INTERNAL ROUTINES *
*****************************************************************************/
+/*
+ * Hash function for value+kind combination.
+ */
static inline uint32
hash_resource_elem(Datum value, ResourceOwnerFuncs *kind)
{
- Datum data[2];
-
- data[0] = value;
- data[1] = PointerGetDatum(kind);
-
- return hash_bytes((unsigned char *) &data, 2 * SIZEOF_DATUM);
+ /*
+ * Most resources types store a pointer in 'value', and pointers are
+ * unique all on their own. But some resources store plain integers (Files
+ * and Buffers as of this writing), so we want to incorporate the 'kind'
+ * in the hash too, otherwise those resources will collide a lot. But
+ * because there are only a few resource kinds like that - and only a few
+ * resource kinds to begin with - we don't need to work too hard to mix
+ * 'kind' into the hash. Just add it with hash_combine(), it perturbs the
+ * result enough for our purposes.
+ */
+#if SIZEOF_DATUM == 8
+ return hash_combine64(murmurhash64((uint64) value), (uint64) kind);
+#else
+ return hash_combine(murmurhash32((uint32) value), (uint32) kind);
+#endif
}
static void
diff --git a/src/include/common/hashfn.h b/src/include/common/hashfn.h
index c634cc067a1..009ffbbdd38 100644
--- a/src/include/common/hashfn.h
+++ b/src/include/common/hashfn.h
@@ -101,4 +101,19 @@ murmurhash32(uint32 data)
return h;
}
+/* 64-bit variant */
+static inline uint64
+murmurhash64(uint64 data)
+{
+ uint64 h = data;
+
+ h ^= h >> 33;
+ h *= 0xff51afd7ed558ccd;
+ h ^= h >> 33;
+ h *= 0xc4ceb9fe1a85ec53;
+ h ^= h >> 33;
+
+ return h;
+}
+
#endif /* HASHFN_H */
--
2.30.2
[application/octet-stream] v9-0001-Move-a-few-ResourceOwnerEnlarge-calls-for-safety.patch (8.6K, ../../CAJ7c6TP8v=2Ym5S1C4YenaCMgwu5FrFTGi40Nn5TyDS4ig-gRA@mail.gmail.com/3-v9-0001-Move-a-few-ResourceOwnerEnlarge-calls-for-safety.patch)
download | inline diff:
From 19fb4369e3b7bfb29afc6cfe597af5a89698dd3f Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Wed, 13 Jan 2021 12:21:28 +0200
Subject: [PATCH v8 1/3] Move a few ResourceOwnerEnlarge() calls for safety and
clarity.
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 86ef607ff38..2383e4cac08 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -810,9 +810,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1165,9 +1162,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1668,8 +1667,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1680,6 +1677,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1760,7 +1759,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1936,9 +1936,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2412,9 +2409,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2496,8 +2490,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2507,7 +2499,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3559,9 +3553,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3575,7 +3566,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3632,9 +3625,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3671,7 +3661,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3711,9 +3703,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3727,7 +3716,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 4fbdc62d8c7..13eed587601 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1609,8 +1609,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1698,13 +1696,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.2
[application/octet-stream] v9-0002-Make-resowners-more-easily-extensible.patch (90.1K, ../../CAJ7c6TP8v=2Ym5S1C4YenaCMgwu5FrFTGi40Nn5TyDS4ig-gRA@mail.gmail.com/4-v9-0002-Make-resowners-more-easily-extensible.patch)
download | inline diff:
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 4c63bd4dc6..f2d4442c93 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -30,9 +30,27 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
+/* ResourceOwner callbacks to hold tupledesc references */
+static void ResOwnerReleaseTupleDesc(Datum res);
+static void ResOwnerPrintTupleDescLeakWarning(Datum res);
+
+static ResourceOwnerFuncs tupdesc_resowner_funcs =
+{
+ /* relcache references */
+ .name = "tupdesc reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseTupleDesc,
+ .PrintLeakWarning = ResOwnerPrintTupleDescLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberTupleDesc(owner, tupdesc) \
+ ResourceOwnerRemember(owner, PointerGetDatum(tupdesc), &tupdesc_resowner_funcs)
+#define ResourceOwnerForgetTupleDesc(owner, tupdesc) \
+ ResourceOwnerForget(owner, PointerGetDatum(tupdesc), &tupdesc_resowner_funcs)
/*
* CreateTemplateTupleDesc
@@ -367,7 +385,7 @@ IncrTupleDescRefCount(TupleDesc tupdesc)
{
Assert(tupdesc->tdrefcount >= 0);
- ResourceOwnerEnlargeTupleDescs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
tupdesc->tdrefcount++;
ResourceOwnerRememberTupleDesc(CurrentResourceOwner, tupdesc);
}
@@ -910,3 +928,23 @@ BuildDescFromLists(List *names, List *types, List *typmods, List *collations)
return desc;
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseTupleDesc(Datum res)
+{
+ DecrTupleDescRefCount((TupleDesc) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintTupleDescLeakWarning(Datum res)
+{
+ TupleDesc tupdesc = (TupleDesc) DatumGetPointer(res);
+
+ elog(WARNING,
+ "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
+ tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
+}
diff --git a/src/backend/jit/jit.c b/src/backend/jit/jit.c
index 91b8ae6c51..d42d1bc6c4 100644
--- a/src/backend/jit/jit.c
+++ b/src/backend/jit/jit.c
@@ -26,7 +26,6 @@
#include "jit/jit.h"
#include "miscadmin.h"
#include "utils/fmgrprotos.h"
-#include "utils/resowner_private.h"
/* GUCs */
bool jit_enabled = true;
@@ -140,7 +139,6 @@ jit_release_context(JitContext *context)
if (provider_successfully_loaded)
provider.release_context(context);
- ResourceOwnerForgetJIT(context->resowner, PointerGetDatum(context));
pfree(context);
}
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index df691cbf1c..57e3144fae 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -40,7 +40,7 @@
#include "portability/instr_time.h"
#include "storage/ipc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Handle of a module emitted via ORC JIT */
typedef struct LLVMJitHandle
@@ -121,8 +121,26 @@ static LLVMOrcLLJITRef llvm_create_jit_instance(LLVMTargetMachineRef tm);
static char *llvm_error_message(LLVMErrorRef error);
#endif /* LLVM_VERSION_MAJOR > 11 */
-PG_MODULE_MAGIC;
+/* ResourceOwner callbacks to hold JitContexts */
+static void ResOwnerReleaseJitContext(Datum res);
+static void ResOwnerPrintJitContextLeakWarning(Datum res);
+
+static ResourceOwnerFuncs jit_resowner_funcs =
+{
+ /* relcache references */
+ .name = "LLVM JIT context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseJitContext,
+ .PrintLeakWarning = ResOwnerPrintJitContextLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberJIT(owner, handle) \
+ ResourceOwnerRemember(owner, PointerGetDatum(handle), &jit_resowner_funcs)
+#define ResourceOwnerForgetJIT(owner, handle) \
+ ResourceOwnerForget(owner, PointerGetDatum(handle), &jit_resowner_funcs)
+PG_MODULE_MAGIC;
/*
* Initialize LLVM JIT provider.
@@ -151,7 +169,7 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
- ResourceOwnerEnlargeJIT(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
context = MemoryContextAllocZero(TopMemoryContext,
sizeof(LLVMJitContext));
@@ -159,7 +177,7 @@ llvm_create_context(int jitFlags)
/* ensure cleanup */
context->base.resowner = CurrentResourceOwner;
- ResourceOwnerRememberJIT(CurrentResourceOwner, PointerGetDatum(context));
+ ResourceOwnerRememberJIT(CurrentResourceOwner, context);
return context;
}
@@ -221,6 +239,8 @@ llvm_release_context(JitContext *context)
pfree(jit_handle);
}
+
+ ResourceOwnerForgetJIT(context->resowner, context);
}
/*
@@ -1234,3 +1254,21 @@ llvm_error_message(LLVMErrorRef error)
}
#endif /* LLVM_VERSION_MAJOR > 11 */
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseJitContext(Datum res)
+{
+ jit_release_context((JitContext *) PointerGetDatum(res));
+}
+
+static void
+ResOwnerPrintJitContextLeakWarning(Datum res)
+{
+ /* XXX: We used to not print these. Was that intentional? */
+ JitContext *context = (JitContext *) PointerGetDatum(res);
+
+ elog(WARNING, "JIT context leak: context %p still referenced", context);
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 983e67029e..8772151ebc 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,7 +52,7 @@
#include "utils/memdebug.h"
#include "utils/ps_status.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/timestamp.h"
@@ -206,6 +206,18 @@ static PrivateRefCountEntry *GetPrivateRefCountEntry(Buffer buffer, bool do_move
static inline int32 GetPrivateRefCount(Buffer buffer);
static void ForgetPrivateRefCountEntry(PrivateRefCountEntry *ref);
+/* ResourceOwner callbacks to hold buffer pins */
+static void ResOwnerReleaseBuffer(Datum res);
+static void ResOwnerPrintBufferLeakWarning(Datum res);
+
+ResourceOwnerFuncs buffer_resowner_funcs =
+{
+ .name = "buffer",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseBuffer,
+ .PrintLeakWarning = ResOwnerPrintBufferLeakWarning
+};
+
/*
* Ensure that the PrivateRefCountArray has sufficient space to store one more
* entry. This has to be called before using NewPrivateRefCountEntry() to fill
@@ -625,7 +637,7 @@ ReadRecentBuffer(RelFileNode rnode, ForkNumber forkNum, BlockNumber blockNum,
Assert(BufferIsValid(recent_buffer));
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ReservePrivateRefCountEntry();
INIT_BUFFERTAG(tag, rnode, forkNum, blockNum);
@@ -1166,7 +1178,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* pin.
*/
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1677,7 +1689,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ref = GetPrivateRefCountEntry(b, true);
@@ -1760,7 +1772,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
*
* As this function is called with the spinlock held, the caller has to
* previously call ReservePrivateRefCountEntry() and
- * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ * ResourceOwnerEnlarge(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -2501,7 +2513,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3555,7 +3567,7 @@ FlushRelationBuffers(Relation rel)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3650,7 +3662,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3705,7 +3717,7 @@ FlushDatabaseBuffers(Oid dbid)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
@@ -3788,7 +3800,7 @@ void
IncrBufferRefCount(Buffer buffer)
{
Assert(BufferIsPinned(buffer));
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
if (BufferIsLocal(buffer))
LocalRefCount[-buffer - 1]++;
else
@@ -4835,3 +4847,18 @@ TestForOldSnapshot_impl(Snapshot snapshot, Relation relation)
(errcode(ERRCODE_SNAPSHOT_TOO_OLD),
errmsg("snapshot too old")));
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseBuffer(Datum res)
+{
+ ReleaseBuffer(DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintBufferLeakWarning(Datum res)
+{
+ PrintBufferLeakWarning(DatumGetInt32(res));
+}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f7c15ea8a4..3f22d7127b 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -22,7 +22,7 @@
#include "storage/bufmgr.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/*#define LBDEBUG*/
@@ -124,7 +124,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
InitLocalBuffers();
/* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index f9cda6906d..be5576451d 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -99,7 +99,7 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Define PG_FLUSH_DATA_WORKS if we have an implementation for pg_flush_data */
#if defined(HAVE_SYNC_FILE_RANGE)
@@ -350,6 +350,24 @@ static void unlink_if_exists_fname(const char *fname, bool isdir, int elevel);
static int fsync_parent_path(const char *fname, int elevel);
+/* ResourceOwner callbacks to hold virtual file descriptors */
+static void ResOwnerReleaseFile(Datum res);
+static void ResOwnerPrintFileLeakWarning(Datum res);
+
+static ResourceOwnerFuncs file_resowner_funcs =
+{
+ .name = "File",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseFile,
+ .PrintLeakWarning = ResOwnerPrintFileLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberFile(owner, file) \
+ ResourceOwnerRemember(owner, Int32GetDatum(file), &file_resowner_funcs)
+#define ResourceOwnerForgetFile(owner, file) \
+ ResourceOwnerForget(owner, Int32GetDatum(file), &file_resowner_funcs)
+
/*
* pg_fsync --- do fsync with or without writethrough
*/
@@ -1529,7 +1547,7 @@ ReportTemporaryFileUsage(const char *path, off_t size)
/*
* Called to register a temporary file for automatic close.
- * ResourceOwnerEnlargeFiles(CurrentResourceOwner) must have been called
+ * ResourceOwnerEnlarge(CurrentResourceOwner) must have been called
* before the file was opened.
*/
static void
@@ -1713,7 +1731,7 @@ OpenTemporaryFile(bool interXact)
* open it, if we'll be registering it below.
*/
if (!interXact)
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* If some temp tablespace(s) have been given to us, try to use the next
@@ -1845,7 +1863,7 @@ PathNameCreateTemporaryFile(const char *path, bool error_on_failure)
Assert(temporary_files_allowed); /* check temp file access is up */
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Open the file. Note: we don't use O_EXCL, in case there is an orphaned
@@ -1885,7 +1903,7 @@ PathNameOpenTemporaryFile(const char *path, int mode)
Assert(temporary_files_allowed); /* check temp file access is up */
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
file = PathNameOpenFile(path, mode | PG_BINARY);
@@ -3870,3 +3888,19 @@ pg_pwritev_with_retry(int fd, const struct iovec *iov, int iovcnt, off_t offset)
return sum;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseFile(Datum res)
+{
+ FileClose((File) DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintFileLeakWarning(Datum res)
+{
+ elog(WARNING, "temporary file leak: File %d still referenced",
+ DatumGetInt32(res));
+}
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index b461a5f7e9..77b6e58fd9 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -37,13 +37,15 @@
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "storage/dsm.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
+#include "storage/shmem.h"
#include "utils/freepage.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#define PG_DYNSHMEM_CONTROL_MAGIC 0x9a503d32
@@ -139,6 +141,25 @@ static dsm_control_header *dsm_control;
static Size dsm_control_mapped_size = 0;
static void *dsm_control_impl_private = NULL;
+
+/* ResourceOwner callbacks to hold DSM segments */
+static void ResOwnerReleaseDSM(Datum res);
+static void ResOwnerPrintDSMLeakWarning(Datum res);
+
+static ResourceOwnerFuncs dsm_resowner_funcs =
+{
+ .name = "dynamic shared memory segment",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseDSM,
+ .PrintLeakWarning = ResOwnerPrintDSMLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberDSM(owner, seg) \
+ ResourceOwnerRemember(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+#define ResourceOwnerForgetDSM(owner, seg) \
+ ResourceOwnerForget(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+
/*
* Start up the dynamic shared memory system.
*
@@ -900,7 +921,7 @@ void
dsm_unpin_mapping(dsm_segment *seg)
{
Assert(seg->resowner == NULL);
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg->resowner = CurrentResourceOwner;
ResourceOwnerRememberDSM(seg->resowner, seg);
}
@@ -1167,7 +1188,7 @@ dsm_create_descriptor(void)
dsm_segment *seg;
if (CurrentResourceOwner)
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg = MemoryContextAlloc(TopMemoryContext, sizeof(dsm_segment));
dlist_push_head(&dsm_segment_list, &seg->node);
@@ -1246,3 +1267,20 @@ is_main_region_dsm_handle(dsm_handle handle)
{
return handle & 1;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseDSM(Datum res)
+{
+ dsm_detach((dsm_segment *) DatumGetPointer(res));
+}
+static void
+ResOwnerPrintDSMLeakWarning(Datum res)
+{
+ dsm_segment *seg = (dsm_segment *) res;
+
+ elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
+ dsm_segment_handle(seg));
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 364654e106..933f9121be 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -48,7 +48,7 @@
#include "storage/standby.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* This configuration variable is used to set the lock table size */
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 13eed58760..ed046db57e 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -31,12 +31,13 @@
#endif
#include "storage/lmgr.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
@@ -104,6 +105,42 @@ static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
* internal support functions
*/
+/* ResourceOwner callbacks to hold catcache references */
+
+static void ResOwnerReleaseCatCache(Datum res);
+static void ResOwnerPrintCatCacheLeakWarning(Datum res);
+static void ResOwnerReleaseCatCacheList(Datum res);
+static void ResOwnerPrintCatCacheListLeakWarning(Datum res);
+
+static ResourceOwnerFuncs catcache_resowner_funcs =
+{
+ /* catcache references */
+ .name = "catcache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCache,
+ .PrintLeakWarning = ResOwnerPrintCatCacheLeakWarning
+};
+
+static ResourceOwnerFuncs catlistref_resowner_funcs =
+{
+ /* catcache-list pins */
+ .name = "catcache list reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCacheList,
+ .PrintLeakWarning = ResOwnerPrintCatCacheListLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberCatCacheRef(owner, tuple) \
+ ResourceOwnerRemember(owner, PointerGetDatum(tuple), &catcache_resowner_funcs)
+#define ResourceOwnerForgetCatCacheRef(owner, tuple) \
+ ResourceOwnerForget(owner, PointerGetDatum(tuple), &catcache_resowner_funcs)
+#define ResourceOwnerRememberCatCacheListRef(owner, list) \
+ ResourceOwnerRemember(owner, PointerGetDatum(list), &catlistref_resowner_funcs)
+#define ResourceOwnerForgetCatCacheListRef(owner, list) \
+ ResourceOwnerForget(owner, PointerGetDatum(list), &catlistref_resowner_funcs)
+
+
/*
* Hash and equality functions for system types that are used as cache key
* fields. In some cases, we just call the regular SQL-callable functions for
@@ -1272,7 +1309,7 @@ SearchCatCacheInternal(CatCache *cache,
*/
if (!ct->negative)
{
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
@@ -1373,7 +1410,7 @@ SearchCatCacheMiss(CatCache *cache,
hashValue, hashIndex,
false);
/* immediately set the refcount to 1 */
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
break; /* assume only one match */
@@ -1585,7 +1622,7 @@ SearchCatCacheList(CatCache *cache,
dlist_move_head(&cache->cc_lists, &cl->cache_elem);
/* Bump the list's refcount and return it */
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
cl->refcount++;
ResourceOwnerRememberCatCacheListRef(CurrentResourceOwner, cl);
@@ -1697,7 +1734,7 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
/* Make sure the resource owner has room to remember this entry. */
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
@@ -2071,14 +2108,19 @@ PrepareToInvalidateCacheTuple(Relation relation,
}
}
-
/*
- * Subroutines for warning about reference leaks. These are exported so
- * that resowner.c can call them.
+ * ResourceOwner callbacks
*/
-void
-PrintCatCacheLeakWarning(HeapTuple tuple)
+static void
+ResOwnerReleaseCatCache(Datum res)
+{
+ ReleaseCatCache((HeapTuple) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheLeakWarning(Datum res)
{
+ HeapTuple tuple = (HeapTuple) DatumGetPointer(res);
CatCTup *ct = (CatCTup *) (((char *) tuple) -
offsetof(CatCTup, tuple));
@@ -2092,9 +2134,17 @@ PrintCatCacheLeakWarning(HeapTuple tuple)
ct->refcount);
}
-void
-PrintCatCacheListLeakWarning(CatCList *list)
+static void
+ResOwnerReleaseCatCacheList(Datum res)
{
+ ReleaseCatCacheList((CatCList *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheListLeakWarning(Datum res)
+{
+ CatCList *list = (CatCList *) DatumGetPointer(res);
+
elog(WARNING, "cache reference leak: cache %s (%d), list %p has count %d",
list->my_cache->cc_relname, list->my_cache->id,
list, list->refcount);
diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c
index 6767eae8f2..a10362629f 100644
--- a/src/backend/utils/cache/plancache.c
+++ b/src/backend/utils/cache/plancache.c
@@ -69,7 +69,7 @@
#include "tcop/utility.h"
#include "utils/inval.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -115,6 +115,26 @@ static void PlanCacheRelCallback(Datum arg, Oid relid);
static void PlanCacheObjectCallback(Datum arg, int cacheid, uint32 hashvalue);
static void PlanCacheSysCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* ResourceOwner callbacks to track plancache references */
+static void ResOwnerReleaseCachedPlan(Datum res);
+static void ResOwnerPrintPlanCacheLeakWarning(Datum res);
+
+/* this is exported for ResourceOwnerReleaseAllPlanCacheRefs() */
+ResourceOwnerFuncs planref_resowner_funcs =
+{
+ .name = "plancache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCachedPlan,
+ .PrintLeakWarning = ResOwnerPrintPlanCacheLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberPlanCacheRef(owner, plan) \
+ ResourceOwnerRemember(owner, PointerGetDatum(plan), &planref_resowner_funcs)
+#define ResourceOwnerForgetPlanCacheRef(owner, plan) \
+ ResourceOwnerForget(owner, PointerGetDatum(plan), &planref_resowner_funcs)
+
+
/* GUC parameter */
int plan_cache_mode;
@@ -1229,7 +1249,7 @@ GetCachedPlan(CachedPlanSource *plansource, ParamListInfo boundParams,
/* Flag the plan as in use by caller */
if (owner)
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
if (owner)
ResourceOwnerRememberPlanCacheRef(owner, plan);
@@ -1392,7 +1412,7 @@ CachedPlanAllowsSimpleValidityCheck(CachedPlanSource *plansource,
/* Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -1451,7 +1471,7 @@ CachedPlanIsSimplyValid(CachedPlanSource *plansource, CachedPlan *plan,
/* It's still good. Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -2205,3 +2225,20 @@ ResetPlanCache(void)
cexpr->is_valid = false;
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+
+static void
+ResOwnerReleaseCachedPlan(Datum res)
+{
+ ReleaseCachedPlan((CachedPlan *) DatumGetPointer(res), CurrentResourceOwner);
+}
+
+static void
+ResOwnerPrintPlanCacheLeakWarning(Datum res)
+{
+ elog(WARNING, "plancache reference leak: plan %p not closed",
+ DatumGetPointer(res));
+}
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 13d9994af3..90d2892489 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -77,13 +77,14 @@
#include "storage/smgr.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/relmapper.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -2056,6 +2057,24 @@ RelationIdGetRelation(Oid relationId)
* ----------------------------------------------------------------
*/
+/* ResourceOwner callbacks to track relcache references */
+static void ResOwnerReleaseRelation(Datum res);
+static void ResOwnerPrintRelCacheLeakWarning(Datum res);
+
+static ResourceOwnerFuncs relref_resowner_funcs =
+{
+ .name = "relcache reference",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseRelation,
+ .PrintLeakWarning = ResOwnerPrintRelCacheLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberRelationRef(owner, rel) \
+ ResourceOwnerRemember(owner, PointerGetDatum(rel), &relref_resowner_funcs)
+#define ResourceOwnerForgetRelationRef(owner, rel) \
+ ResourceOwnerForget(owner, PointerGetDatum(rel), &relref_resowner_funcs)
+
/*
* RelationIncrementReferenceCount
* Increments relation reference count.
@@ -2067,7 +2086,7 @@ RelationIdGetRelation(Oid relationId)
void
RelationIncrementReferenceCount(Relation rel)
{
- ResourceOwnerEnlargeRelationRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
rel->rd_refcnt += 1;
if (!IsBootstrapProcessingMode())
ResourceOwnerRememberRelationRef(CurrentResourceOwner, rel);
@@ -6546,3 +6565,21 @@ unlink_initfile(const char *initfilename, int elevel)
initfilename)));
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerPrintRelCacheLeakWarning(Datum res)
+{
+ Relation rel = (Relation) res;
+
+ elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
+ RelationGetRelationName(rel));
+}
+
+static void
+ResOwnerReleaseRelation(Datum res)
+{
+ RelationClose((Relation) res);
+}
diff --git a/src/backend/utils/resowner/README b/src/backend/utils/resowner/README
index 2998f6bb36..4ed13afa10 100644
--- a/src/backend/utils/resowner/README
+++ b/src/backend/utils/resowner/README
@@ -60,13 +60,18 @@ subtransaction or portal. Therefore, the "release" operation on a child
ResourceOwner transfers lock ownership to the parent instead of actually
releasing the lock, if isCommit is true.
-Currently, ResourceOwners contain direct support for recording ownership of
-buffer pins, lmgr locks, and catcache, relcache, plancache, tupdesc, and
-snapshot references. Other objects can be associated with a ResourceOwner by
-recording the address of the owning ResourceOwner in such an object. There is
-an API for other modules to get control during ResourceOwner release, so that
-they can scan their own data structures to find the objects that need to be
-deleted.
+ResourceOwner can record ownership of many different kinds of resources.
+As of this writing, it's used internally for buffer pins, lmgr locks, and
+catcache, relcache, plancache, tupdesc, snapshot, DSM and JIT context
+references. ResourceOwner treats all resources the same, and extensions
+can define new kinds of resources by filling in a ResourceOwnerFuncs
+struct with a suitable callback functions.
+
+There is also an API for other modules to get control during ResourceOwner
+release, so that they can scan their own data structures to find the objects
+that need to be deleted. This used to be the only way to register new kinds
+of objects with a resource owner; nowadays it easier to write custom
+ResourceOwnerFuncs callbacks.
Whenever we are inside a transaction, the global variable
CurrentResourceOwner shows which resource owner should be assigned
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index e24f00f060..d62b1495dd 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -20,76 +20,45 @@
*/
#include "postgres.h"
-#include "common/cryptohash.h"
#include "common/hashfn.h"
-#include "common/hmac.h"
-#include "jit/jit.h"
-#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
#include "utils/memutils.h"
-#include "utils/rel.h"
-#include "utils/resowner_private.h"
-#include "utils/snapmgr.h"
-
-
-/*
- * All resource IDs managed by this code are required to fit into a Datum,
- * which is fine since they are generally pointers or integers.
- *
- * Provide Datum conversion macros for a couple of things that are really
- * just "int".
- */
-#define FileGetDatum(file) Int32GetDatum(file)
-#define DatumGetFile(datum) ((File) DatumGetInt32(datum))
-#define BufferGetDatum(buffer) Int32GetDatum(buffer)
-#define DatumGetBuffer(datum) ((Buffer) DatumGetInt32(datum))
+#include "utils/plancache.h"
+#include "utils/resowner.h"
/*
- * ResourceArray is a common structure for storing all types of resource IDs.
- *
- * We manage small sets of resource IDs by keeping them in a simple array:
- * itemsarr[k] holds an ID, for 0 <= k < nitems <= maxitems = capacity.
- *
- * If a set grows large, we switch over to using open-addressing hashing.
- * Then, itemsarr[] is a hash table of "capacity" slots, with each
- * slot holding either an ID or "invalidval". nitems is the number of valid
- * items present; if it would exceed maxitems, we enlarge the array and
- * re-hash. In this mode, maxitems should be rather less than capacity so
- * that we don't waste too much time searching for empty slots.
+ * ResourceElem represents a reference associated with a resource owner.
*
- * In either mode, lastidx remembers the location of the last item inserted
- * or returned by GetAny; this speeds up searches in ResourceArrayRemove.
+ * All objects managed by this code are required to fit into a Datum,
+ * which is fine since they are generally pointers or integers.
*/
-typedef struct ResourceArray
+typedef struct ResourceElem
{
- Datum *itemsarr; /* buffer for storing values */
- Datum invalidval; /* value that is considered invalid */
- uint32 capacity; /* allocated length of itemsarr[] */
- uint32 nitems; /* how many items are stored in items array */
- uint32 maxitems; /* current limit on nitems before enlarging */
- uint32 lastidx; /* index of last item returned by GetAny */
-} ResourceArray;
+ Datum item;
+ ResourceOwnerFuncs *kind;
+} ResourceElem;
/*
- * Initially allocated size of a ResourceArray. Must be power of two since
- * we'll use (arraysize - 1) as mask for hashing.
+ * Size of the small fixed-size array to hold most-recently remembered resources.
*/
-#define RESARRAY_INIT_SIZE 16
+#define RESOWNER_ARRAY_SIZE 8
/*
- * When to switch to hashing vs. simple array logic in a ResourceArray.
+ * Initially allocated size of a ResourceOwner's hash. Must be power of two since
+ * we'll use (capacity - 1) as mask for hashing.
*/
-#define RESARRAY_MAX_ARRAY 64
-#define RESARRAY_IS_ARRAY(resarr) ((resarr)->capacity <= RESARRAY_MAX_ARRAY)
+#define RESOWNER_HASH_INIT_SIZE 32
/*
- * How many items may be stored in a resource array of given capacity.
+ * How many items may be stored in a hash of given capacity.
* When this number is reached, we must resize.
*/
-#define RESARRAY_MAX_ITEMS(capacity) \
- ((capacity) <= RESARRAY_MAX_ARRAY ? (capacity) : (capacity)/4 * 3)
+#define RESOWNER_HASH_MAX_ITEMS(capacity) ((capacity)/4 * 3)
+
+StaticAssertDecl(RESOWNER_HASH_MAX_ITEMS(RESOWNER_HASH_INIT_SIZE) > RESOWNER_ARRAY_SIZE,
+ "initial hash size too small compared to array size");
/*
* To speed up bulk releasing or reassigning locks from a resource owner to
@@ -119,24 +88,33 @@ typedef struct ResourceOwnerData
ResourceOwner nextchild; /* next child of same parent */
const char *name; /* name (just for debugging) */
- /* We have built-in support for remembering: */
- ResourceArray bufferarr; /* owned buffers */
- ResourceArray catrefarr; /* catcache references */
- ResourceArray catlistrefarr; /* catcache-list pins */
- ResourceArray relrefarr; /* relcache references */
- ResourceArray planrefarr; /* plancache references */
- ResourceArray tupdescarr; /* tupdesc references */
- ResourceArray snapshotarr; /* snapshot references */
- ResourceArray filearr; /* open temporary files */
- ResourceArray dsmarr; /* dynamic shmem segments */
- ResourceArray jitarr; /* JIT contexts */
- ResourceArray cryptohasharr; /* cryptohash contexts */
- ResourceArray hmacarr; /* HMAC contexts */
+ /*
+ * These structs keep track of the objects registered with this owner.
+ *
+ * We manage a small set of references by keeping them in a simple array.
+ * When the array gets full, all the elements in the array are moved to a
+ * hash table. This way, the array always contains a few most recently
+ * remembered references. To find a particular reference, you need to
+ * search both the array and the hash table.
+ */
+ ResourceElem arr[RESOWNER_ARRAY_SIZE];
+ uint32 narr; /* how many items are stored in the array */
+
+ /*
+ * The hash table. Uses open-addressing. 'nhash' is the number of items
+ * present; if it would exceed 'grow_at', we enlarge it and re-hash.
+ * 'grow_at' should be rather less than 'capacity' so that we don't waste
+ * too much time searching for empty slots.
+ */
+ ResourceElem *hash;
+ uint32 nhash; /* how many items are stored in the hash */
+ uint32 capacity; /* allocated length of hash[] */
+ uint32 grow_at; /* grow hash when reach this */
/* We can remember up to MAX_RESOWNER_LOCKS references to local locks. */
int nlocks; /* number of owned locks */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
-} ResourceOwnerData;
+} ResourceOwnerData;
/*****************************************************************************
@@ -148,6 +126,18 @@ ResourceOwner CurTransactionResourceOwner = NULL;
ResourceOwner TopTransactionResourceOwner = NULL;
ResourceOwner AuxProcessResourceOwner = NULL;
+/* #define RESOWNER_STATS */
+/* #define RESOWNER_TRACE */
+
+#ifdef RESOWNER_STATS
+static int narray_lookups = 0;
+static int nhash_lookups = 0;
+#endif
+
+#ifdef RESOWNER_TRACE
+static int resowner_trace_counter = 0;
+#endif
+
/*
* List of add-on callbacks for resource releasing
*/
@@ -162,298 +152,315 @@ static ResourceReleaseCallbackItem *ResourceRelease_callbacks = NULL;
/* Internal routines */
-static void ResourceArrayInit(ResourceArray *resarr, Datum invalidval);
-static void ResourceArrayEnlarge(ResourceArray *resarr);
-static void ResourceArrayAdd(ResourceArray *resarr, Datum value);
-static bool ResourceArrayRemove(ResourceArray *resarr, Datum value);
-static bool ResourceArrayGetAny(ResourceArray *resarr, Datum *value);
-static void ResourceArrayFree(ResourceArray *resarr);
static void ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
-static void PrintRelCacheLeakWarning(Relation rel);
-static void PrintPlanCacheLeakWarning(CachedPlan *plan);
-static void PrintTupleDescLeakWarning(TupleDesc tupdesc);
-static void PrintSnapshotLeakWarning(Snapshot snapshot);
-static void PrintFileLeakWarning(File file);
-static void PrintDSMLeakWarning(dsm_segment *seg);
-static void PrintCryptoHashLeakWarning(Datum handle);
-static void PrintHMACLeakWarning(Datum handle);
/*****************************************************************************
* INTERNAL ROUTINES *
*****************************************************************************/
+static inline uint32
+hash_resource_elem(Datum value, ResourceOwnerFuncs *kind)
+{
+ Datum data[2];
+
+ data[0] = value;
+ data[1] = PointerGetDatum(kind);
+
+ return hash_bytes((unsigned char *) &data, 2 * SIZEOF_DATUM);
+}
-/*
- * Initialize a ResourceArray
- */
static void
-ResourceArrayInit(ResourceArray *resarr, Datum invalidval)
+ResourceOwnerAddToHash(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- /* Assert it's empty */
- Assert(resarr->itemsarr == NULL);
- Assert(resarr->capacity == 0);
- Assert(resarr->nitems == 0);
- Assert(resarr->maxitems == 0);
- /* Remember the appropriate "invalid" value */
- resarr->invalidval = invalidval;
- /* We don't allocate any storage until needed */
+ /* Insert into first free slot at or after hash location. */
+ uint32 mask = owner->capacity - 1;
+ uint32 idx;
+
+ idx = hash_resource_elem(value, kind) & mask;
+ for (;;)
+ {
+ if (owner->hash[idx].kind == NULL)
+ break;
+ idx = (idx + 1) & mask;
+ }
+ owner->hash[idx].item = value;
+ owner->hash[idx].kind = kind;
+ owner->nhash++;
}
/*
- * Make sure there is room for at least one more resource in an array.
- *
- * This is separate from actually inserting a resource because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
+ * Call the ReleaseResource callback on entries with given 'phase'.
*/
static void
-ResourceArrayEnlarge(ResourceArray *resarr)
+ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
+ bool printLeakWarnings)
{
- uint32 i,
- oldcap,
- newcap;
- Datum *olditemsarr;
- Datum *newitemsarr;
-
- if (resarr->nitems < resarr->maxitems)
- return; /* no work needed */
+ bool found;
+ int capacity;
- olditemsarr = resarr->itemsarr;
- oldcap = resarr->capacity;
-
- /* Double the capacity of the array (capacity must stay a power of 2!) */
- newcap = (oldcap > 0) ? oldcap * 2 : RESARRAY_INIT_SIZE;
- newitemsarr = (Datum *) MemoryContextAlloc(TopMemoryContext,
- newcap * sizeof(Datum));
- for (i = 0; i < newcap; i++)
- newitemsarr[i] = resarr->invalidval;
+ /* First handle all the entries in the array. */
+ do
+ {
+ found = false;
+ for (int i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].kind->phase == phase)
+ {
+ Datum value = owner->arr[i].item;
+ ResourceOwnerFuncs *kind = owner->arr[i].kind;
- /* We assume we can't fail below this point, so OK to scribble on resarr */
- resarr->itemsarr = newitemsarr;
- resarr->capacity = newcap;
- resarr->maxitems = RESARRAY_MAX_ITEMS(newcap);
- resarr->nitems = 0;
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+ found = true;
+ }
+ }
- if (olditemsarr != NULL)
- {
/*
- * Transfer any pre-existing entries into the new array; they don't
- * necessarily go where they were before, so this simple logic is the
- * best way. Note that if we were managing the set as a simple array,
- * the entries after nitems are garbage, but that shouldn't matter
- * because we won't get here unless nitems was equal to oldcap.
+ * If any resources were released, check again because some of the
+ * elements might have been moved by the callbacks. We don't want to
+ * miss them.
*/
- for (i = 0; i < oldcap; i++)
+ } while (found && owner->narr > 0);
+
+ /* Ok, the array has now been handled. Then the hash */
+ do
+ {
+ capacity = owner->capacity;
+ for (int idx = 0; idx < capacity; idx++)
{
- if (olditemsarr[i] != resarr->invalidval)
- ResourceArrayAdd(resarr, olditemsarr[i]);
+ while (owner->hash[idx].kind != NULL &&
+ owner->hash[idx].kind->phase == phase)
+ {
+ Datum value = owner->hash[idx].item;
+ ResourceOwnerFuncs *kind = owner->hash[idx].kind;
+
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+
+ /*
+ * If the resource is remembered more than once in this
+ * resource owner, the ReleaseResource callback might've
+ * released a different copy of it. Because of that, loop to
+ * check the same index again.
+ */
+ }
}
- /* And release old array. */
- pfree(olditemsarr);
- }
-
- Assert(resarr->nitems < resarr->maxitems);
+ /*
+ * It's possible that the callbacks acquired more resources, causing
+ * the hash table to grow and the existing entries to be moved around.
+ * If that happened, scan the hash table again, so that we don't miss
+ * entries that were moved. (XXX: I'm not sure if any of the callbacks
+ * actually do that, but this is cheap to check, and better safe than
+ * sorry.)
+ */
+ Assert(owner->capacity >= capacity);
+ } while (capacity != owner->capacity);
}
+
+/*****************************************************************************
+ * EXPORTED ROUTINES *
+ *****************************************************************************/
+
+
/*
- * Add a resource to ResourceArray
+ * ResourceOwnerCreate
+ * Create an empty ResourceOwner.
*
- * Caller must have previously done ResourceArrayEnlarge()
+ * All ResourceOwner objects are kept in TopMemoryContext, since they should
+ * only be freed explicitly.
*/
-static void
-ResourceArrayAdd(ResourceArray *resarr, Datum value)
+ResourceOwner
+ResourceOwnerCreate(ResourceOwner parent, const char *name)
{
- uint32 idx;
+ ResourceOwner owner;
- Assert(value != resarr->invalidval);
- Assert(resarr->nitems < resarr->maxitems);
+ owner = (ResourceOwner) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(ResourceOwnerData));
+ owner->name = name;
- if (RESARRAY_IS_ARRAY(resarr))
+ if (parent)
{
- /* Append to linear array. */
- idx = resarr->nitems;
+ owner->parent = parent;
+ owner->nextchild = parent->firstchild;
+ parent->firstchild = owner;
}
- else
- {
- /* Insert into first free slot at or after hash location. */
- uint32 mask = resarr->capacity - 1;
- idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (;;)
- {
- if (resarr->itemsarr[idx] == resarr->invalidval)
- break;
- idx = (idx + 1) & mask;
- }
- }
- resarr->lastidx = idx;
- resarr->itemsarr[idx] = value;
- resarr->nitems++;
+#ifdef RESOWNER_TRACE
+ elog(LOG, "CREATE %d: %p %s",
+ resowner_trace_counter++, owner, name);
+#endif
+
+ return owner;
}
/*
- * Remove a resource from ResourceArray
- *
- * Returns true on success, false if resource was not found.
+ * Make sure there is room for at least one more resource in an array.
*
- * Note: if same resource ID appears more than once, one instance is removed.
+ * This is separate from actually inserting a resource because if we run out
+ * of memory, it's critical to do so *before* acquiring the resource.
*/
-static bool
-ResourceArrayRemove(ResourceArray *resarr, Datum value)
+void
+ResourceOwnerEnlarge(ResourceOwner owner)
{
- uint32 i,
- idx,
- lastidx = resarr->lastidx;
-
- Assert(value != resarr->invalidval);
+ if (owner->narr < RESOWNER_ARRAY_SIZE)
+ return; /* no work needed */
- /* Search through all items, but try lastidx first. */
- if (RESARRAY_IS_ARRAY(resarr))
+ /* Is there space in the hash? If not, enlarge it. */
+ if (owner->narr + owner->nhash >= owner->grow_at)
{
- if (lastidx < resarr->nitems &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
- }
- for (i = 0; i < resarr->nitems; i++)
+ uint32 i,
+ oldcap,
+ newcap;
+ ResourceElem *oldhash;
+ ResourceElem *newhash;
+
+ oldhash = owner->hash;
+ oldcap = owner->capacity;
+
+ /* Double the capacity (it must stay a power of 2!) */
+ newcap = (oldcap > 0) ? oldcap * 2 : RESOWNER_HASH_INIT_SIZE;
+ newhash = (ResourceElem *) MemoryContextAllocZero(TopMemoryContext,
+ newcap * sizeof(ResourceElem));
+
+ /*
+ * We assume we can't fail below this point, so OK to scribble on the
+ * owner
+ */
+ owner->hash = newhash;
+ owner->capacity = newcap;
+ owner->grow_at = RESOWNER_HASH_MAX_ITEMS(newcap);
+ owner->nhash = 0;
+
+ if (oldhash != NULL)
{
- if (resarr->itemsarr[i] == value)
+ /*
+ * Transfer any pre-existing entries into the new hash table; they
+ * don't necessarily go where they were before, so this simple
+ * logic is the best way.
+ */
+ for (i = 0; i < oldcap; i++)
{
- resarr->itemsarr[i] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
+ if (oldhash[i].kind != NULL)
+ ResourceOwnerAddToHash(owner, oldhash[i].item, oldhash[i].kind);
}
+
+ /* And release old hash table. */
+ pfree(oldhash);
}
}
- else
- {
- uint32 mask = resarr->capacity - 1;
- if (lastidx < resarr->capacity &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->invalidval;
- resarr->nitems--;
- return true;
- }
- idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (i = 0; i < resarr->capacity; i++)
- {
- if (resarr->itemsarr[idx] == value)
- {
- resarr->itemsarr[idx] = resarr->invalidval;
- resarr->nitems--;
- return true;
- }
- idx = (idx + 1) & mask;
- }
+ /* Move items from the array to the hash */
+ Assert(owner->narr == RESOWNER_ARRAY_SIZE);
+ for (int i = 0; i < owner->narr; i++)
+ {
+ ResourceOwnerAddToHash(owner, owner->arr[i].item, owner->arr[i].kind);
}
+ owner->narr = 0;
- return false;
+ Assert(owner->nhash < owner->grow_at);
}
/*
- * Get any convenient entry in a ResourceArray.
+ * Remember that an object is owner by a ReourceOwner
*
- * "Convenient" is defined as "easy for ResourceArrayRemove to remove";
- * we help that along by setting lastidx to match. This avoids O(N^2) cost
- * when removing all ResourceArray items during ResourceOwner destruction.
- *
- * Returns true if we found an element, or false if the array is empty.
+ * Caller must have previously done ResourceOwnerEnlarge()
*/
-static bool
-ResourceArrayGetAny(ResourceArray *resarr, Datum *value)
+void
+ResourceOwnerRemember(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- if (resarr->nitems == 0)
- return false;
+ uint32 idx;
- if (RESARRAY_IS_ARRAY(resarr))
- {
- /* Linear array: just return the first element. */
- resarr->lastidx = 0;
- }
- else
- {
- /* Hash: search forward from wherever we were last. */
- uint32 mask = resarr->capacity - 1;
+#ifdef RESOWNER_TRACE
+ elog(LOG, "REMEMBER %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
- for (;;)
- {
- resarr->lastidx &= mask;
- if (resarr->itemsarr[resarr->lastidx] != resarr->invalidval)
- break;
- resarr->lastidx++;
- }
- }
+ Assert(owner->narr < RESOWNER_ARRAY_SIZE);
- *value = resarr->itemsarr[resarr->lastidx];
- return true;
+ /* Append to linear array. */
+ idx = owner->narr;
+ owner->arr[idx].item = value;
+ owner->arr[idx].kind = kind;
+ owner->narr++;
}
/*
- * Trash a ResourceArray (we don't care about its state after this)
+ * Forget that an object is owned by a ResourceOwner
+ *
+ * Returns true on success. If the resource was not found, returns false,
+ * and calls kind->ForgetError callback.
+ *
+ * Note: if same resource ID is associated with the ResourceOwner more than once,
+ * one instance is removed.
*/
-static void
-ResourceArrayFree(ResourceArray *resarr)
+void
+ResourceOwnerForget(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- if (resarr->itemsarr)
- pfree(resarr->itemsarr);
-}
-
+ uint32 i,
+ idx;
-/*****************************************************************************
- * EXPORTED ROUTINES *
- *****************************************************************************/
+#ifdef RESOWNER_TRACE
+ elog(LOG, "FORGET %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
+ /* Search through all items, but check the array first. */
+ for (i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].item == value &&
+ owner->arr[i].kind == kind)
+ {
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
-/*
- * ResourceOwnerCreate
- * Create an empty ResourceOwner.
- *
- * All ResourceOwner objects are kept in TopMemoryContext, since they should
- * only be freed explicitly.
- */
-ResourceOwner
-ResourceOwnerCreate(ResourceOwner parent, const char *name)
-{
- ResourceOwner owner;
+#ifdef RESOWNER_STATS
+ narray_lookups++;
+#endif
- owner = (ResourceOwner) MemoryContextAllocZero(TopMemoryContext,
- sizeof(ResourceOwnerData));
- owner->name = name;
+ return;
+ }
+ }
- if (parent)
+ /* Search hash */
+ if (owner->nhash > 0)
{
- owner->parent = parent;
- owner->nextchild = parent->firstchild;
- parent->firstchild = owner;
- }
+ uint32 mask = owner->capacity - 1;
- ResourceArrayInit(&(owner->bufferarr), BufferGetDatum(InvalidBuffer));
- ResourceArrayInit(&(owner->catrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->catlistrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->relrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->planrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->tupdescarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->snapshotarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->filearr), FileGetDatum(-1));
- ResourceArrayInit(&(owner->dsmarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->jitarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->cryptohasharr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->hmacarr), PointerGetDatum(NULL));
+ idx = hash_resource_elem(value, kind) & mask;
+ for (i = 0; i < owner->capacity; i++)
+ {
+ if (owner->hash[idx].item == value &&
+ owner->hash[idx].kind == kind)
+ {
+ owner->hash[idx].item = (Datum) 0;
+ owner->hash[idx].kind = NULL;
+ owner->nhash--;
+
+#ifdef RESOWNER_STATS
+ nhash_lookups++;
+#endif
+ return;
+ }
+ idx = (idx + 1) & mask;
+ }
+ }
- return owner;
+ /*
+ * Use %p to print the reference, since most objects tracked by a resource
+ * owner are pointers. It's a bit misleading if it's not a pointer, but
+ * this is a programmer error, anyway.
+ */
+ elog(ERROR, "%s %p is not owned by resource owner %s",
+ kind->name, DatumGetPointer(value), owner->name);
}
/*
@@ -490,6 +497,15 @@ ResourceOwnerRelease(ResourceOwner owner,
{
/* There's not currently any setup needed before recursing */
ResourceOwnerReleaseInternal(owner, phase, isCommit, isTopLevel);
+
+#ifdef RESOWNER_STATS
+ if (isTopLevel)
+ {
+ elog(LOG, "RESOWNER STATS: lookups: array %d, hash %d", narray_lookups, nhash_lookups);
+ narray_lookups = 0;
+ nhash_lookups = 0;
+ }
+#endif
}
static void
@@ -501,7 +517,6 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceOwner child;
ResourceOwner save;
ResourceReleaseCallbackItem *item;
- Datum foundres;
/* Recurse to handle descendants */
for (child = owner->firstchild; child != NULL; child = child->nextchild)
@@ -517,71 +532,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
if (phase == RESOURCE_RELEASE_BEFORE_LOCKS)
{
/*
- * Release buffer pins. Note that ReleaseBuffer will remove the
- * buffer entry from our array, so we just have to iterate till there
- * are none.
+ * Release all references that need to be released before the locks.
*
- * During a commit, there shouldn't be any remaining pins --- that
- * would indicate failure to clean up the executor correctly --- so
- * issue warnings. In the abort case, just clean up quietly.
+ * During a commit, there shouldn't be any remaining references ---
+ * that would indicate failure to clean up the executor correctly ---
+ * so issue warnings. In the abort case, just clean up quietly.
*/
- while (ResourceArrayGetAny(&(owner->bufferarr), &foundres))
- {
- Buffer res = DatumGetBuffer(foundres);
-
- if (isCommit)
- PrintBufferLeakWarning(res);
- ReleaseBuffer(res);
- }
-
- /* Ditto for relcache references */
- while (ResourceArrayGetAny(&(owner->relrefarr), &foundres))
- {
- Relation res = (Relation) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintRelCacheLeakWarning(res);
- RelationClose(res);
- }
-
- /* Ditto for dynamic shared memory segments */
- while (ResourceArrayGetAny(&(owner->dsmarr), &foundres))
- {
- dsm_segment *res = (dsm_segment *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintDSMLeakWarning(res);
- dsm_detach(res);
- }
-
- /* Ditto for JIT contexts */
- while (ResourceArrayGetAny(&(owner->jitarr), &foundres))
- {
- JitContext *context = (JitContext *) PointerGetDatum(foundres);
-
- jit_release_context(context);
- }
-
- /* Ditto for cryptohash contexts */
- while (ResourceArrayGetAny(&(owner->cryptohasharr), &foundres))
- {
- pg_cryptohash_ctx *context =
- (pg_cryptohash_ctx *) PointerGetDatum(foundres);
-
- if (isCommit)
- PrintCryptoHashLeakWarning(foundres);
- pg_cryptohash_free(context);
- }
-
- /* Ditto for HMAC contexts */
- while (ResourceArrayGetAny(&(owner->hmacarr), &foundres))
- {
- pg_hmac_ctx *context = (pg_hmac_ctx *) PointerGetDatum(foundres);
-
- if (isCommit)
- PrintHMACLeakWarning(foundres);
- pg_hmac_free(context);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -590,7 +547,7 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
/*
* For a top-level xact we are going to release all locks (or at
* least all non-session locks), so just do a single lmgr call at
- * the top of the recursion.
+ * the top of the recursion
*/
if (owner == TopTransactionResourceOwner)
{
@@ -634,70 +591,9 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
else if (phase == RESOURCE_RELEASE_AFTER_LOCKS)
{
/*
- * Release catcache references. Note that ReleaseCatCache will remove
- * the catref entry from our array, so we just have to iterate till
- * there are none.
- *
- * As with buffer pins, warn if any are left at commit time.
+ * Release all references that need to be released after the locks.
*/
- while (ResourceArrayGetAny(&(owner->catrefarr), &foundres))
- {
- HeapTuple res = (HeapTuple) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheLeakWarning(res);
- ReleaseCatCache(res);
- }
-
- /* Ditto for catcache lists */
- while (ResourceArrayGetAny(&(owner->catlistrefarr), &foundres))
- {
- CatCList *res = (CatCList *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheListLeakWarning(res);
- ReleaseCatCacheList(res);
- }
-
- /* Ditto for plancache references */
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
- {
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintPlanCacheLeakWarning(res);
- ReleaseCachedPlan(res, owner);
- }
-
- /* Ditto for tupdesc references */
- while (ResourceArrayGetAny(&(owner->tupdescarr), &foundres))
- {
- TupleDesc res = (TupleDesc) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintTupleDescLeakWarning(res);
- DecrTupleDescRefCount(res);
- }
-
- /* Ditto for snapshot references */
- while (ResourceArrayGetAny(&(owner->snapshotarr), &foundres))
- {
- Snapshot res = (Snapshot) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintSnapshotLeakWarning(res);
- UnregisterSnapshot(res);
- }
-
- /* Ditto for temporary files */
- while (ResourceArrayGetAny(&(owner->filearr), &foundres))
- {
- File res = DatumGetFile(foundres);
-
- if (isCommit)
- PrintFileLeakWarning(res);
- FileClose(res);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
/* Let add-on modules get a chance too */
@@ -717,13 +613,42 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
void
ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner)
{
- Datum foundres;
+ /* array first */
+ for (int i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->arr[i].item);
+
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
+ i--;
+
+ /*
+ * pass 'NULL' because we already removed the entry from the
+ * resowner
+ */
+ ReleaseCachedPlan(planref, NULL);
+ }
+ }
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
+ /* Then hash */
+ for (int i = 0; i < owner->capacity; i++)
{
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
+ if (owner->hash[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->hash[i].item);
+
+ owner->hash[i].item = (Datum) 0;
+ owner->hash[i].kind = NULL;
+ owner->nhash--;
- ReleaseCachedPlan(res, owner);
+ /*
+ * pass 'NULL' because we already removed the entry from the
+ * resowner
+ */
+ ReleaseCachedPlan(planref, NULL);
+ }
}
}
@@ -740,20 +665,15 @@ ResourceOwnerDelete(ResourceOwner owner)
Assert(owner != CurrentResourceOwner);
/* And it better not own any resources, either */
- Assert(owner->bufferarr.nitems == 0);
- Assert(owner->catrefarr.nitems == 0);
- Assert(owner->catlistrefarr.nitems == 0);
- Assert(owner->relrefarr.nitems == 0);
- Assert(owner->planrefarr.nitems == 0);
- Assert(owner->tupdescarr.nitems == 0);
- Assert(owner->snapshotarr.nitems == 0);
- Assert(owner->filearr.nitems == 0);
- Assert(owner->dsmarr.nitems == 0);
- Assert(owner->jitarr.nitems == 0);
- Assert(owner->cryptohasharr.nitems == 0);
- Assert(owner->hmacarr.nitems == 0);
+ Assert(owner->narr == 0);
+ Assert(owner->nhash == 0);
Assert(owner->nlocks == 0 || owner->nlocks == MAX_RESOWNER_LOCKS + 1);
+#ifdef RESOWNER_TRACE
+ elog(LOG, "DELETE %d: %p %s",
+ resowner_trace_counter++, owner, owner->name);
+#endif
+
/*
* Delete children. The recursive call will delink the child from me, so
* just iterate as long as there is a child.
@@ -769,19 +689,8 @@ ResourceOwnerDelete(ResourceOwner owner)
ResourceOwnerNewParent(owner, NULL);
/* And free the object. */
- ResourceArrayFree(&(owner->bufferarr));
- ResourceArrayFree(&(owner->catrefarr));
- ResourceArrayFree(&(owner->catlistrefarr));
- ResourceArrayFree(&(owner->relrefarr));
- ResourceArrayFree(&(owner->planrefarr));
- ResourceArrayFree(&(owner->tupdescarr));
- ResourceArrayFree(&(owner->snapshotarr));
- ResourceArrayFree(&(owner->filearr));
- ResourceArrayFree(&(owner->dsmarr));
- ResourceArrayFree(&(owner->jitarr));
- ResourceArrayFree(&(owner->cryptohasharr));
- ResourceArrayFree(&(owner->hmacarr));
-
+ if (owner->hash)
+ pfree(owner->hash);
pfree(owner);
}
@@ -839,11 +748,10 @@ ResourceOwnerNewParent(ResourceOwner owner,
/*
* Register or deregister callback functions for resource cleanup
*
- * These functions are intended for use by dynamically loaded modules.
- * For built-in modules we generally just hardwire the appropriate calls.
- *
- * Note that the callback occurs post-commit or post-abort, so the callback
- * functions can only do noncritical cleanup.
+ * These functions can be used by dynamically loaded modules. These used
+ * to be the only way for an extension to register custom resource types
+ * with a resource owner, but nowadays it is easier to define a new
+ * ResourceOwnerFuncs instance with custom callbacks.
*/
void
RegisterResourceReleaseCallback(ResourceReleaseCallback callback, void *arg)
@@ -934,58 +842,20 @@ ReleaseAuxProcessResourcesCallback(int code, Datum arg)
ReleaseAuxProcessResources(isCommit);
}
-
/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * buffer array.
+ * Remember that a Local Lock is owned by a ResourceOwner
*
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
+ * This is different from the other Remember functions in that the list of
+ * locks is only a lossy cache. It can hold up to MAX_RESOWNER_LOCKS entries,
+ * and when it overflows, we stop tracking locks. The point of only remembering
+ * only up to MAX_RESOWNER_LOCKS entries is that if a lot of locks are held,
+ * ResourceOwnerForgetLock doesn't need to scan through a large array to find
+ * the entry.
*/
void
-ResourceOwnerEnlargeBuffers(ResourceOwner owner)
+ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock)
{
- /* We used to allow pinning buffers without a resowner, but no more */
- Assert(owner != NULL);
- ResourceArrayEnlarge(&(owner->bufferarr));
-}
-
-/*
- * Remember that a buffer pin is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeBuffers()
- */
-void
-ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer)
-{
- ResourceArrayAdd(&(owner->bufferarr), BufferGetDatum(buffer));
-}
-
-/*
- * Forget that a buffer pin is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer)
-{
- if (!ResourceArrayRemove(&(owner->bufferarr), BufferGetDatum(buffer)))
- elog(ERROR, "buffer %d is not owned by resource owner %s",
- buffer, owner->name);
-}
-
-/*
- * Remember that a Local Lock is owned by a ResourceOwner
- *
- * This is different from the other Remember functions in that the list of
- * locks is only a lossy cache. It can hold up to MAX_RESOWNER_LOCKS entries,
- * and when it overflows, we stop tracking locks. The point of only remembering
- * only up to MAX_RESOWNER_LOCKS entries is that if a lot of locks are held,
- * ResourceOwnerForgetLock doesn't need to scan through a large array to find
- * the entry.
- */
-void
-ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock)
-{
- Assert(locallock != NULL);
+ Assert(locallock != NULL);
if (owner->nlocks > MAX_RESOWNER_LOCKS)
return; /* we have already overflowed */
@@ -1023,469 +893,3 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catrefarr));
-}
-
-/*
- * Remember that a catcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheRefs()
- */
-void
-ResourceOwnerRememberCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- ResourceArrayAdd(&(owner->catrefarr), PointerGetDatum(tuple));
-}
-
-/*
- * Forget that a catcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- if (!ResourceArrayRemove(&(owner->catrefarr), PointerGetDatum(tuple)))
- elog(ERROR, "catcache reference %p is not owned by resource owner %s",
- tuple, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache-list reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catlistrefarr));
-}
-
-/*
- * Remember that a catcache-list reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheListRefs()
- */
-void
-ResourceOwnerRememberCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- ResourceArrayAdd(&(owner->catlistrefarr), PointerGetDatum(list));
-}
-
-/*
- * Forget that a catcache-list reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- if (!ResourceArrayRemove(&(owner->catlistrefarr), PointerGetDatum(list)))
- elog(ERROR, "catcache list reference %p is not owned by resource owner %s",
- list, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * relcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeRelationRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->relrefarr));
-}
-
-/*
- * Remember that a relcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeRelationRefs()
- */
-void
-ResourceOwnerRememberRelationRef(ResourceOwner owner, Relation rel)
-{
- ResourceArrayAdd(&(owner->relrefarr), PointerGetDatum(rel));
-}
-
-/*
- * Forget that a relcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetRelationRef(ResourceOwner owner, Relation rel)
-{
- if (!ResourceArrayRemove(&(owner->relrefarr), PointerGetDatum(rel)))
- elog(ERROR, "relcache reference %s is not owned by resource owner %s",
- RelationGetRelationName(rel), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintRelCacheLeakWarning(Relation rel)
-{
- elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
- RelationGetRelationName(rel));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * plancache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->planrefarr));
-}
-
-/*
- * Remember that a plancache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargePlanCacheRefs()
- */
-void
-ResourceOwnerRememberPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- ResourceArrayAdd(&(owner->planrefarr), PointerGetDatum(plan));
-}
-
-/*
- * Forget that a plancache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- if (!ResourceArrayRemove(&(owner->planrefarr), PointerGetDatum(plan)))
- elog(ERROR, "plancache reference %p is not owned by resource owner %s",
- plan, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintPlanCacheLeakWarning(CachedPlan *plan)
-{
- elog(WARNING, "plancache reference leak: plan %p not closed", plan);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * tupdesc reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeTupleDescs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->tupdescarr));
-}
-
-/*
- * Remember that a tupdesc reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeTupleDescs()
- */
-void
-ResourceOwnerRememberTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- ResourceArrayAdd(&(owner->tupdescarr), PointerGetDatum(tupdesc));
-}
-
-/*
- * Forget that a tupdesc reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- if (!ResourceArrayRemove(&(owner->tupdescarr), PointerGetDatum(tupdesc)))
- elog(ERROR, "tupdesc reference %p is not owned by resource owner %s",
- tupdesc, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintTupleDescLeakWarning(TupleDesc tupdesc)
-{
- elog(WARNING,
- "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
- tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * snapshot reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeSnapshots(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->snapshotarr));
-}
-
-/*
- * Remember that a snapshot reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeSnapshots()
- */
-void
-ResourceOwnerRememberSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- ResourceArrayAdd(&(owner->snapshotarr), PointerGetDatum(snapshot));
-}
-
-/*
- * Forget that a snapshot reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- if (!ResourceArrayRemove(&(owner->snapshotarr), PointerGetDatum(snapshot)))
- elog(ERROR, "snapshot reference %p is not owned by resource owner %s",
- snapshot, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintSnapshotLeakWarning(Snapshot snapshot)
-{
- elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
- snapshot);
-}
-
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * files reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeFiles(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->filearr));
-}
-
-/*
- * Remember that a temporary file is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeFiles()
- */
-void
-ResourceOwnerRememberFile(ResourceOwner owner, File file)
-{
- ResourceArrayAdd(&(owner->filearr), FileGetDatum(file));
-}
-
-/*
- * Forget that a temporary file is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetFile(ResourceOwner owner, File file)
-{
- if (!ResourceArrayRemove(&(owner->filearr), FileGetDatum(file)))
- elog(ERROR, "temporary file %d is not owned by resource owner %s",
- file, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintFileLeakWarning(File file)
-{
- elog(WARNING, "temporary file leak: File %d still referenced",
- file);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * dynamic shmem segment reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeDSMs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->dsmarr));
-}
-
-/*
- * Remember that a dynamic shmem segment is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeDSMs()
- */
-void
-ResourceOwnerRememberDSM(ResourceOwner owner, dsm_segment *seg)
-{
- ResourceArrayAdd(&(owner->dsmarr), PointerGetDatum(seg));
-}
-
-/*
- * Forget that a dynamic shmem segment is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetDSM(ResourceOwner owner, dsm_segment *seg)
-{
- if (!ResourceArrayRemove(&(owner->dsmarr), PointerGetDatum(seg)))
- elog(ERROR, "dynamic shared memory segment %u is not owned by resource owner %s",
- dsm_segment_handle(seg), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintDSMLeakWarning(dsm_segment *seg)
-{
- elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
- dsm_segment_handle(seg));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * JIT context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeJIT(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->jitarr));
-}
-
-/*
- * Remember that a JIT context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeJIT()
- */
-void
-ResourceOwnerRememberJIT(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->jitarr), handle);
-}
-
-/*
- * Forget that a JIT context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetJIT(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->jitarr), handle))
- elog(ERROR, "JIT context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * cryptohash context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCryptoHash(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->cryptohasharr));
-}
-
-/*
- * Remember that a cryptohash context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCryptoHash()
- */
-void
-ResourceOwnerRememberCryptoHash(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->cryptohasharr), handle);
-}
-
-/*
- * Forget that a cryptohash context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCryptoHash(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->cryptohasharr), handle))
- elog(ERROR, "cryptohash context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintCryptoHashLeakWarning(Datum handle)
-{
- elog(WARNING, "cryptohash context reference leak: context %p still referenced",
- DatumGetPointer(handle));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * hmac context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeHMAC(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->hmacarr));
-}
-
-/*
- * Remember that a HMAC context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeHMAC()
- */
-void
-ResourceOwnerRememberHMAC(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->hmacarr), handle);
-}
-
-/*
- * Forget that a HMAC context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetHMAC(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->hmacarr), handle))
- elog(ERROR, "HMAC context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintHMACLeakWarning(Datum handle)
-{
- elog(WARNING, "HMAC context reference leak: context %p still referenced",
- DatumGetPointer(handle));
-}
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 2968c7f7b7..1f46cbe58c 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -66,7 +66,7 @@
#include "utils/memutils.h"
#include "utils/old_snapshot.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
#include "utils/timestamp.h"
@@ -174,6 +174,24 @@ static Snapshot CopySnapshot(Snapshot snapshot);
static void FreeSnapshot(Snapshot snapshot);
static void SnapshotResetXmin(void);
+/* ResourceOwner callbacks to track snapshot references */
+static void ResOwnerReleaseSnapshot(Datum res);
+static void ResOwnerPrintSnapshotLeakWarning(Datum res);
+
+static ResourceOwnerFuncs snapshot_resowner_funcs =
+{
+ .name = "snapshot reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseSnapshot,
+ .PrintLeakWarning = ResOwnerPrintSnapshotLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberSnapshot(owner, snap) \
+ ResourceOwnerRemember(owner, PointerGetDatum(snap), &snapshot_resowner_funcs)
+#define ResourceOwnerForgetSnapshot(owner, snap) \
+ ResourceOwnerForget(owner, PointerGetDatum(snap), &snapshot_resowner_funcs)
+
/*
* Snapshot fields to be serialized.
*
@@ -831,7 +849,7 @@ RegisterSnapshotOnOwner(Snapshot snapshot, ResourceOwner owner)
snap = snapshot->copied ? snapshot : CopySnapshot(snapshot);
/* and tell resowner.c about it */
- ResourceOwnerEnlargeSnapshots(owner);
+ ResourceOwnerEnlarge(owner);
snap->regd_count++;
ResourceOwnerRememberSnapshot(owner, snap);
@@ -2349,3 +2367,19 @@ XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot)
return false;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseSnapshot(Datum res)
+{
+ UnregisterSnapshot((Snapshot) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintSnapshotLeakWarning(Datum res)
+{
+ elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
+ DatumGetPointer(res));
+}
diff --git a/src/common/cryptohash_openssl.c b/src/common/cryptohash_openssl.c
index 643cc7aea2..cf5fc9eacb 100644
--- a/src/common/cryptohash_openssl.c
+++ b/src/common/cryptohash_openssl.c
@@ -30,7 +30,6 @@
#ifndef FRONTEND
#include "utils/memutils.h"
#include "utils/resowner.h"
-#include "utils/resowner_private.h"
#endif
/*
@@ -63,6 +62,27 @@ struct pg_cryptohash_ctx
#endif
};
+/* ResourceOwner callbacks to hold cryptohash contexts */
+#ifndef FRONTEND
+static void ResOwnerReleaseCryptoHash(Datum res);
+static void ResOwnerPrintCryptoHashLeakWarning(Datum res);
+
+static ResourceOwnerFuncs cryptohash_resowner_funcs =
+{
+ /* relcache references */
+ .name = "OpenSSL cryptohash context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCryptoHash,
+ .PrintLeakWarning = ResOwnerPrintCryptoHashLeakWarning,
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberCryptoHash(owner, ctx) \
+ ResourceOwnerRemember(owner, PointerGetDatum(ctx), &cryptohash_resowner_funcs)
+#define ResourceOwnerForgetCryptoHash(owner, ctx) \
+ ResourceOwnerForget(owner, PointerGetDatum(ctx), &cryptohash_resowner_funcs)
+#endif
+
/*
* pg_cryptohash_create
*
@@ -80,7 +100,7 @@ pg_cryptohash_create(pg_cryptohash_type type)
* allocation to avoid leaking.
*/
#ifndef FRONTEND
- ResourceOwnerEnlargeCryptoHash(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
#endif
ctx = ALLOC(sizeof(pg_cryptohash_ctx));
@@ -109,8 +129,7 @@ pg_cryptohash_create(pg_cryptohash_type type)
#ifndef FRONTEND
ctx->resowner = CurrentResourceOwner;
- ResourceOwnerRememberCryptoHash(CurrentResourceOwner,
- PointerGetDatum(ctx));
+ ResourceOwnerRememberCryptoHash(CurrentResourceOwner, ctx);
#endif
return ctx;
@@ -241,10 +260,28 @@ pg_cryptohash_free(pg_cryptohash_ctx *ctx)
EVP_MD_CTX_destroy(ctx->evpctx);
#ifndef FRONTEND
- ResourceOwnerForgetCryptoHash(ctx->resowner,
- PointerGetDatum(ctx));
+ ResourceOwnerForgetCryptoHash(ctx->resowner, ctx);
#endif
explicit_bzero(ctx, sizeof(pg_cryptohash_ctx));
FREE(ctx);
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+#ifndef FRONTEND
+static void
+ResOwnerReleaseCryptoHash(Datum res)
+{
+ pg_cryptohash_free((pg_cryptohash_ctx *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCryptoHashLeakWarning(Datum res)
+{
+ elog(WARNING, "cryptohash context reference leak: context %p still referenced",
+ DatumGetPointer(res));
+}
+#endif
diff --git a/src/common/hmac_openssl.c b/src/common/hmac_openssl.c
index 1acf59476e..0a65284bae 100644
--- a/src/common/hmac_openssl.c
+++ b/src/common/hmac_openssl.c
@@ -29,7 +29,6 @@
#ifndef FRONTEND
#include "utils/memutils.h"
#include "utils/resowner.h"
-#include "utils/resowner_private.h"
#endif
/*
@@ -63,6 +62,27 @@ struct pg_hmac_ctx
#endif
};
+/* ResourceOwner callbacks to hold HMAC contexts */
+#ifndef FRONTEND
+static void ResOwnerReleaseHMAC(Datum res);
+static void ResOwnerPrintHMACLeakWarning(Datum res);
+
+static ResourceOwnerFuncs hmac_resowner_funcs =
+{
+ /* relcache references */
+ .name = "OpenSSL HMAC context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseHMAC,
+ .PrintLeakWarning = ResOwnerPrintHMACLeakWarning,
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberHMAC(owner, ctx) \
+ ResourceOwnerRemember(owner, PointerGetDatum(ctx), &hmac_resowner_funcs)
+#define ResourceOwnerForgetHMAC(owner, ctx) \
+ ResourceOwnerForget(owner, PointerGetDatum(ctx), &hmac_resowner_funcs)
+#endif
+
/*
* pg_hmac_create
*
@@ -86,7 +106,7 @@ pg_hmac_create(pg_cryptohash_type type)
*/
#ifdef HAVE_HMAC_CTX_NEW
#ifndef FRONTEND
- ResourceOwnerEnlargeHMAC(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
#endif
ctx->hmacctx = HMAC_CTX_new();
#else
@@ -108,7 +128,7 @@ pg_hmac_create(pg_cryptohash_type type)
#ifdef HAVE_HMAC_CTX_NEW
#ifndef FRONTEND
ctx->resowner = CurrentResourceOwner;
- ResourceOwnerRememberHMAC(CurrentResourceOwner, PointerGetDatum(ctx));
+ ResourceOwnerRememberHMAC(CurrentResourceOwner, ctx);
#endif
#else
memset(ctx->hmacctx, 0, sizeof(HMAC_CTX));
@@ -244,7 +264,7 @@ pg_hmac_free(pg_hmac_ctx *ctx)
#ifdef HAVE_HMAC_CTX_FREE
HMAC_CTX_free(ctx->hmacctx);
#ifndef FRONTEND
- ResourceOwnerForgetHMAC(ctx->resowner, PointerGetDatum(ctx));
+ ResourceOwnerForgetHMAC(ctx->resowner, ctx);
#endif
#else
explicit_bzero(ctx->hmacctx, sizeof(HMAC_CTX));
@@ -254,3 +274,22 @@ pg_hmac_free(pg_hmac_ctx *ctx)
explicit_bzero(ctx, sizeof(pg_hmac_ctx));
FREE(ctx);
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+#ifndef FRONTEND
+static void
+ResOwnerReleaseHMAC(Datum res)
+{
+ pg_hmac_free((pg_hmac_ctx *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintHMACLeakWarning(Datum res)
+{
+ elog(WARNING, "HMAC context reference leak: context %p still referenced",
+ DatumGetPointer(res));
+}
+#endif
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 33fcaf5c9a..10ef8d850e 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -300,6 +300,15 @@ typedef struct CkptSortItem
extern CkptSortItem *CkptBufferIds;
+/* ResourceOwner callbacks to hold buffer pins */
+extern ResourceOwnerFuncs buffer_resowner_funcs;
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberBuffer(owner, buffer) \
+ ResourceOwnerRemember(owner, Int32GetDatum(buffer), &buffer_resowner_funcs)
+#define ResourceOwnerForgetBuffer(owner, buffer) \
+ ResourceOwnerForget(owner, Int32GetDatum(buffer), &buffer_resowner_funcs)
+
/*
* Internal buffer management routines
*/
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index ddc2762eb3..2bf95c22e8 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -225,7 +225,4 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
HeapTuple newtuple,
void (*function) (int, uint32, Oid));
-extern void PrintCatCacheLeakWarning(HeapTuple tuple);
-extern void PrintCatCacheListLeakWarning(CatCList *list);
-
#endif /* CATCACHE_H */
diff --git a/src/include/utils/plancache.h b/src/include/utils/plancache.h
index ff09c63a02..f3e3e50fb2 100644
--- a/src/include/utils/plancache.h
+++ b/src/include/utils/plancache.h
@@ -233,4 +233,6 @@ extern bool CachedPlanIsSimplyValid(CachedPlanSource *plansource,
extern CachedExpression *GetCachedExpression(Node *expr);
extern void FreeCachedExpression(CachedExpression *cexpr);
+extern ResourceOwnerFuncs planref_resowner_funcs;
+
#endif /* PLANCACHE_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index 109ac31b24..d59d14c0d0 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -50,6 +50,36 @@ typedef enum
RESOURCE_RELEASE_AFTER_LOCKS
} ResourceReleasePhase;
+/*
+ * In order to track an object, resowner.c needs a few callbacks for it.
+ * The callbacks for an resource of a specific kind are encapsulated in
+ * ResourceOwnerFuncs.
+ *
+ * Note that the callback occurs post-commit or post-abort, so these callback
+ * functions can only do noncritical cleanup.
+ */
+typedef struct ResourceOwnerFuncs
+{
+ const char *name; /* name for the object kind, for debugging */
+
+ ResourceReleasePhase phase; /* when are these objects released? */
+
+ /*
+ * Release resource.
+ *
+ * NOTE: this must call ResourceOwnerForget to disassociate it with the
+ * resource owner.
+ */
+ void (*ReleaseResource)(Datum res);
+
+ /*
+ * Print a warning, when a resource has not been properly released before
+ * commit.
+ */
+ void (*PrintLeakWarning)(Datum res);
+
+} ResourceOwnerFuncs;
+
/*
* Dynamically loaded modules can get control during ResourceOwnerRelease
* by providing a callback of this form.
@@ -71,16 +101,29 @@ extern void ResourceOwnerRelease(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
-extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
extern void ResourceOwnerDelete(ResourceOwner owner);
extern ResourceOwner ResourceOwnerGetParent(ResourceOwner owner);
extern void ResourceOwnerNewParent(ResourceOwner owner,
ResourceOwner newparent);
+
+extern void ResourceOwnerEnlarge(ResourceOwner owner);
+extern void ResourceOwnerRemember(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+extern void ResourceOwnerForget(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+
extern void RegisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
extern void UnregisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
+
extern void CreateAuxProcessResourceOwner(void);
extern void ReleaseAuxProcessResources(bool isCommit);
+/* special support for local lock management */
+struct LOCALLOCK;
+extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+
+/* special function to relase all plancache references */
+extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
+
#endif /* RESOWNER_H */
diff --git a/src/include/utils/resowner_private.h b/src/include/utils/resowner_private.h
deleted file mode 100644
index 6dafc87e28..0000000000
--- a/src/include/utils/resowner_private.h
+++ /dev/null
@@ -1,112 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * resowner_private.h
- * POSTGRES resource owner private definitions.
- *
- * See utils/resowner/README for more info.
- *
- *
- * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
- * Portions Copyright (c) 1994, Regents of the University of California
- *
- * src/include/utils/resowner_private.h
- *
- *-------------------------------------------------------------------------
- */
-#ifndef RESOWNER_PRIVATE_H
-#define RESOWNER_PRIVATE_H
-
-#include "storage/dsm.h"
-#include "storage/fd.h"
-#include "storage/lock.h"
-#include "utils/catcache.h"
-#include "utils/plancache.h"
-#include "utils/resowner.h"
-#include "utils/snapshot.h"
-
-
-/* support for buffer refcount management */
-extern void ResourceOwnerEnlargeBuffers(ResourceOwner owner);
-extern void ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer);
-extern void ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer);
-
-/* support for local lock management */
-extern void ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock);
-extern void ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock);
-
-/* support for catcache refcount management */
-extern void ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerForgetCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-extern void ResourceOwnerForgetCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-
-/* support for relcache refcount management */
-extern void ResourceOwnerEnlargeRelationRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberRelationRef(ResourceOwner owner,
- Relation rel);
-extern void ResourceOwnerForgetRelationRef(ResourceOwner owner,
- Relation rel);
-
-/* support for plancache refcount management */
-extern void ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-extern void ResourceOwnerForgetPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-
-/* support for tupledesc refcount management */
-extern void ResourceOwnerEnlargeTupleDescs(ResourceOwner owner);
-extern void ResourceOwnerRememberTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-extern void ResourceOwnerForgetTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-
-/* support for snapshot refcount management */
-extern void ResourceOwnerEnlargeSnapshots(ResourceOwner owner);
-extern void ResourceOwnerRememberSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-extern void ResourceOwnerForgetSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-
-/* support for temporary file management */
-extern void ResourceOwnerEnlargeFiles(ResourceOwner owner);
-extern void ResourceOwnerRememberFile(ResourceOwner owner,
- File file);
-extern void ResourceOwnerForgetFile(ResourceOwner owner,
- File file);
-
-/* support for dynamic shared memory management */
-extern void ResourceOwnerEnlargeDSMs(ResourceOwner owner);
-extern void ResourceOwnerRememberDSM(ResourceOwner owner,
- dsm_segment *);
-extern void ResourceOwnerForgetDSM(ResourceOwner owner,
- dsm_segment *);
-
-/* support for JITContext management */
-extern void ResourceOwnerEnlargeJIT(ResourceOwner owner);
-extern void ResourceOwnerRememberJIT(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetJIT(ResourceOwner owner,
- Datum handle);
-
-/* support for cryptohash context management */
-extern void ResourceOwnerEnlargeCryptoHash(ResourceOwner owner);
-extern void ResourceOwnerRememberCryptoHash(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetCryptoHash(ResourceOwner owner,
- Datum handle);
-
-/* support for HMAC context management */
-extern void ResourceOwnerEnlargeHMAC(ResourceOwner owner);
-extern void ResourceOwnerRememberHMAC(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetHMAC(ResourceOwner owner,
- Datum handle);
-
-#endif /* RESOWNER_PRIVATE_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 423780652f..5eaef46702 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2226,8 +2226,10 @@ ReplicationStateOnDisk
ResTarget
ReservoirState
ReservoirStateData
-ResourceArray
+ResourceElem
ResourceOwner
+ResourceOwnerData
+ResourceOwnerFuncs
ResourceReleaseCallback
ResourceReleaseCallbackItem
ResourceReleasePhase
[application/octet-stream] v9-0004-Minor-tweaks.patch (1.4K, ../../CAJ7c6TP8v=2Ym5S1C4YenaCMgwu5FrFTGi40Nn5TyDS4ig-gRA@mail.gmail.com/5-v9-0004-Minor-tweaks.patch)
download | inline diff:
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 6189d05107..e6b136ca04 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -37,7 +37,7 @@
typedef struct ResourceElem
{
Datum item;
- ResourceOwnerFuncs *kind;
+ ResourceOwnerFuncs *kind; /* kind == NULL indicates a free hash table slot */
} ResourceElem;
/*
@@ -186,9 +186,14 @@ hash_resource_elem(Datum value, ResourceOwnerFuncs *kind)
#endif
}
+/*
+ * Adds 'value' of given 'kind' to the ResourceOwner's hash table
+ */
static void
ResourceOwnerAddToHash(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
+ Assert(kind != NULL);
+
/* Insert into first free slot at or after hash location. */
uint32 mask = owner->capacity - 1;
uint32 idx;
@@ -197,7 +202,7 @@ ResourceOwnerAddToHash(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kin
for (;;)
{
if (owner->hash[idx].kind == NULL)
- break;
+ break; /* free slot found */
idx = (idx + 1) & mask;
}
owner->hash[idx].item = value;
@@ -408,9 +413,6 @@ ResourceOwnerRemember(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind
/*
* Forget that an object is owned by a ResourceOwner
*
- * Returns true on success. If the resource was not found, returns false,
- * and calls kind->ForgetError callback.
- *
* Note: if same resource ID is associated with the ResourceOwner more than once,
* one instance is removed.
*/
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-10-01 07:18 Michael Paquier <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Michael Paquier @ 2021-10-01 07:18 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Heikki Linnakangas <[email protected]>
On Wed, Sep 08, 2021 at 01:19:19PM +0300, Aleksander Alekseev wrote:
> Thanks for working on this!
The latest patch does not apply anymore, and has been waiting on
author for a couple of weeks now, so switched as RwF for now.
--
Michael
Attachments:
[application/pgp-signature] signature.asc (833B, ../../[email protected]/2-signature.asc)
download
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-10-18 11:17 Heikki Linnakangas <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Heikki Linnakangas @ 2021-10-18 11:17 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; pgsql-hackers
On 08/09/2021 13:19, Aleksander Alekseev wrote:
> Hi Heikki,
>
>> Yeah, needed some manual fixing, but here you go.
>
> Thanks for working on this!
>
> v8-0002 didn't apply to the current master, so I rebased it. See
> attached v9-* patches. I also included v9-0004 with some minor tweaks
> from me. I have several notes regarding the code.
Thanks!
Attached is a newly rebased version. It includes your tweaks, and a few
more comment and indentation tweaks.
> 1. Not sure if I understand this code of ResourceOwnerReleaseAll():
> ```
> /* First handle all the entries in the array. */
> do
> {
> found = false;
> for (int i = 0; i < owner->narr; i++)
> {
> if (owner->arr[i].kind->phase == phase)
> {
> Datum value = owner->arr[i].item;
> ResourceOwnerFuncs *kind = owner->arr[i].kind;
>
> if (printLeakWarnings)
> kind->PrintLeakWarning(value);
> kind->ReleaseResource(value);
> found = true;
> }
> }
>
> /*
> * If any resources were released, check again because some of the
> * elements might have been moved by the callbacks. We don't want to
> * miss them.
> */
> } while (found && owner->narr > 0);
> ```
>
> Shouldn't we mark the resource as released and/or decrease narr and/or
> save the last processed i? Why this will not call ReleaseResource()
> endlessly on the same resource (array item)? Same question for the
> following code that iterates over the hash table.
ReleaseResource() must call ResourceOwnerForget(), which removes the
item from the or hash table. This works the same as the code in 'master':
> /*
> * Release buffer pins. Note that ReleaseBuffer will remove the
> * buffer entry from our array, so we just have to iterate till there
> * are none.
> *
> * During a commit, there shouldn't be any remaining pins --- that
> * would indicate failure to clean up the executor correctly --- so
> * issue warnings. In the abort case, just clean up quietly.
> */
> while (ResourceArrayGetAny(&(owner->bufferarr), &foundres))
> {
> Buffer res = DatumGetBuffer(foundres);
>
> if (isCommit)
> PrintBufferLeakWarning(res);
> ReleaseBuffer(res);
> }
That comment explains how it works. I added a comment like that in this
patch, too.
> 2. Just an idea/observation. It's possible that the performance of
> ResourceOwnerEnlarge() can be slightly improved. Since the size of the
> hash table is always a power of 2 and the code always doubles the size
> of the hash table, (idx & mask) value will get one extra bit, which
> can be 0 or 1. If it's 0, the value is already in its place,
> otherwise, it should be moved on the known distance. In other words,
> it's possible to copy `oldhash` to `newhash` and then move only half
> of the items. I don't claim that this code necessarily should be
> faster, or that this should be checked in the scope of this work.
Hmm, the hash table uses open addressing, I think we want to also
rearrange any existing entries that might not be at their right place
because of collisions. We don't actually do that currently when an
element is removed (even before this patch), but enlarging the array is
a good opportunity for it. In any case, I haven't seen
ResourceOwnerEnlarge() show up while profiling, so I'm going to leave it
as it is.
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-11-23 15:37 Aleksander Alekseev <[email protected]>
parent: Heikki Linnakangas <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Aleksander Alekseev @ 2021-11-23 15:37 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers
Hi Heikki,
> Attached is a newly rebased version. It includes your tweaks, and a few
> more comment and indentation tweaks.
v10-0002 rotted a little so I rebased it. The new patch set passed `make
installcheck-world` locally, but let's see if cfbot has a second opinion.
I'm a bit busy with various things at the moment, so (hopefully) I will
submit the actual code review in the follow-up email.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v11-0001-Move-a-few-ResourceOwnerEnlarge-calls-for-safety.patch (8.6K, ../../CAJ7c6TOj8RsUkfnVZnAPZ6jgWxqTLuv-o5mHYTc5goEhfBXaTA@mail.gmail.com/3-v11-0001-Move-a-few-ResourceOwnerEnlarge-calls-for-safety.patch)
download | inline diff:
From 98758f3774484f54f010d4bec663ac60b764170f Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Wed, 13 Jan 2021 12:21:28 +0200
Subject: [PATCH v10 1/3] Move a few ResourceOwnerEnlarge() calls for safety
and clarity.
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code
inbetween, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(),
to ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls.
Move the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index e88e4e918b0..cde9184c7d6 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -810,9 +810,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1174,9 +1171,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resoure owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1677,8 +1676,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1689,6 +1686,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1769,7 +1768,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1945,9 +1945,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2421,9 +2418,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2505,8 +2499,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2516,7 +2508,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3555,9 +3549,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3571,7 +3562,9 @@ FlushRelationBuffers(Relation rel)
if (!RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3628,9 +3621,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rnode_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3667,7 +3657,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3707,9 +3699,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3723,7 +3712,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.rnode.dbNode != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 04b3558ea33..f7c15ea8a44 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 4fbdc62d8c7..13eed587601 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1609,8 +1609,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1698,13 +1696,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.2
[application/octet-stream] v11-0002-Make-resowners-more-easily-extensible.patch (90.3K, ../../CAJ7c6TOj8RsUkfnVZnAPZ6jgWxqTLuv-o5mHYTc5goEhfBXaTA@mail.gmail.com/4-v11-0002-Make-resowners-more-easily-extensible.patch)
download | inline diff:
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index 4c63bd4dc6..f2d4442c93 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -30,9 +30,27 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
+/* ResourceOwner callbacks to hold tupledesc references */
+static void ResOwnerReleaseTupleDesc(Datum res);
+static void ResOwnerPrintTupleDescLeakWarning(Datum res);
+
+static ResourceOwnerFuncs tupdesc_resowner_funcs =
+{
+ /* relcache references */
+ .name = "tupdesc reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseTupleDesc,
+ .PrintLeakWarning = ResOwnerPrintTupleDescLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberTupleDesc(owner, tupdesc) \
+ ResourceOwnerRemember(owner, PointerGetDatum(tupdesc), &tupdesc_resowner_funcs)
+#define ResourceOwnerForgetTupleDesc(owner, tupdesc) \
+ ResourceOwnerForget(owner, PointerGetDatum(tupdesc), &tupdesc_resowner_funcs)
/*
* CreateTemplateTupleDesc
@@ -367,7 +385,7 @@ IncrTupleDescRefCount(TupleDesc tupdesc)
{
Assert(tupdesc->tdrefcount >= 0);
- ResourceOwnerEnlargeTupleDescs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
tupdesc->tdrefcount++;
ResourceOwnerRememberTupleDesc(CurrentResourceOwner, tupdesc);
}
@@ -910,3 +928,23 @@ BuildDescFromLists(List *names, List *types, List *typmods, List *collations)
return desc;
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseTupleDesc(Datum res)
+{
+ DecrTupleDescRefCount((TupleDesc) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintTupleDescLeakWarning(Datum res)
+{
+ TupleDesc tupdesc = (TupleDesc) DatumGetPointer(res);
+
+ elog(WARNING,
+ "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
+ tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
+}
diff --git a/src/backend/jit/jit.c b/src/backend/jit/jit.c
index 91b8ae6c51..d42d1bc6c4 100644
--- a/src/backend/jit/jit.c
+++ b/src/backend/jit/jit.c
@@ -26,7 +26,6 @@
#include "jit/jit.h"
#include "miscadmin.h"
#include "utils/fmgrprotos.h"
-#include "utils/resowner_private.h"
/* GUCs */
bool jit_enabled = true;
@@ -140,7 +139,6 @@ jit_release_context(JitContext *context)
if (provider_successfully_loaded)
provider.release_context(context);
- ResourceOwnerForgetJIT(context->resowner, PointerGetDatum(context));
pfree(context);
}
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index fb29449573..8e84ac5de6 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -40,7 +40,7 @@
#include "portability/instr_time.h"
#include "storage/ipc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Handle of a module emitted via ORC JIT */
typedef struct LLVMJitHandle
@@ -121,6 +121,25 @@ static LLVMOrcLLJITRef llvm_create_jit_instance(LLVMTargetMachineRef tm);
static char *llvm_error_message(LLVMErrorRef error);
#endif /* LLVM_VERSION_MAJOR > 11 */
+/* ResourceOwner callbacks to hold JitContexts */
+static void ResOwnerReleaseJitContext(Datum res);
+static void ResOwnerPrintJitContextLeakWarning(Datum res);
+
+static ResourceOwnerFuncs jit_resowner_funcs =
+{
+ /* relcache references */
+ .name = "LLVM JIT context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseJitContext,
+ .PrintLeakWarning = ResOwnerPrintJitContextLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberJIT(owner, handle) \
+ ResourceOwnerRemember(owner, PointerGetDatum(handle), &jit_resowner_funcs)
+#define ResourceOwnerForgetJIT(owner, handle) \
+ ResourceOwnerForget(owner, PointerGetDatum(handle), &jit_resowner_funcs)
+
PG_MODULE_MAGIC;
@@ -151,7 +170,7 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
- ResourceOwnerEnlargeJIT(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
context = MemoryContextAllocZero(TopMemoryContext,
sizeof(LLVMJitContext));
@@ -159,7 +178,7 @@ llvm_create_context(int jitFlags)
/* ensure cleanup */
context->base.resowner = CurrentResourceOwner;
- ResourceOwnerRememberJIT(CurrentResourceOwner, PointerGetDatum(context));
+ ResourceOwnerRememberJIT(CurrentResourceOwner, context);
return context;
}
@@ -221,6 +240,8 @@ llvm_release_context(JitContext *context)
}
list_free(llvm_context->handles);
llvm_context->handles = NIL;
+
+ ResourceOwnerForgetJIT(context->resowner, context);
}
/*
@@ -1248,3 +1269,20 @@ llvm_error_message(LLVMErrorRef error)
}
#endif /* LLVM_VERSION_MAJOR > 11 */
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseJitContext(Datum res)
+{
+ jit_release_context((JitContext *) PointerGetDatum(res));
+}
+
+static void
+ResOwnerPrintJitContextLeakWarning(Datum res)
+{
+ /* XXX: We used to not print these. Was that intentional? */
+ JitContext *context = (JitContext *) PointerGetDatum(res);
+
+ elog(WARNING, "JIT context leak: context %p still referenced", context);
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 67c5a12211..d347d428a9 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -52,7 +52,7 @@
#include "utils/memdebug.h"
#include "utils/ps_status.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/timestamp.h"
@@ -206,6 +206,18 @@ static PrivateRefCountEntry *GetPrivateRefCountEntry(Buffer buffer, bool do_move
static inline int32 GetPrivateRefCount(Buffer buffer);
static void ForgetPrivateRefCountEntry(PrivateRefCountEntry *ref);
+/* ResourceOwner callbacks to hold buffer pins */
+static void ResOwnerReleaseBuffer(Datum res);
+static void ResOwnerPrintBufferLeakWarning(Datum res);
+
+ResourceOwnerFuncs buffer_resowner_funcs =
+{
+ .name = "buffer",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseBuffer,
+ .PrintLeakWarning = ResOwnerPrintBufferLeakWarning
+};
+
/*
* Ensure that the PrivateRefCountArray has sufficient space to store one more
* entry. This has to be called before using NewPrivateRefCountEntry() to fill
@@ -625,7 +637,7 @@ ReadRecentBuffer(RelFileNode rnode, ForkNumber forkNum, BlockNumber blockNum,
Assert(BufferIsValid(recent_buffer));
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ReservePrivateRefCountEntry();
INIT_BUFFERTAG(tag, rnode, forkNum, blockNum);
@@ -1175,7 +1187,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* pin.
*/
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1686,7 +1698,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ref = GetPrivateRefCountEntry(b, true);
@@ -1769,7 +1781,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
*
* As this function is called with the spinlock held, the caller has to
* previously call ReservePrivateRefCountEntry() and
- * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ * ResourceOwnerEnlarge(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -2510,7 +2522,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3564,7 +3576,7 @@ FlushRelationBuffers(Relation rel)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, rel->rd_node) &&
@@ -3659,7 +3671,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (RelFileNodeEquals(bufHdr->tag.rnode, srelent->rnode) &&
@@ -3714,7 +3726,7 @@ FlushDatabaseBuffers(Oid dbid)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.rnode.dbNode == dbid &&
@@ -3797,7 +3809,7 @@ void
IncrBufferRefCount(Buffer buffer)
{
Assert(BufferIsPinned(buffer));
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
if (BufferIsLocal(buffer))
LocalRefCount[-buffer - 1]++;
else
@@ -4844,3 +4856,18 @@ TestForOldSnapshot_impl(Snapshot snapshot, Relation relation)
(errcode(ERRCODE_SNAPSHOT_TOO_OLD),
errmsg("snapshot too old")));
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseBuffer(Datum res)
+{
+ ReleaseBuffer(DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintBufferLeakWarning(Datum res)
+{
+ PrintBufferLeakWarning(DatumGetInt32(res));
+}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index f7c15ea8a4..3f22d7127b 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -22,7 +22,7 @@
#include "storage/bufmgr.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/*#define LBDEBUG*/
@@ -124,7 +124,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
InitLocalBuffers();
/* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index cb1a8dd34f..3991d7b2ba 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -100,7 +100,7 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Define PG_FLUSH_DATA_WORKS if we have an implementation for pg_flush_data */
#if defined(HAVE_SYNC_FILE_RANGE)
@@ -351,6 +351,24 @@ static void unlink_if_exists_fname(const char *fname, bool isdir, int elevel);
static int fsync_parent_path(const char *fname, int elevel);
+/* ResourceOwner callbacks to hold virtual file descriptors */
+static void ResOwnerReleaseFile(Datum res);
+static void ResOwnerPrintFileLeakWarning(Datum res);
+
+static ResourceOwnerFuncs file_resowner_funcs =
+{
+ .name = "File",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseFile,
+ .PrintLeakWarning = ResOwnerPrintFileLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberFile(owner, file) \
+ ResourceOwnerRemember(owner, Int32GetDatum(file), &file_resowner_funcs)
+#define ResourceOwnerForgetFile(owner, file) \
+ ResourceOwnerForget(owner, Int32GetDatum(file), &file_resowner_funcs)
+
/*
* pg_fsync --- do fsync with or without writethrough
*/
@@ -1530,7 +1548,7 @@ ReportTemporaryFileUsage(const char *path, off_t size)
/*
* Called to register a temporary file for automatic close.
- * ResourceOwnerEnlargeFiles(CurrentResourceOwner) must have been called
+ * ResourceOwnerEnlarge(CurrentResourceOwner) must have been called
* before the file was opened.
*/
static void
@@ -1714,7 +1732,7 @@ OpenTemporaryFile(bool interXact)
* open it, if we'll be registering it below.
*/
if (!interXact)
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* If some temp tablespace(s) have been given to us, try to use the next
@@ -1846,7 +1864,7 @@ PathNameCreateTemporaryFile(const char *path, bool error_on_failure)
Assert(temporary_files_allowed); /* check temp file access is up */
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Open the file. Note: we don't use O_EXCL, in case there is an orphaned
@@ -1886,7 +1904,7 @@ PathNameOpenTemporaryFile(const char *path, int mode)
Assert(temporary_files_allowed); /* check temp file access is up */
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
file = PathNameOpenFile(path, mode | PG_BINARY);
@@ -3889,3 +3907,19 @@ pg_pwritev_with_retry(int fd, const struct iovec *iov, int iovcnt, off_t offset)
return sum;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseFile(Datum res)
+{
+ FileClose((File) DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintFileLeakWarning(Datum res)
+{
+ elog(WARNING, "temporary file leak: File %d still referenced",
+ DatumGetInt32(res));
+}
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index b461a5f7e9..77b6e58fd9 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -37,13 +37,15 @@
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "storage/dsm.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
+#include "storage/shmem.h"
#include "utils/freepage.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#define PG_DYNSHMEM_CONTROL_MAGIC 0x9a503d32
@@ -139,6 +141,25 @@ static dsm_control_header *dsm_control;
static Size dsm_control_mapped_size = 0;
static void *dsm_control_impl_private = NULL;
+
+/* ResourceOwner callbacks to hold DSM segments */
+static void ResOwnerReleaseDSM(Datum res);
+static void ResOwnerPrintDSMLeakWarning(Datum res);
+
+static ResourceOwnerFuncs dsm_resowner_funcs =
+{
+ .name = "dynamic shared memory segment",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseDSM,
+ .PrintLeakWarning = ResOwnerPrintDSMLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberDSM(owner, seg) \
+ ResourceOwnerRemember(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+#define ResourceOwnerForgetDSM(owner, seg) \
+ ResourceOwnerForget(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+
/*
* Start up the dynamic shared memory system.
*
@@ -900,7 +921,7 @@ void
dsm_unpin_mapping(dsm_segment *seg)
{
Assert(seg->resowner == NULL);
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg->resowner = CurrentResourceOwner;
ResourceOwnerRememberDSM(seg->resowner, seg);
}
@@ -1167,7 +1188,7 @@ dsm_create_descriptor(void)
dsm_segment *seg;
if (CurrentResourceOwner)
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg = MemoryContextAlloc(TopMemoryContext, sizeof(dsm_segment));
dlist_push_head(&dsm_segment_list, &seg->node);
@@ -1246,3 +1267,20 @@ is_main_region_dsm_handle(dsm_handle handle)
{
return handle & 1;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseDSM(Datum res)
+{
+ dsm_detach((dsm_segment *) DatumGetPointer(res));
+}
+static void
+ResOwnerPrintDSMLeakWarning(Datum res)
+{
+ dsm_segment *seg = (dsm_segment *) res;
+
+ elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
+ dsm_segment_handle(seg));
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index c25af7fe09..a73abacf68 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -48,7 +48,7 @@
#include "storage/standby.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* This configuration variable is used to set the lock table size */
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 13eed58760..ed046db57e 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -31,12 +31,13 @@
#endif
#include "storage/lmgr.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
@@ -104,6 +105,42 @@ static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
* internal support functions
*/
+/* ResourceOwner callbacks to hold catcache references */
+
+static void ResOwnerReleaseCatCache(Datum res);
+static void ResOwnerPrintCatCacheLeakWarning(Datum res);
+static void ResOwnerReleaseCatCacheList(Datum res);
+static void ResOwnerPrintCatCacheListLeakWarning(Datum res);
+
+static ResourceOwnerFuncs catcache_resowner_funcs =
+{
+ /* catcache references */
+ .name = "catcache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCache,
+ .PrintLeakWarning = ResOwnerPrintCatCacheLeakWarning
+};
+
+static ResourceOwnerFuncs catlistref_resowner_funcs =
+{
+ /* catcache-list pins */
+ .name = "catcache list reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCacheList,
+ .PrintLeakWarning = ResOwnerPrintCatCacheListLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberCatCacheRef(owner, tuple) \
+ ResourceOwnerRemember(owner, PointerGetDatum(tuple), &catcache_resowner_funcs)
+#define ResourceOwnerForgetCatCacheRef(owner, tuple) \
+ ResourceOwnerForget(owner, PointerGetDatum(tuple), &catcache_resowner_funcs)
+#define ResourceOwnerRememberCatCacheListRef(owner, list) \
+ ResourceOwnerRemember(owner, PointerGetDatum(list), &catlistref_resowner_funcs)
+#define ResourceOwnerForgetCatCacheListRef(owner, list) \
+ ResourceOwnerForget(owner, PointerGetDatum(list), &catlistref_resowner_funcs)
+
+
/*
* Hash and equality functions for system types that are used as cache key
* fields. In some cases, we just call the regular SQL-callable functions for
@@ -1272,7 +1309,7 @@ SearchCatCacheInternal(CatCache *cache,
*/
if (!ct->negative)
{
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
@@ -1373,7 +1410,7 @@ SearchCatCacheMiss(CatCache *cache,
hashValue, hashIndex,
false);
/* immediately set the refcount to 1 */
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
break; /* assume only one match */
@@ -1585,7 +1622,7 @@ SearchCatCacheList(CatCache *cache,
dlist_move_head(&cache->cc_lists, &cl->cache_elem);
/* Bump the list's refcount and return it */
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
cl->refcount++;
ResourceOwnerRememberCatCacheListRef(CurrentResourceOwner, cl);
@@ -1697,7 +1734,7 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
/* Make sure the resource owner has room to remember this entry. */
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
@@ -2071,14 +2108,19 @@ PrepareToInvalidateCacheTuple(Relation relation,
}
}
-
/*
- * Subroutines for warning about reference leaks. These are exported so
- * that resowner.c can call them.
+ * ResourceOwner callbacks
*/
-void
-PrintCatCacheLeakWarning(HeapTuple tuple)
+static void
+ResOwnerReleaseCatCache(Datum res)
+{
+ ReleaseCatCache((HeapTuple) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheLeakWarning(Datum res)
{
+ HeapTuple tuple = (HeapTuple) DatumGetPointer(res);
CatCTup *ct = (CatCTup *) (((char *) tuple) -
offsetof(CatCTup, tuple));
@@ -2092,9 +2134,17 @@ PrintCatCacheLeakWarning(HeapTuple tuple)
ct->refcount);
}
-void
-PrintCatCacheListLeakWarning(CatCList *list)
+static void
+ResOwnerReleaseCatCacheList(Datum res)
{
+ ReleaseCatCacheList((CatCList *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheListLeakWarning(Datum res)
+{
+ CatCList *list = (CatCList *) DatumGetPointer(res);
+
elog(WARNING, "cache reference leak: cache %s (%d), list %p has count %d",
list->my_cache->cc_relname, list->my_cache->id,
list, list->refcount);
diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c
index 6767eae8f2..a10362629f 100644
--- a/src/backend/utils/cache/plancache.c
+++ b/src/backend/utils/cache/plancache.c
@@ -69,7 +69,7 @@
#include "tcop/utility.h"
#include "utils/inval.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -115,6 +115,26 @@ static void PlanCacheRelCallback(Datum arg, Oid relid);
static void PlanCacheObjectCallback(Datum arg, int cacheid, uint32 hashvalue);
static void PlanCacheSysCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* ResourceOwner callbacks to track plancache references */
+static void ResOwnerReleaseCachedPlan(Datum res);
+static void ResOwnerPrintPlanCacheLeakWarning(Datum res);
+
+/* this is exported for ResourceOwnerReleaseAllPlanCacheRefs() */
+ResourceOwnerFuncs planref_resowner_funcs =
+{
+ .name = "plancache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCachedPlan,
+ .PrintLeakWarning = ResOwnerPrintPlanCacheLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberPlanCacheRef(owner, plan) \
+ ResourceOwnerRemember(owner, PointerGetDatum(plan), &planref_resowner_funcs)
+#define ResourceOwnerForgetPlanCacheRef(owner, plan) \
+ ResourceOwnerForget(owner, PointerGetDatum(plan), &planref_resowner_funcs)
+
+
/* GUC parameter */
int plan_cache_mode;
@@ -1229,7 +1249,7 @@ GetCachedPlan(CachedPlanSource *plansource, ParamListInfo boundParams,
/* Flag the plan as in use by caller */
if (owner)
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
if (owner)
ResourceOwnerRememberPlanCacheRef(owner, plan);
@@ -1392,7 +1412,7 @@ CachedPlanAllowsSimpleValidityCheck(CachedPlanSource *plansource,
/* Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -1451,7 +1471,7 @@ CachedPlanIsSimplyValid(CachedPlanSource *plansource, CachedPlan *plan,
/* It's still good. Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -2205,3 +2225,20 @@ ResetPlanCache(void)
cexpr->is_valid = false;
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+
+static void
+ResOwnerReleaseCachedPlan(Datum res)
+{
+ ReleaseCachedPlan((CachedPlan *) DatumGetPointer(res), CurrentResourceOwner);
+}
+
+static void
+ResOwnerPrintPlanCacheLeakWarning(Datum res)
+{
+ elog(WARNING, "plancache reference leak: plan %p not closed",
+ DatumGetPointer(res));
+}
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index 9fa9e671a1..1f7382ad32 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -77,13 +77,14 @@
#include "storage/smgr.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/relmapper.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -2107,6 +2108,24 @@ RelationIdGetRelation(Oid relationId)
* ----------------------------------------------------------------
*/
+/* ResourceOwner callbacks to track relcache references */
+static void ResOwnerReleaseRelation(Datum res);
+static void ResOwnerPrintRelCacheLeakWarning(Datum res);
+
+static ResourceOwnerFuncs relref_resowner_funcs =
+{
+ .name = "relcache reference",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseRelation,
+ .PrintLeakWarning = ResOwnerPrintRelCacheLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberRelationRef(owner, rel) \
+ ResourceOwnerRemember(owner, PointerGetDatum(rel), &relref_resowner_funcs)
+#define ResourceOwnerForgetRelationRef(owner, rel) \
+ ResourceOwnerForget(owner, PointerGetDatum(rel), &relref_resowner_funcs)
+
/*
* RelationIncrementReferenceCount
* Increments relation reference count.
@@ -2118,7 +2137,7 @@ RelationIdGetRelation(Oid relationId)
void
RelationIncrementReferenceCount(Relation rel)
{
- ResourceOwnerEnlargeRelationRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
rel->rd_refcnt += 1;
if (!IsBootstrapProcessingMode())
ResourceOwnerRememberRelationRef(CurrentResourceOwner, rel);
@@ -6656,3 +6675,21 @@ unlink_initfile(const char *initfilename, int elevel)
initfilename)));
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerPrintRelCacheLeakWarning(Datum res)
+{
+ Relation rel = (Relation) res;
+
+ elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
+ RelationGetRelationName(rel));
+}
+
+static void
+ResOwnerReleaseRelation(Datum res)
+{
+ RelationClose((Relation) res);
+}
diff --git a/src/backend/utils/resowner/README b/src/backend/utils/resowner/README
index f94c9700df..adaa4a9d00 100644
--- a/src/backend/utils/resowner/README
+++ b/src/backend/utils/resowner/README
@@ -54,12 +54,17 @@ The basic operations on a ResourceOwner are:
* delete a ResourceOwner (including child owner objects); all resources
must have been released beforehand
-This API directly supports the resource types listed in the definition of
-ResourceOwnerData struct in src/backend/utils/resowner/resowner.c.
-Other objects can be associated with a ResourceOwner by recording the address
-of the owning ResourceOwner in such an object. There is an API for other
-modules to get control during ResourceOwner release, so that they can scan
-their own data structures to find the objects that need to be deleted.
+ResourceOwner can record ownership of many different kinds of resources. In
+core PostgreSQL, it is used for buffer pins, lmgr locks, and catalog cache
+references, to name a few examples. ResourceOwner treats all resources the
+same, and extensions can define new kinds of resources by filling in a
+ResourceOwnerFuncs struct with suitable callback functions.
+
+There is also an API for other modules to get control during ResourceOwner
+release, so that they can scan their own data structures to find the objects
+that need to be deleted. This used to be the only way to register new kinds
+of objects with a resource owner; nowadays it easier to write custom
+ResourceOwnerFuncs callbacks.
Locks are handled specially because in non-error situations a lock should
be held until end of transaction, even if it was originally taken by a
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index e24f00f060..35e824d453 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -20,76 +20,45 @@
*/
#include "postgres.h"
-#include "common/cryptohash.h"
#include "common/hashfn.h"
-#include "common/hmac.h"
-#include "jit/jit.h"
-#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
#include "utils/memutils.h"
-#include "utils/rel.h"
-#include "utils/resowner_private.h"
-#include "utils/snapmgr.h"
-
-
-/*
- * All resource IDs managed by this code are required to fit into a Datum,
- * which is fine since they are generally pointers or integers.
- *
- * Provide Datum conversion macros for a couple of things that are really
- * just "int".
- */
-#define FileGetDatum(file) Int32GetDatum(file)
-#define DatumGetFile(datum) ((File) DatumGetInt32(datum))
-#define BufferGetDatum(buffer) Int32GetDatum(buffer)
-#define DatumGetBuffer(datum) ((Buffer) DatumGetInt32(datum))
+#include "utils/plancache.h"
+#include "utils/resowner.h"
/*
- * ResourceArray is a common structure for storing all types of resource IDs.
- *
- * We manage small sets of resource IDs by keeping them in a simple array:
- * itemsarr[k] holds an ID, for 0 <= k < nitems <= maxitems = capacity.
- *
- * If a set grows large, we switch over to using open-addressing hashing.
- * Then, itemsarr[] is a hash table of "capacity" slots, with each
- * slot holding either an ID or "invalidval". nitems is the number of valid
- * items present; if it would exceed maxitems, we enlarge the array and
- * re-hash. In this mode, maxitems should be rather less than capacity so
- * that we don't waste too much time searching for empty slots.
+ * ResourceElem represents a reference associated with a resource owner.
*
- * In either mode, lastidx remembers the location of the last item inserted
- * or returned by GetAny; this speeds up searches in ResourceArrayRemove.
+ * All objects managed by this code are required to fit into a Datum,
+ * which is fine since they are generally pointers or integers.
*/
-typedef struct ResourceArray
+typedef struct ResourceElem
{
- Datum *itemsarr; /* buffer for storing values */
- Datum invalidval; /* value that is considered invalid */
- uint32 capacity; /* allocated length of itemsarr[] */
- uint32 nitems; /* how many items are stored in items array */
- uint32 maxitems; /* current limit on nitems before enlarging */
- uint32 lastidx; /* index of last item returned by GetAny */
-} ResourceArray;
+ Datum item;
+ ResourceOwnerFuncs *kind; /* NULL indicates a free hash table slot */
+} ResourceElem;
/*
- * Initially allocated size of a ResourceArray. Must be power of two since
- * we'll use (arraysize - 1) as mask for hashing.
+ * Size of the small fixed-size array to hold most-recently remembered resources.
*/
-#define RESARRAY_INIT_SIZE 16
+#define RESOWNER_ARRAY_SIZE 8
/*
- * When to switch to hashing vs. simple array logic in a ResourceArray.
+ * Initially allocated size of a ResourceOwner's hash. Must be power of two since
+ * we'll use (capacity - 1) as mask for hashing.
*/
-#define RESARRAY_MAX_ARRAY 64
-#define RESARRAY_IS_ARRAY(resarr) ((resarr)->capacity <= RESARRAY_MAX_ARRAY)
+#define RESOWNER_HASH_INIT_SIZE 32
/*
- * How many items may be stored in a resource array of given capacity.
+ * How many items may be stored in a hash of given capacity.
* When this number is reached, we must resize.
*/
-#define RESARRAY_MAX_ITEMS(capacity) \
- ((capacity) <= RESARRAY_MAX_ARRAY ? (capacity) : (capacity)/4 * 3)
+#define RESOWNER_HASH_MAX_ITEMS(capacity) ((capacity)/4 * 3)
+
+StaticAssertDecl(RESOWNER_HASH_MAX_ITEMS(RESOWNER_HASH_INIT_SIZE) > RESOWNER_ARRAY_SIZE,
+ "initial hash size too small compared to array size");
/*
* To speed up bulk releasing or reassigning locks from a resource owner to
@@ -119,24 +88,33 @@ typedef struct ResourceOwnerData
ResourceOwner nextchild; /* next child of same parent */
const char *name; /* name (just for debugging) */
- /* We have built-in support for remembering: */
- ResourceArray bufferarr; /* owned buffers */
- ResourceArray catrefarr; /* catcache references */
- ResourceArray catlistrefarr; /* catcache-list pins */
- ResourceArray relrefarr; /* relcache references */
- ResourceArray planrefarr; /* plancache references */
- ResourceArray tupdescarr; /* tupdesc references */
- ResourceArray snapshotarr; /* snapshot references */
- ResourceArray filearr; /* open temporary files */
- ResourceArray dsmarr; /* dynamic shmem segments */
- ResourceArray jitarr; /* JIT contexts */
- ResourceArray cryptohasharr; /* cryptohash contexts */
- ResourceArray hmacarr; /* HMAC contexts */
+ /*
+ * These structs keep track of the objects registered with this owner.
+ *
+ * We manage a small set of references by keeping them in a simple array.
+ * When the array gets full, all the elements in the array are moved to a
+ * hash table. This way, the array always contains a few most recently
+ * remembered references. To find a particular reference, you need to
+ * search both the array and the hash table.
+ */
+ ResourceElem arr[RESOWNER_ARRAY_SIZE];
+ uint32 narr; /* how many items are stored in the array */
+
+ /*
+ * The hash table. Uses open-addressing. 'nhash' is the number of items
+ * present; if it would exceed 'grow_at', we enlarge it and re-hash.
+ * 'grow_at' should be rather less than 'capacity' so that we don't waste
+ * too much time searching for empty slots.
+ */
+ ResourceElem *hash;
+ uint32 nhash; /* how many items are stored in the hash */
+ uint32 capacity; /* allocated length of hash[] */
+ uint32 grow_at; /* grow hash when reach this */
/* We can remember up to MAX_RESOWNER_LOCKS references to local locks. */
int nlocks; /* number of owned locks */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
-} ResourceOwnerData;
+} ResourceOwnerData;
/*****************************************************************************
@@ -148,6 +126,18 @@ ResourceOwner CurTransactionResourceOwner = NULL;
ResourceOwner TopTransactionResourceOwner = NULL;
ResourceOwner AuxProcessResourceOwner = NULL;
+/* #define RESOWNER_STATS */
+/* #define RESOWNER_TRACE */
+
+#ifdef RESOWNER_STATS
+static int narray_lookups = 0;
+static int nhash_lookups = 0;
+#endif
+
+#ifdef RESOWNER_TRACE
+static int resowner_trace_counter = 0;
+#endif
+
/*
* List of add-on callbacks for resource releasing
*/
@@ -162,298 +152,332 @@ static ResourceReleaseCallbackItem *ResourceRelease_callbacks = NULL;
/* Internal routines */
-static void ResourceArrayInit(ResourceArray *resarr, Datum invalidval);
-static void ResourceArrayEnlarge(ResourceArray *resarr);
-static void ResourceArrayAdd(ResourceArray *resarr, Datum value);
-static bool ResourceArrayRemove(ResourceArray *resarr, Datum value);
-static bool ResourceArrayGetAny(ResourceArray *resarr, Datum *value);
-static void ResourceArrayFree(ResourceArray *resarr);
+static inline uint32 hash_resource_elem(Datum value, ResourceOwnerFuncs *kind);
+static void ResourceOwnerAddToHash(ResourceOwner owner, Datum value,
+ ResourceOwnerFuncs *kind);
+static void ResourceOwnerReleaseAll(ResourceOwner owner,
+ ResourceReleasePhase phase,
+ bool printLeakWarnings);
static void ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
-static void PrintRelCacheLeakWarning(Relation rel);
-static void PrintPlanCacheLeakWarning(CachedPlan *plan);
-static void PrintTupleDescLeakWarning(TupleDesc tupdesc);
-static void PrintSnapshotLeakWarning(Snapshot snapshot);
-static void PrintFileLeakWarning(File file);
-static void PrintDSMLeakWarning(dsm_segment *seg);
-static void PrintCryptoHashLeakWarning(Datum handle);
-static void PrintHMACLeakWarning(Datum handle);
/*****************************************************************************
* INTERNAL ROUTINES *
*****************************************************************************/
+static inline uint32
+hash_resource_elem(Datum value, ResourceOwnerFuncs *kind)
+{
+ Datum data[2];
+
+ data[0] = value;
+ data[1] = PointerGetDatum(kind);
+
+ return hash_bytes((unsigned char *) &data, 2 * SIZEOF_DATUM);
+}
/*
- * Initialize a ResourceArray
+ * Adds 'value' of given 'kind' to the ResourceOwner's hash table
*/
static void
-ResourceArrayInit(ResourceArray *resarr, Datum invalidval)
+ResourceOwnerAddToHash(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- /* Assert it's empty */
- Assert(resarr->itemsarr == NULL);
- Assert(resarr->capacity == 0);
- Assert(resarr->nitems == 0);
- Assert(resarr->maxitems == 0);
- /* Remember the appropriate "invalid" value */
- resarr->invalidval = invalidval;
- /* We don't allocate any storage until needed */
+ /* Insert into first free slot at or after hash location. */
+ uint32 mask = owner->capacity - 1;
+ uint32 idx;
+
+ Assert(kind != NULL);
+
+ idx = hash_resource_elem(value, kind) & mask;
+ for (;;)
+ {
+ if (owner->hash[idx].kind == NULL)
+ break; /* found a free slot */
+ idx = (idx + 1) & mask;
+ }
+ owner->hash[idx].item = value;
+ owner->hash[idx].kind = kind;
+ owner->nhash++;
}
/*
- * Make sure there is room for at least one more resource in an array.
- *
- * This is separate from actually inserting a resource because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
+ * Call the ReleaseResource callback on entries with given 'phase'.
*/
static void
-ResourceArrayEnlarge(ResourceArray *resarr)
+ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
+ bool printLeakWarnings)
{
- uint32 i,
- oldcap,
- newcap;
- Datum *olditemsarr;
- Datum *newitemsarr;
+ bool found;
+ int capacity;
- if (resarr->nitems < resarr->maxitems)
- return; /* no work needed */
-
- olditemsarr = resarr->itemsarr;
- oldcap = resarr->capacity;
-
- /* Double the capacity of the array (capacity must stay a power of 2!) */
- newcap = (oldcap > 0) ? oldcap * 2 : RESARRAY_INIT_SIZE;
- newitemsarr = (Datum *) MemoryContextAlloc(TopMemoryContext,
- newcap * sizeof(Datum));
- for (i = 0; i < newcap; i++)
- newitemsarr[i] = resarr->invalidval;
+ /*
+ * First handle all the entries in the array.
+ *
+ * Note that ReleaseResource() will call ResourceOwnerForget) and remove
+ * the entry from our array, so we just have to iterate till there is
+ * nothing left to remove.
+ */
+ do
+ {
+ found = false;
+ for (int i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].kind->phase == phase)
+ {
+ Datum value = owner->arr[i].item;
+ ResourceOwnerFuncs *kind = owner->arr[i].kind;
- /* We assume we can't fail below this point, so OK to scribble on resarr */
- resarr->itemsarr = newitemsarr;
- resarr->capacity = newcap;
- resarr->maxitems = RESARRAY_MAX_ITEMS(newcap);
- resarr->nitems = 0;
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+ found = true;
+ }
+ }
- if (olditemsarr != NULL)
- {
/*
- * Transfer any pre-existing entries into the new array; they don't
- * necessarily go where they were before, so this simple logic is the
- * best way. Note that if we were managing the set as a simple array,
- * the entries after nitems are garbage, but that shouldn't matter
- * because we won't get here unless nitems was equal to oldcap.
+ * If any resources were released, check again because some of the
+ * elements might have been moved by the callbacks. We don't want to
+ * miss them.
*/
- for (i = 0; i < oldcap; i++)
+ } while (found && owner->narr > 0);
+
+ /*
+ * Ok, the array has now been handled. Then the hash. Like with the
+ * array, ReleaseResource() will remove the entry from the hash.
+ */
+ do
+ {
+ capacity = owner->capacity;
+ for (int idx = 0; idx < capacity; idx++)
{
- if (olditemsarr[i] != resarr->invalidval)
- ResourceArrayAdd(resarr, olditemsarr[i]);
+ while (owner->hash[idx].kind != NULL &&
+ owner->hash[idx].kind->phase == phase)
+ {
+ Datum value = owner->hash[idx].item;
+ ResourceOwnerFuncs *kind = owner->hash[idx].kind;
+
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+
+ /*
+ * If the same resource is remembered more than once in this
+ * resource owner, the ReleaseResource callback might've
+ * released a different copy of it. Because of that, loop to
+ * check the same index again.
+ */
+ }
}
- /* And release old array. */
- pfree(olditemsarr);
- }
-
- Assert(resarr->nitems < resarr->maxitems);
+ /*
+ * It's possible that the callbacks acquired more resources, causing
+ * the hash table to grow and the existing entries to be moved around.
+ * If that happened, scan the hash table again, so that we don't miss
+ * entries that were moved. (XXX: I'm not sure if any of the
+ * callbacks actually do that, but this is cheap to check, and better
+ * safe than sorry.)
+ */
+ Assert(owner->capacity >= capacity);
+ } while (capacity != owner->capacity);
}
+
+/*****************************************************************************
+ * EXPORTED ROUTINES *
+ *****************************************************************************/
+
+
/*
- * Add a resource to ResourceArray
+ * ResourceOwnerCreate
+ * Create an empty ResourceOwner.
*
- * Caller must have previously done ResourceArrayEnlarge()
+ * All ResourceOwner objects are kept in TopMemoryContext, since they should
+ * only be freed explicitly.
*/
-static void
-ResourceArrayAdd(ResourceArray *resarr, Datum value)
+ResourceOwner
+ResourceOwnerCreate(ResourceOwner parent, const char *name)
{
- uint32 idx;
+ ResourceOwner owner;
- Assert(value != resarr->invalidval);
- Assert(resarr->nitems < resarr->maxitems);
+ owner = (ResourceOwner) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(ResourceOwnerData));
+ owner->name = name;
- if (RESARRAY_IS_ARRAY(resarr))
+ if (parent)
{
- /* Append to linear array. */
- idx = resarr->nitems;
+ owner->parent = parent;
+ owner->nextchild = parent->firstchild;
+ parent->firstchild = owner;
}
- else
- {
- /* Insert into first free slot at or after hash location. */
- uint32 mask = resarr->capacity - 1;
- idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (;;)
- {
- if (resarr->itemsarr[idx] == resarr->invalidval)
- break;
- idx = (idx + 1) & mask;
- }
- }
- resarr->lastidx = idx;
- resarr->itemsarr[idx] = value;
- resarr->nitems++;
+#ifdef RESOWNER_TRACE
+ elog(LOG, "CREATE %d: %p %s",
+ resowner_trace_counter++, owner, name);
+#endif
+
+ return owner;
}
/*
- * Remove a resource from ResourceArray
- *
- * Returns true on success, false if resource was not found.
+ * Make sure there is room for at least one more resource in an array.
*
- * Note: if same resource ID appears more than once, one instance is removed.
+ * This is separate from actually inserting a resource because if we run out
+ * of memory, it's critical to do so *before* acquiring the resource.
*/
-static bool
-ResourceArrayRemove(ResourceArray *resarr, Datum value)
+void
+ResourceOwnerEnlarge(ResourceOwner owner)
{
- uint32 i,
- idx,
- lastidx = resarr->lastidx;
-
- Assert(value != resarr->invalidval);
+ if (owner->narr < RESOWNER_ARRAY_SIZE)
+ return; /* no work needed */
- /* Search through all items, but try lastidx first. */
- if (RESARRAY_IS_ARRAY(resarr))
+ /* Is there space in the hash? If not, enlarge it. */
+ if (owner->narr + owner->nhash >= owner->grow_at)
{
- if (lastidx < resarr->nitems &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
- }
- for (i = 0; i < resarr->nitems; i++)
+ uint32 i,
+ oldcap,
+ newcap;
+ ResourceElem *oldhash;
+ ResourceElem *newhash;
+
+ oldhash = owner->hash;
+ oldcap = owner->capacity;
+
+ /* Double the capacity (it must stay a power of 2!) */
+ newcap = (oldcap > 0) ? oldcap * 2 : RESOWNER_HASH_INIT_SIZE;
+ newhash = (ResourceElem *) MemoryContextAllocZero(TopMemoryContext,
+ newcap * sizeof(ResourceElem));
+
+ /*
+ * We assume we can't fail below this point, so OK to scribble on the
+ * owner
+ */
+ owner->hash = newhash;
+ owner->capacity = newcap;
+ owner->grow_at = RESOWNER_HASH_MAX_ITEMS(newcap);
+ owner->nhash = 0;
+
+ if (oldhash != NULL)
{
- if (resarr->itemsarr[i] == value)
+ /*
+ * Transfer any pre-existing entries into the new hash table; they
+ * don't necessarily go where they were before, so this simple
+ * logic is the best way.
+ */
+ for (i = 0; i < oldcap; i++)
{
- resarr->itemsarr[i] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
+ if (oldhash[i].kind != NULL)
+ ResourceOwnerAddToHash(owner, oldhash[i].item, oldhash[i].kind);
}
+
+ /* And release old hash table. */
+ pfree(oldhash);
}
}
- else
- {
- uint32 mask = resarr->capacity - 1;
- if (lastidx < resarr->capacity &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->invalidval;
- resarr->nitems--;
- return true;
- }
- idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (i = 0; i < resarr->capacity; i++)
- {
- if (resarr->itemsarr[idx] == value)
- {
- resarr->itemsarr[idx] = resarr->invalidval;
- resarr->nitems--;
- return true;
- }
- idx = (idx + 1) & mask;
- }
+ /* Move items from the array to the hash */
+ Assert(owner->narr == RESOWNER_ARRAY_SIZE);
+ for (int i = 0; i < owner->narr; i++)
+ {
+ ResourceOwnerAddToHash(owner, owner->arr[i].item, owner->arr[i].kind);
}
+ owner->narr = 0;
- return false;
+ Assert(owner->nhash < owner->grow_at);
}
/*
- * Get any convenient entry in a ResourceArray.
+ * Remember that an object is owner by a ReourceOwner
*
- * "Convenient" is defined as "easy for ResourceArrayRemove to remove";
- * we help that along by setting lastidx to match. This avoids O(N^2) cost
- * when removing all ResourceArray items during ResourceOwner destruction.
- *
- * Returns true if we found an element, or false if the array is empty.
+ * Caller must have previously done ResourceOwnerEnlarge()
*/
-static bool
-ResourceArrayGetAny(ResourceArray *resarr, Datum *value)
+void
+ResourceOwnerRemember(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- if (resarr->nitems == 0)
- return false;
+ uint32 idx;
- if (RESARRAY_IS_ARRAY(resarr))
- {
- /* Linear array: just return the first element. */
- resarr->lastidx = 0;
- }
- else
- {
- /* Hash: search forward from wherever we were last. */
- uint32 mask = resarr->capacity - 1;
+#ifdef RESOWNER_TRACE
+ elog(LOG, "REMEMBER %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
- for (;;)
- {
- resarr->lastidx &= mask;
- if (resarr->itemsarr[resarr->lastidx] != resarr->invalidval)
- break;
- resarr->lastidx++;
- }
- }
+ Assert(owner->narr < RESOWNER_ARRAY_SIZE);
- *value = resarr->itemsarr[resarr->lastidx];
- return true;
+ /* Append to linear array. */
+ idx = owner->narr;
+ owner->arr[idx].item = value;
+ owner->arr[idx].kind = kind;
+ owner->narr++;
}
/*
- * Trash a ResourceArray (we don't care about its state after this)
+ * Forget that an object is owned by a ResourceOwner
+ *
+ * Note: if same resource ID is associated with the ResourceOwner more than once,
+ * one instance is removed.
*/
-static void
-ResourceArrayFree(ResourceArray *resarr)
+void
+ResourceOwnerForget(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- if (resarr->itemsarr)
- pfree(resarr->itemsarr);
-}
-
+ uint32 i,
+ idx;
-/*****************************************************************************
- * EXPORTED ROUTINES *
- *****************************************************************************/
+#ifdef RESOWNER_TRACE
+ elog(LOG, "FORGET %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
+ /* Search through all items, but check the array first. */
+ for (i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].item == value &&
+ owner->arr[i].kind == kind)
+ {
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
-/*
- * ResourceOwnerCreate
- * Create an empty ResourceOwner.
- *
- * All ResourceOwner objects are kept in TopMemoryContext, since they should
- * only be freed explicitly.
- */
-ResourceOwner
-ResourceOwnerCreate(ResourceOwner parent, const char *name)
-{
- ResourceOwner owner;
+#ifdef RESOWNER_STATS
+ narray_lookups++;
+#endif
- owner = (ResourceOwner) MemoryContextAllocZero(TopMemoryContext,
- sizeof(ResourceOwnerData));
- owner->name = name;
+ return;
+ }
+ }
- if (parent)
+ /* Search hash */
+ if (owner->nhash > 0)
{
- owner->parent = parent;
- owner->nextchild = parent->firstchild;
- parent->firstchild = owner;
- }
+ uint32 mask = owner->capacity - 1;
- ResourceArrayInit(&(owner->bufferarr), BufferGetDatum(InvalidBuffer));
- ResourceArrayInit(&(owner->catrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->catlistrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->relrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->planrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->tupdescarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->snapshotarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->filearr), FileGetDatum(-1));
- ResourceArrayInit(&(owner->dsmarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->jitarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->cryptohasharr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->hmacarr), PointerGetDatum(NULL));
+ idx = hash_resource_elem(value, kind) & mask;
+ for (i = 0; i < owner->capacity; i++)
+ {
+ if (owner->hash[idx].item == value &&
+ owner->hash[idx].kind == kind)
+ {
+ owner->hash[idx].item = (Datum) 0;
+ owner->hash[idx].kind = NULL;
+ owner->nhash--;
+
+#ifdef RESOWNER_STATS
+ nhash_lookups++;
+#endif
+ return;
+ }
+ idx = (idx + 1) & mask;
+ }
+ }
- return owner;
+ /*
+ * Use %p to print the reference, since most objects tracked by a resource
+ * owner are pointers. It's a bit misleading if it's not a pointer, but
+ * this is a programmer error, anyway.
+ */
+ elog(ERROR, "%s %p is not owned by resource owner %s",
+ kind->name, DatumGetPointer(value), owner->name);
}
/*
@@ -490,6 +514,15 @@ ResourceOwnerRelease(ResourceOwner owner,
{
/* There's not currently any setup needed before recursing */
ResourceOwnerReleaseInternal(owner, phase, isCommit, isTopLevel);
+
+#ifdef RESOWNER_STATS
+ if (isTopLevel)
+ {
+ elog(LOG, "RESOWNER STATS: lookups: array %d, hash %d", narray_lookups, nhash_lookups);
+ narray_lookups = 0;
+ nhash_lookups = 0;
+ }
+#endif
}
static void
@@ -501,7 +534,6 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceOwner child;
ResourceOwner save;
ResourceReleaseCallbackItem *item;
- Datum foundres;
/* Recurse to handle descendants */
for (child = owner->firstchild; child != NULL; child = child->nextchild)
@@ -517,71 +549,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
if (phase == RESOURCE_RELEASE_BEFORE_LOCKS)
{
/*
- * Release buffer pins. Note that ReleaseBuffer will remove the
- * buffer entry from our array, so we just have to iterate till there
- * are none.
+ * Release all resources that need to be released before the locks.
*
- * During a commit, there shouldn't be any remaining pins --- that
- * would indicate failure to clean up the executor correctly --- so
- * issue warnings. In the abort case, just clean up quietly.
+ * During a commit, there shouldn't be any remaining resources ---
+ * that would indicate failure to clean up the executor correctly ---
+ * so issue warnings. In the abort case, just clean up quietly.
*/
- while (ResourceArrayGetAny(&(owner->bufferarr), &foundres))
- {
- Buffer res = DatumGetBuffer(foundres);
-
- if (isCommit)
- PrintBufferLeakWarning(res);
- ReleaseBuffer(res);
- }
-
- /* Ditto for relcache references */
- while (ResourceArrayGetAny(&(owner->relrefarr), &foundres))
- {
- Relation res = (Relation) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintRelCacheLeakWarning(res);
- RelationClose(res);
- }
-
- /* Ditto for dynamic shared memory segments */
- while (ResourceArrayGetAny(&(owner->dsmarr), &foundres))
- {
- dsm_segment *res = (dsm_segment *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintDSMLeakWarning(res);
- dsm_detach(res);
- }
-
- /* Ditto for JIT contexts */
- while (ResourceArrayGetAny(&(owner->jitarr), &foundres))
- {
- JitContext *context = (JitContext *) PointerGetDatum(foundres);
-
- jit_release_context(context);
- }
-
- /* Ditto for cryptohash contexts */
- while (ResourceArrayGetAny(&(owner->cryptohasharr), &foundres))
- {
- pg_cryptohash_ctx *context =
- (pg_cryptohash_ctx *) PointerGetDatum(foundres);
-
- if (isCommit)
- PrintCryptoHashLeakWarning(foundres);
- pg_cryptohash_free(context);
- }
-
- /* Ditto for HMAC contexts */
- while (ResourceArrayGetAny(&(owner->hmacarr), &foundres))
- {
- pg_hmac_ctx *context = (pg_hmac_ctx *) PointerGetDatum(foundres);
-
- if (isCommit)
- PrintHMACLeakWarning(foundres);
- pg_hmac_free(context);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -634,70 +608,9 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
else if (phase == RESOURCE_RELEASE_AFTER_LOCKS)
{
/*
- * Release catcache references. Note that ReleaseCatCache will remove
- * the catref entry from our array, so we just have to iterate till
- * there are none.
- *
- * As with buffer pins, warn if any are left at commit time.
+ * Release all resources that need to be released after the locks.
*/
- while (ResourceArrayGetAny(&(owner->catrefarr), &foundres))
- {
- HeapTuple res = (HeapTuple) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheLeakWarning(res);
- ReleaseCatCache(res);
- }
-
- /* Ditto for catcache lists */
- while (ResourceArrayGetAny(&(owner->catlistrefarr), &foundres))
- {
- CatCList *res = (CatCList *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheListLeakWarning(res);
- ReleaseCatCacheList(res);
- }
-
- /* Ditto for plancache references */
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
- {
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintPlanCacheLeakWarning(res);
- ReleaseCachedPlan(res, owner);
- }
-
- /* Ditto for tupdesc references */
- while (ResourceArrayGetAny(&(owner->tupdescarr), &foundres))
- {
- TupleDesc res = (TupleDesc) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintTupleDescLeakWarning(res);
- DecrTupleDescRefCount(res);
- }
-
- /* Ditto for snapshot references */
- while (ResourceArrayGetAny(&(owner->snapshotarr), &foundres))
- {
- Snapshot res = (Snapshot) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintSnapshotLeakWarning(res);
- UnregisterSnapshot(res);
- }
-
- /* Ditto for temporary files */
- while (ResourceArrayGetAny(&(owner->filearr), &foundres))
- {
- File res = DatumGetFile(foundres);
-
- if (isCommit)
- PrintFileLeakWarning(res);
- FileClose(res);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
/* Let add-on modules get a chance too */
@@ -717,13 +630,42 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
void
ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner)
{
- Datum foundres;
+ /* array first */
+ for (int i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->arr[i].item);
+
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
+ i--;
+
+ /*
+ * pass 'NULL' because we already removed the entry from the
+ * resowner
+ */
+ ReleaseCachedPlan(planref, NULL);
+ }
+ }
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
+ /* Then hash */
+ for (int i = 0; i < owner->capacity; i++)
{
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
+ if (owner->hash[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->hash[i].item);
- ReleaseCachedPlan(res, owner);
+ owner->hash[i].item = (Datum) 0;
+ owner->hash[i].kind = NULL;
+ owner->nhash--;
+
+ /*
+ * pass 'NULL' because we already removed the entry from the
+ * resowner
+ */
+ ReleaseCachedPlan(planref, NULL);
+ }
}
}
@@ -740,20 +682,15 @@ ResourceOwnerDelete(ResourceOwner owner)
Assert(owner != CurrentResourceOwner);
/* And it better not own any resources, either */
- Assert(owner->bufferarr.nitems == 0);
- Assert(owner->catrefarr.nitems == 0);
- Assert(owner->catlistrefarr.nitems == 0);
- Assert(owner->relrefarr.nitems == 0);
- Assert(owner->planrefarr.nitems == 0);
- Assert(owner->tupdescarr.nitems == 0);
- Assert(owner->snapshotarr.nitems == 0);
- Assert(owner->filearr.nitems == 0);
- Assert(owner->dsmarr.nitems == 0);
- Assert(owner->jitarr.nitems == 0);
- Assert(owner->cryptohasharr.nitems == 0);
- Assert(owner->hmacarr.nitems == 0);
+ Assert(owner->narr == 0);
+ Assert(owner->nhash == 0);
Assert(owner->nlocks == 0 || owner->nlocks == MAX_RESOWNER_LOCKS + 1);
+#ifdef RESOWNER_TRACE
+ elog(LOG, "DELETE %d: %p %s",
+ resowner_trace_counter++, owner, owner->name);
+#endif
+
/*
* Delete children. The recursive call will delink the child from me, so
* just iterate as long as there is a child.
@@ -769,19 +706,8 @@ ResourceOwnerDelete(ResourceOwner owner)
ResourceOwnerNewParent(owner, NULL);
/* And free the object. */
- ResourceArrayFree(&(owner->bufferarr));
- ResourceArrayFree(&(owner->catrefarr));
- ResourceArrayFree(&(owner->catlistrefarr));
- ResourceArrayFree(&(owner->relrefarr));
- ResourceArrayFree(&(owner->planrefarr));
- ResourceArrayFree(&(owner->tupdescarr));
- ResourceArrayFree(&(owner->snapshotarr));
- ResourceArrayFree(&(owner->filearr));
- ResourceArrayFree(&(owner->dsmarr));
- ResourceArrayFree(&(owner->jitarr));
- ResourceArrayFree(&(owner->cryptohasharr));
- ResourceArrayFree(&(owner->hmacarr));
-
+ if (owner->hash)
+ pfree(owner->hash);
pfree(owner);
}
@@ -839,11 +765,10 @@ ResourceOwnerNewParent(ResourceOwner owner,
/*
* Register or deregister callback functions for resource cleanup
*
- * These functions are intended for use by dynamically loaded modules.
- * For built-in modules we generally just hardwire the appropriate calls.
- *
- * Note that the callback occurs post-commit or post-abort, so the callback
- * functions can only do noncritical cleanup.
+ * These functions can be used by dynamically loaded modules. These used
+ * to be the only way for an extension to register custom resource types
+ * with a resource owner, but nowadays it is easier to define a new
+ * ResourceOwnerFuncs instance with custom callbacks.
*/
void
RegisterResourceReleaseCallback(ResourceReleaseCallback callback, void *arg)
@@ -934,58 +859,20 @@ ReleaseAuxProcessResourcesCallback(int code, Datum arg)
ReleaseAuxProcessResources(isCommit);
}
-
/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * buffer array.
+ * Remember that a Local Lock is owned by a ResourceOwner
*
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
+ * This is different from the other Remember functions in that the list of
+ * locks is only a lossy cache. It can hold up to MAX_RESOWNER_LOCKS entries,
+ * and when it overflows, we stop tracking locks. The point of only remembering
+ * only up to MAX_RESOWNER_LOCKS entries is that if a lot of locks are held,
+ * ResourceOwnerForgetLock doesn't need to scan through a large array to find
+ * the entry.
*/
void
-ResourceOwnerEnlargeBuffers(ResourceOwner owner)
+ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock)
{
- /* We used to allow pinning buffers without a resowner, but no more */
- Assert(owner != NULL);
- ResourceArrayEnlarge(&(owner->bufferarr));
-}
-
-/*
- * Remember that a buffer pin is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeBuffers()
- */
-void
-ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer)
-{
- ResourceArrayAdd(&(owner->bufferarr), BufferGetDatum(buffer));
-}
-
-/*
- * Forget that a buffer pin is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer)
-{
- if (!ResourceArrayRemove(&(owner->bufferarr), BufferGetDatum(buffer)))
- elog(ERROR, "buffer %d is not owned by resource owner %s",
- buffer, owner->name);
-}
-
-/*
- * Remember that a Local Lock is owned by a ResourceOwner
- *
- * This is different from the other Remember functions in that the list of
- * locks is only a lossy cache. It can hold up to MAX_RESOWNER_LOCKS entries,
- * and when it overflows, we stop tracking locks. The point of only remembering
- * only up to MAX_RESOWNER_LOCKS entries is that if a lot of locks are held,
- * ResourceOwnerForgetLock doesn't need to scan through a large array to find
- * the entry.
- */
-void
-ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock)
-{
- Assert(locallock != NULL);
+ Assert(locallock != NULL);
if (owner->nlocks > MAX_RESOWNER_LOCKS)
return; /* we have already overflowed */
@@ -1023,469 +910,3 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catrefarr));
-}
-
-/*
- * Remember that a catcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheRefs()
- */
-void
-ResourceOwnerRememberCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- ResourceArrayAdd(&(owner->catrefarr), PointerGetDatum(tuple));
-}
-
-/*
- * Forget that a catcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- if (!ResourceArrayRemove(&(owner->catrefarr), PointerGetDatum(tuple)))
- elog(ERROR, "catcache reference %p is not owned by resource owner %s",
- tuple, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache-list reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catlistrefarr));
-}
-
-/*
- * Remember that a catcache-list reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheListRefs()
- */
-void
-ResourceOwnerRememberCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- ResourceArrayAdd(&(owner->catlistrefarr), PointerGetDatum(list));
-}
-
-/*
- * Forget that a catcache-list reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- if (!ResourceArrayRemove(&(owner->catlistrefarr), PointerGetDatum(list)))
- elog(ERROR, "catcache list reference %p is not owned by resource owner %s",
- list, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * relcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeRelationRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->relrefarr));
-}
-
-/*
- * Remember that a relcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeRelationRefs()
- */
-void
-ResourceOwnerRememberRelationRef(ResourceOwner owner, Relation rel)
-{
- ResourceArrayAdd(&(owner->relrefarr), PointerGetDatum(rel));
-}
-
-/*
- * Forget that a relcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetRelationRef(ResourceOwner owner, Relation rel)
-{
- if (!ResourceArrayRemove(&(owner->relrefarr), PointerGetDatum(rel)))
- elog(ERROR, "relcache reference %s is not owned by resource owner %s",
- RelationGetRelationName(rel), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintRelCacheLeakWarning(Relation rel)
-{
- elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
- RelationGetRelationName(rel));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * plancache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->planrefarr));
-}
-
-/*
- * Remember that a plancache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargePlanCacheRefs()
- */
-void
-ResourceOwnerRememberPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- ResourceArrayAdd(&(owner->planrefarr), PointerGetDatum(plan));
-}
-
-/*
- * Forget that a plancache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- if (!ResourceArrayRemove(&(owner->planrefarr), PointerGetDatum(plan)))
- elog(ERROR, "plancache reference %p is not owned by resource owner %s",
- plan, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintPlanCacheLeakWarning(CachedPlan *plan)
-{
- elog(WARNING, "plancache reference leak: plan %p not closed", plan);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * tupdesc reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeTupleDescs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->tupdescarr));
-}
-
-/*
- * Remember that a tupdesc reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeTupleDescs()
- */
-void
-ResourceOwnerRememberTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- ResourceArrayAdd(&(owner->tupdescarr), PointerGetDatum(tupdesc));
-}
-
-/*
- * Forget that a tupdesc reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- if (!ResourceArrayRemove(&(owner->tupdescarr), PointerGetDatum(tupdesc)))
- elog(ERROR, "tupdesc reference %p is not owned by resource owner %s",
- tupdesc, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintTupleDescLeakWarning(TupleDesc tupdesc)
-{
- elog(WARNING,
- "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
- tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * snapshot reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeSnapshots(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->snapshotarr));
-}
-
-/*
- * Remember that a snapshot reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeSnapshots()
- */
-void
-ResourceOwnerRememberSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- ResourceArrayAdd(&(owner->snapshotarr), PointerGetDatum(snapshot));
-}
-
-/*
- * Forget that a snapshot reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- if (!ResourceArrayRemove(&(owner->snapshotarr), PointerGetDatum(snapshot)))
- elog(ERROR, "snapshot reference %p is not owned by resource owner %s",
- snapshot, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintSnapshotLeakWarning(Snapshot snapshot)
-{
- elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
- snapshot);
-}
-
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * files reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeFiles(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->filearr));
-}
-
-/*
- * Remember that a temporary file is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeFiles()
- */
-void
-ResourceOwnerRememberFile(ResourceOwner owner, File file)
-{
- ResourceArrayAdd(&(owner->filearr), FileGetDatum(file));
-}
-
-/*
- * Forget that a temporary file is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetFile(ResourceOwner owner, File file)
-{
- if (!ResourceArrayRemove(&(owner->filearr), FileGetDatum(file)))
- elog(ERROR, "temporary file %d is not owned by resource owner %s",
- file, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintFileLeakWarning(File file)
-{
- elog(WARNING, "temporary file leak: File %d still referenced",
- file);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * dynamic shmem segment reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeDSMs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->dsmarr));
-}
-
-/*
- * Remember that a dynamic shmem segment is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeDSMs()
- */
-void
-ResourceOwnerRememberDSM(ResourceOwner owner, dsm_segment *seg)
-{
- ResourceArrayAdd(&(owner->dsmarr), PointerGetDatum(seg));
-}
-
-/*
- * Forget that a dynamic shmem segment is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetDSM(ResourceOwner owner, dsm_segment *seg)
-{
- if (!ResourceArrayRemove(&(owner->dsmarr), PointerGetDatum(seg)))
- elog(ERROR, "dynamic shared memory segment %u is not owned by resource owner %s",
- dsm_segment_handle(seg), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintDSMLeakWarning(dsm_segment *seg)
-{
- elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
- dsm_segment_handle(seg));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * JIT context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeJIT(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->jitarr));
-}
-
-/*
- * Remember that a JIT context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeJIT()
- */
-void
-ResourceOwnerRememberJIT(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->jitarr), handle);
-}
-
-/*
- * Forget that a JIT context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetJIT(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->jitarr), handle))
- elog(ERROR, "JIT context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * cryptohash context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCryptoHash(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->cryptohasharr));
-}
-
-/*
- * Remember that a cryptohash context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCryptoHash()
- */
-void
-ResourceOwnerRememberCryptoHash(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->cryptohasharr), handle);
-}
-
-/*
- * Forget that a cryptohash context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCryptoHash(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->cryptohasharr), handle))
- elog(ERROR, "cryptohash context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintCryptoHashLeakWarning(Datum handle)
-{
- elog(WARNING, "cryptohash context reference leak: context %p still referenced",
- DatumGetPointer(handle));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * hmac context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeHMAC(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->hmacarr));
-}
-
-/*
- * Remember that a HMAC context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeHMAC()
- */
-void
-ResourceOwnerRememberHMAC(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->hmacarr), handle);
-}
-
-/*
- * Forget that a HMAC context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetHMAC(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->hmacarr), handle))
- elog(ERROR, "HMAC context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintHMACLeakWarning(Datum handle)
-{
- elog(WARNING, "HMAC context reference leak: context %p still referenced",
- DatumGetPointer(handle));
-}
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index 5001efdf7a..1a8a3dd105 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -66,7 +66,7 @@
#include "utils/memutils.h"
#include "utils/old_snapshot.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
#include "utils/timestamp.h"
@@ -174,6 +174,24 @@ static Snapshot CopySnapshot(Snapshot snapshot);
static void FreeSnapshot(Snapshot snapshot);
static void SnapshotResetXmin(void);
+/* ResourceOwner callbacks to track snapshot references */
+static void ResOwnerReleaseSnapshot(Datum res);
+static void ResOwnerPrintSnapshotLeakWarning(Datum res);
+
+static ResourceOwnerFuncs snapshot_resowner_funcs =
+{
+ .name = "snapshot reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseSnapshot,
+ .PrintLeakWarning = ResOwnerPrintSnapshotLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberSnapshot(owner, snap) \
+ ResourceOwnerRemember(owner, PointerGetDatum(snap), &snapshot_resowner_funcs)
+#define ResourceOwnerForgetSnapshot(owner, snap) \
+ ResourceOwnerForget(owner, PointerGetDatum(snap), &snapshot_resowner_funcs)
+
/*
* Snapshot fields to be serialized.
*
@@ -846,7 +864,7 @@ RegisterSnapshotOnOwner(Snapshot snapshot, ResourceOwner owner)
snap = snapshot->copied ? snapshot : CopySnapshot(snapshot);
/* and tell resowner.c about it */
- ResourceOwnerEnlargeSnapshots(owner);
+ ResourceOwnerEnlarge(owner);
snap->regd_count++;
ResourceOwnerRememberSnapshot(owner, snap);
@@ -2364,3 +2382,19 @@ XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot)
return false;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseSnapshot(Datum res)
+{
+ UnregisterSnapshot((Snapshot) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintSnapshotLeakWarning(Datum res)
+{
+ elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
+ DatumGetPointer(res));
+}
diff --git a/src/common/cryptohash_openssl.c b/src/common/cryptohash_openssl.c
index 643cc7aea2..cf5fc9eacb 100644
--- a/src/common/cryptohash_openssl.c
+++ b/src/common/cryptohash_openssl.c
@@ -30,7 +30,6 @@
#ifndef FRONTEND
#include "utils/memutils.h"
#include "utils/resowner.h"
-#include "utils/resowner_private.h"
#endif
/*
@@ -63,6 +62,27 @@ struct pg_cryptohash_ctx
#endif
};
+/* ResourceOwner callbacks to hold cryptohash contexts */
+#ifndef FRONTEND
+static void ResOwnerReleaseCryptoHash(Datum res);
+static void ResOwnerPrintCryptoHashLeakWarning(Datum res);
+
+static ResourceOwnerFuncs cryptohash_resowner_funcs =
+{
+ /* relcache references */
+ .name = "OpenSSL cryptohash context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCryptoHash,
+ .PrintLeakWarning = ResOwnerPrintCryptoHashLeakWarning,
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberCryptoHash(owner, ctx) \
+ ResourceOwnerRemember(owner, PointerGetDatum(ctx), &cryptohash_resowner_funcs)
+#define ResourceOwnerForgetCryptoHash(owner, ctx) \
+ ResourceOwnerForget(owner, PointerGetDatum(ctx), &cryptohash_resowner_funcs)
+#endif
+
/*
* pg_cryptohash_create
*
@@ -80,7 +100,7 @@ pg_cryptohash_create(pg_cryptohash_type type)
* allocation to avoid leaking.
*/
#ifndef FRONTEND
- ResourceOwnerEnlargeCryptoHash(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
#endif
ctx = ALLOC(sizeof(pg_cryptohash_ctx));
@@ -109,8 +129,7 @@ pg_cryptohash_create(pg_cryptohash_type type)
#ifndef FRONTEND
ctx->resowner = CurrentResourceOwner;
- ResourceOwnerRememberCryptoHash(CurrentResourceOwner,
- PointerGetDatum(ctx));
+ ResourceOwnerRememberCryptoHash(CurrentResourceOwner, ctx);
#endif
return ctx;
@@ -241,10 +260,28 @@ pg_cryptohash_free(pg_cryptohash_ctx *ctx)
EVP_MD_CTX_destroy(ctx->evpctx);
#ifndef FRONTEND
- ResourceOwnerForgetCryptoHash(ctx->resowner,
- PointerGetDatum(ctx));
+ ResourceOwnerForgetCryptoHash(ctx->resowner, ctx);
#endif
explicit_bzero(ctx, sizeof(pg_cryptohash_ctx));
FREE(ctx);
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+#ifndef FRONTEND
+static void
+ResOwnerReleaseCryptoHash(Datum res)
+{
+ pg_cryptohash_free((pg_cryptohash_ctx *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCryptoHashLeakWarning(Datum res)
+{
+ elog(WARNING, "cryptohash context reference leak: context %p still referenced",
+ DatumGetPointer(res));
+}
+#endif
diff --git a/src/common/hmac_openssl.c b/src/common/hmac_openssl.c
index 1acf59476e..0a65284bae 100644
--- a/src/common/hmac_openssl.c
+++ b/src/common/hmac_openssl.c
@@ -29,7 +29,6 @@
#ifndef FRONTEND
#include "utils/memutils.h"
#include "utils/resowner.h"
-#include "utils/resowner_private.h"
#endif
/*
@@ -63,6 +62,27 @@ struct pg_hmac_ctx
#endif
};
+/* ResourceOwner callbacks to hold HMAC contexts */
+#ifndef FRONTEND
+static void ResOwnerReleaseHMAC(Datum res);
+static void ResOwnerPrintHMACLeakWarning(Datum res);
+
+static ResourceOwnerFuncs hmac_resowner_funcs =
+{
+ /* relcache references */
+ .name = "OpenSSL HMAC context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseHMAC,
+ .PrintLeakWarning = ResOwnerPrintHMACLeakWarning,
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberHMAC(owner, ctx) \
+ ResourceOwnerRemember(owner, PointerGetDatum(ctx), &hmac_resowner_funcs)
+#define ResourceOwnerForgetHMAC(owner, ctx) \
+ ResourceOwnerForget(owner, PointerGetDatum(ctx), &hmac_resowner_funcs)
+#endif
+
/*
* pg_hmac_create
*
@@ -86,7 +106,7 @@ pg_hmac_create(pg_cryptohash_type type)
*/
#ifdef HAVE_HMAC_CTX_NEW
#ifndef FRONTEND
- ResourceOwnerEnlargeHMAC(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
#endif
ctx->hmacctx = HMAC_CTX_new();
#else
@@ -108,7 +128,7 @@ pg_hmac_create(pg_cryptohash_type type)
#ifdef HAVE_HMAC_CTX_NEW
#ifndef FRONTEND
ctx->resowner = CurrentResourceOwner;
- ResourceOwnerRememberHMAC(CurrentResourceOwner, PointerGetDatum(ctx));
+ ResourceOwnerRememberHMAC(CurrentResourceOwner, ctx);
#endif
#else
memset(ctx->hmacctx, 0, sizeof(HMAC_CTX));
@@ -244,7 +264,7 @@ pg_hmac_free(pg_hmac_ctx *ctx)
#ifdef HAVE_HMAC_CTX_FREE
HMAC_CTX_free(ctx->hmacctx);
#ifndef FRONTEND
- ResourceOwnerForgetHMAC(ctx->resowner, PointerGetDatum(ctx));
+ ResourceOwnerForgetHMAC(ctx->resowner, ctx);
#endif
#else
explicit_bzero(ctx->hmacctx, sizeof(HMAC_CTX));
@@ -254,3 +274,22 @@ pg_hmac_free(pg_hmac_ctx *ctx)
explicit_bzero(ctx, sizeof(pg_hmac_ctx));
FREE(ctx);
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+#ifndef FRONTEND
+static void
+ResOwnerReleaseHMAC(Datum res)
+{
+ pg_hmac_free((pg_hmac_ctx *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintHMACLeakWarning(Datum res)
+{
+ elog(WARNING, "HMAC context reference leak: context %p still referenced",
+ DatumGetPointer(res));
+}
+#endif
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 33fcaf5c9a..10ef8d850e 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -300,6 +300,15 @@ typedef struct CkptSortItem
extern CkptSortItem *CkptBufferIds;
+/* ResourceOwner callbacks to hold buffer pins */
+extern ResourceOwnerFuncs buffer_resowner_funcs;
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberBuffer(owner, buffer) \
+ ResourceOwnerRemember(owner, Int32GetDatum(buffer), &buffer_resowner_funcs)
+#define ResourceOwnerForgetBuffer(owner, buffer) \
+ ResourceOwnerForget(owner, Int32GetDatum(buffer), &buffer_resowner_funcs)
+
/*
* Internal buffer management routines
*/
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index ddc2762eb3..2bf95c22e8 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -225,7 +225,4 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
HeapTuple newtuple,
void (*function) (int, uint32, Oid));
-extern void PrintCatCacheLeakWarning(HeapTuple tuple);
-extern void PrintCatCacheListLeakWarning(CatCList *list);
-
#endif /* CATCACHE_H */
diff --git a/src/include/utils/plancache.h b/src/include/utils/plancache.h
index ff09c63a02..f3e3e50fb2 100644
--- a/src/include/utils/plancache.h
+++ b/src/include/utils/plancache.h
@@ -233,4 +233,6 @@ extern bool CachedPlanIsSimplyValid(CachedPlanSource *plansource,
extern CachedExpression *GetCachedExpression(Node *expr);
extern void FreeCachedExpression(CachedExpression *cexpr);
+extern ResourceOwnerFuncs planref_resowner_funcs;
+
#endif /* PLANCACHE_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index 109ac31b24..d59d14c0d0 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -50,6 +50,36 @@ typedef enum
RESOURCE_RELEASE_AFTER_LOCKS
} ResourceReleasePhase;
+/*
+ * In order to track an object, resowner.c needs a few callbacks for it.
+ * The callbacks for an resource of a specific kind are encapsulated in
+ * ResourceOwnerFuncs.
+ *
+ * Note that the callback occurs post-commit or post-abort, so these callback
+ * functions can only do noncritical cleanup.
+ */
+typedef struct ResourceOwnerFuncs
+{
+ const char *name; /* name for the object kind, for debugging */
+
+ ResourceReleasePhase phase; /* when are these objects released? */
+
+ /*
+ * Release resource.
+ *
+ * NOTE: this must call ResourceOwnerForget to disassociate it with the
+ * resource owner.
+ */
+ void (*ReleaseResource)(Datum res);
+
+ /*
+ * Print a warning, when a resource has not been properly released before
+ * commit.
+ */
+ void (*PrintLeakWarning)(Datum res);
+
+} ResourceOwnerFuncs;
+
/*
* Dynamically loaded modules can get control during ResourceOwnerRelease
* by providing a callback of this form.
@@ -71,16 +101,29 @@ extern void ResourceOwnerRelease(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
-extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
extern void ResourceOwnerDelete(ResourceOwner owner);
extern ResourceOwner ResourceOwnerGetParent(ResourceOwner owner);
extern void ResourceOwnerNewParent(ResourceOwner owner,
ResourceOwner newparent);
+
+extern void ResourceOwnerEnlarge(ResourceOwner owner);
+extern void ResourceOwnerRemember(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+extern void ResourceOwnerForget(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+
extern void RegisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
extern void UnregisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
+
extern void CreateAuxProcessResourceOwner(void);
extern void ReleaseAuxProcessResources(bool isCommit);
+/* special support for local lock management */
+struct LOCALLOCK;
+extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+
+/* special function to relase all plancache references */
+extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
+
#endif /* RESOWNER_H */
diff --git a/src/include/utils/resowner_private.h b/src/include/utils/resowner_private.h
deleted file mode 100644
index 6dafc87e28..0000000000
--- a/src/include/utils/resowner_private.h
+++ /dev/null
@@ -1,112 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * resowner_private.h
- * POSTGRES resource owner private definitions.
- *
- * See utils/resowner/README for more info.
- *
- *
- * Portions Copyright (c) 1996-2021, PostgreSQL Global Development Group
- * Portions Copyright (c) 1994, Regents of the University of California
- *
- * src/include/utils/resowner_private.h
- *
- *-------------------------------------------------------------------------
- */
-#ifndef RESOWNER_PRIVATE_H
-#define RESOWNER_PRIVATE_H
-
-#include "storage/dsm.h"
-#include "storage/fd.h"
-#include "storage/lock.h"
-#include "utils/catcache.h"
-#include "utils/plancache.h"
-#include "utils/resowner.h"
-#include "utils/snapshot.h"
-
-
-/* support for buffer refcount management */
-extern void ResourceOwnerEnlargeBuffers(ResourceOwner owner);
-extern void ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer);
-extern void ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer);
-
-/* support for local lock management */
-extern void ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock);
-extern void ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock);
-
-/* support for catcache refcount management */
-extern void ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerForgetCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-extern void ResourceOwnerForgetCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-
-/* support for relcache refcount management */
-extern void ResourceOwnerEnlargeRelationRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberRelationRef(ResourceOwner owner,
- Relation rel);
-extern void ResourceOwnerForgetRelationRef(ResourceOwner owner,
- Relation rel);
-
-/* support for plancache refcount management */
-extern void ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-extern void ResourceOwnerForgetPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-
-/* support for tupledesc refcount management */
-extern void ResourceOwnerEnlargeTupleDescs(ResourceOwner owner);
-extern void ResourceOwnerRememberTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-extern void ResourceOwnerForgetTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-
-/* support for snapshot refcount management */
-extern void ResourceOwnerEnlargeSnapshots(ResourceOwner owner);
-extern void ResourceOwnerRememberSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-extern void ResourceOwnerForgetSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-
-/* support for temporary file management */
-extern void ResourceOwnerEnlargeFiles(ResourceOwner owner);
-extern void ResourceOwnerRememberFile(ResourceOwner owner,
- File file);
-extern void ResourceOwnerForgetFile(ResourceOwner owner,
- File file);
-
-/* support for dynamic shared memory management */
-extern void ResourceOwnerEnlargeDSMs(ResourceOwner owner);
-extern void ResourceOwnerRememberDSM(ResourceOwner owner,
- dsm_segment *);
-extern void ResourceOwnerForgetDSM(ResourceOwner owner,
- dsm_segment *);
-
-/* support for JITContext management */
-extern void ResourceOwnerEnlargeJIT(ResourceOwner owner);
-extern void ResourceOwnerRememberJIT(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetJIT(ResourceOwner owner,
- Datum handle);
-
-/* support for cryptohash context management */
-extern void ResourceOwnerEnlargeCryptoHash(ResourceOwner owner);
-extern void ResourceOwnerRememberCryptoHash(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetCryptoHash(ResourceOwner owner,
- Datum handle);
-
-/* support for HMAC context management */
-extern void ResourceOwnerEnlargeHMAC(ResourceOwner owner);
-extern void ResourceOwnerRememberHMAC(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetHMAC(ResourceOwner owner,
- Datum handle);
-
-#endif /* RESOWNER_PRIVATE_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index da6ac8ed83..115f9983f6 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2234,8 +2234,10 @@ ReplicationStateOnDisk
ResTarget
ReservoirState
ReservoirStateData
-ResourceArray
+ResourceElem
ResourceOwner
+ResourceOwnerData
+ResourceOwnerFuncs
ResourceReleaseCallback
ResourceReleaseCallbackItem
ResourceReleasePhase
[application/octet-stream] v11-0003-Optimize-hash-function.patch (2.3K, ../../CAJ7c6TOj8RsUkfnVZnAPZ6jgWxqTLuv-o5mHYTc5goEhfBXaTA@mail.gmail.com/5-v11-0003-Optimize-hash-function.patch)
download | inline diff:
From 89d41607292cd2430bffb258ab42cd7045335ded Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Mon, 18 Oct 2021 13:59:11 +0300
Subject: [PATCH v10 3/3] Optimize hash function
---
src/backend/utils/resowner/resowner.c | 24 ++++++++++++++++++------
src/include/common/hashfn.h | 15 +++++++++++++++
2 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 35e824d4534..118bd2b34c9 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -169,15 +169,27 @@ static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
* INTERNAL ROUTINES *
*****************************************************************************/
+/*
+ * Hash function for value+kind combination.
+ */
static inline uint32
hash_resource_elem(Datum value, ResourceOwnerFuncs *kind)
{
- Datum data[2];
-
- data[0] = value;
- data[1] = PointerGetDatum(kind);
-
- return hash_bytes((unsigned char *) &data, 2 * SIZEOF_DATUM);
+ /*
+ * Most resources types store a pointer in 'value', and pointers are
+ * unique all on their own. But some resources store plain integers
+ * (Files and Buffers as of this writing), so we want to incorporate the
+ * 'kind' in the hash too, otherwise those resources will collide a lot.
+ * But because there are only a few resource kinds like that - and only a
+ * few resource kinds to begin with - we don't need to work too hard to
+ * mix 'kind' into the hash. Just add it with hash_combine(), it perturbs
+ * the result enough for our purposes.
+ */
+#if SIZEOF_DATUM == 8
+ return hash_combine64(murmurhash64((uint64) value), (uint64) kind);
+#else
+ return hash_combine(murmurhash32((uint32) value), (uint32) kind);
+#endif
}
/*
diff --git a/src/include/common/hashfn.h b/src/include/common/hashfn.h
index c634cc067a1..009ffbbdd38 100644
--- a/src/include/common/hashfn.h
+++ b/src/include/common/hashfn.h
@@ -101,4 +101,19 @@ murmurhash32(uint32 data)
return h;
}
+/* 64-bit variant */
+static inline uint64
+murmurhash64(uint64 data)
+{
+ uint64 h = data;
+
+ h ^= h >> 33;
+ h *= 0xff51afd7ed558ccd;
+ h ^= h >> 33;
+ h *= 0xc4ceb9fe1a85ec53;
+ h ^= h >> 33;
+
+ return h;
+}
+
#endif /* HASHFN_H */
--
2.30.2
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2021-11-26 12:40 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Aleksander Alekseev @ 2021-11-26 12:40 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: pgsql-hackers
Hi Heikki,
> I will submit the actual code review in the follow-up email
The patchset is in a good shape. I'm changing the status to "Ready for
Committer".
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2022-10-31 09:51 Heikki Linnakangas <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Heikki Linnakangas @ 2022-10-31 09:51 UTC (permalink / raw)
To: Julien Rouhaud <[email protected]>; Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers
On 12/01/2022 07:57, Julien Rouhaud wrote:
> On Fri, Nov 26, 2021 at 8:41 PM Aleksander Alekseev
> <[email protected]> wrote:
>>
>> The patchset is in a good shape. I'm changing the status to "Ready for
>> Committer".
>
> The 2nd patch doesn't apply anymore due to a conflict on
> resowner_private.h: http://cfbot.cputube.org/patch_36_3364.log.
Rebased version attached. Given that Aleksander marked this as Ready for
Committer earlier, I'll add this to the next commitfest in that state,
and will commit in the next few days, barring any new objections.
- Heikki
Attachments:
[text/x-patch] v12-0001-Move-a-few-ResourceOwnerEnlarge-calls-for-safety.patch (8.9K, ../../[email protected]/2-v12-0001-Move-a-few-ResourceOwnerEnlarge-calls-for-safety.patch)
download | inline diff:
From 14d05ab81a3615a0772bf34297fafce406e160ce Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Mon, 31 Oct 2022 10:33:36 +0100
Subject: [PATCH v12 1/3] Move a few ResourceOwnerEnlarge() calls for safety
and clarity.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
These are functions where quite a lot of things happen between the
ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
there are no unrelated ResourceOwnerRemember() calls in the code in
between, otherwise the entry reserved by the ResourceOwnerEnlarge() call
might be used up by the intervening ResourceOwnerRemember() and not be
available at the intended ResourceOwnerRemember() call anymore. The longer
the code path between them is, the harder it is to verify that.
In bufmgr.c, there is a function similar to ResourceOwnerEnlarge(), to
ensure that the private refcount array has enough space. The
ReservePrivateRefCountEntry() calls, analogous to ResourceOwnerEnlarge(),
were made at different places than the ResourceOwnerEnlarge() calls. Move
the ResourceOwnerEnlarge() calls together with the
ReservePrivateRefCountEntry() calls for consistency.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud, Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
---
src/backend/storage/buffer/bufmgr.c | 39 +++++++++++----------------
src/backend/storage/buffer/localbuf.c | 3 +++
src/backend/utils/cache/catcache.c | 13 ++++++---
3 files changed, 28 insertions(+), 27 deletions(-)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 73d30bf6191..d12e81097c5 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -828,9 +828,6 @@ ReadBuffer_common(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
*hit = false;
- /* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
isExtend = (blockNum == P_NEW);
TRACE_POSTGRESQL_BUFFER_READ_START(forkNum, blockNum,
@@ -1192,9 +1189,11 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
{
/*
* Ensure, while the spinlock's not yet held, that there's a free
- * refcount entry.
+ * refcount entry and that the resource owner has room to remember the
+ * pin.
*/
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1695,8 +1694,6 @@ ReleaseAndReadBuffer(Buffer buffer,
* taking the buffer header lock; instead update the state variable in loop of
* CAS operations. Hopefully it's just a single CAS.
*
- * Note that ResourceOwnerEnlargeBuffers must have been done already.
- *
* Returns true if buffer is BM_VALID, else false. This provision allows
* some callers to avoid an extra spinlock cycle.
*/
@@ -1707,6 +1704,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
ref = GetPrivateRefCountEntry(b, true);
if (ref == NULL)
@@ -1787,7 +1786,8 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
* The spinlock is released before return.
*
* As this function is called with the spinlock held, the caller has to
- * previously call ReservePrivateRefCountEntry().
+ * previously call ReservePrivateRefCountEntry() and
+ * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -1960,9 +1960,6 @@ BufferSync(int flags)
int mask = BM_DIRTY;
WritebackContext wb_context;
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
/*
* Unless this is a shutdown checkpoint or we have been explicitly told,
* we write only permanent, dirty buffers. But at shutdown or end of
@@ -2436,9 +2433,6 @@ BgBufferSync(WritebackContext *wb_context)
* requirements, or hit the bgwriter_lru_maxpages limit.
*/
- /* Make sure we can handle the pin inside SyncOneBuffer */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
num_to_scan = bufs_to_lap;
num_written = 0;
reusable_buffers = reusable_buffers_est;
@@ -2520,8 +2514,6 @@ BgBufferSync(WritebackContext *wb_context)
*
* (BUF_WRITTEN could be set in error if FlushBuffer finds the buffer clean
* after locking it, but we don't care all that much.)
- *
- * Note: caller must have done ResourceOwnerEnlargeBuffers.
*/
static int
SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
@@ -2531,7 +2523,9 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
uint32 buf_state;
BufferTag tag;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3562,9 +3556,6 @@ FlushRelationBuffers(Relation rel)
return;
}
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3578,7 +3569,9 @@ FlushRelationBuffers(Relation rel)
if (!BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator))
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) &&
@@ -3635,9 +3628,6 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (use_bsearch)
pg_qsort(srels, nrels, sizeof(SMgrSortArray), rlocator_comparator);
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
SMgrSortArray *srelent = NULL;
@@ -3676,7 +3666,9 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
if (srelent == NULL)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (BufTagMatchesRelFileLocator(&bufHdr->tag, &srelent->rlocator) &&
@@ -3870,9 +3862,6 @@ FlushDatabaseBuffers(Oid dbid)
int i;
BufferDesc *bufHdr;
- /* Make sure we can handle the pin inside the loop */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
-
for (i = 0; i < NBuffers; i++)
{
uint32 buf_state;
@@ -3886,7 +3875,9 @@ FlushDatabaseBuffers(Oid dbid)
if (bufHdr->tag.dbOid != dbid)
continue;
+ /* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.dbOid == dbid &&
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 30d67d1c40d..58e7c32c1e3 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -123,6 +123,9 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
if (LocalBufHash == NULL)
InitLocalBuffers();
+ /* Make sure we will have room to remember the buffer pin */
+ ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
hash_search(LocalBufHash, (void *) &newTag, HASH_FIND, NULL);
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index 38e943fab2b..e7b5b20ea29 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -1604,8 +1604,6 @@ SearchCatCacheList(CatCache *cache,
* block to ensure we can undo those refcounts if we get an error before
* we finish constructing the CatCList.
*/
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
-
ctlist = NIL;
PG_TRY();
@@ -1693,13 +1691,22 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
+ /* Make sure the resource owner has room to remember this entry. */
+ ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
nmembers = list_length(ctlist);
cl = (CatCList *)
palloc(offsetof(CatCList, members) + nmembers * sizeof(CatCTup *));
- /* Extract key values */
+ /*
+ * Extract key values.
+ *
+ * XXX: If we run out of memory while copying the key values, we will
+ * leak any allocations we had already made in the CacheMemoryContext.
+ * That is unlikely enough that we just accept the risk.
+ */
CatCacheCopyKeys(cache->cc_tupdesc, nkeys, cache->cc_keyno,
arguments, cl->keys);
MemoryContextSwitchTo(oldcxt);
--
2.30.2
[text/x-patch] v12-0002-Make-resowners-more-easily-extensible.patch (94.0K, ../../[email protected]/3-v12-0002-Make-resowners-more-easily-extensible.patch)
download | inline diff:
From e0701d958b1fb55ae0d0cea79221ca1653ee0182 Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Mon, 31 Oct 2022 10:48:57 +0100
Subject: [PATCH v12 2/3] Make resowners more easily extensible.
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
Instead of having a separate array/hash for each resource kind, use a
single array and hash to hold all kinds of resources. This makes it
possible to introduce new resource "kinds" without having to modify the
ResourceOwnerData struct. In particular, this makes it possible for
extensions to register custom resource kinds.
The old approach was to have a small array of resources of each kind, and
if it fills up, switch to a hash table. The new approach also uses an
array and a hash, but now the array and a hash are used at the same time.
The array is used to hold the recently added resources, and when it fills
up, they are moved to the hash. This keeps the access to recent entries
fast, even when there are a lot of long-held resources.
All the resource-specific ResourceOwnerEnlarge*(), ResourceOwnerRemember*(),
and ResourceOwnerForget*() functions have been replaced with three generic
functions that take resource kind as argument. For convenience, we still
define resource-specific wrapper macros around the generic functions, with
the same old names, but they are now defined in the source files that use
those resource kinds.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Julien Rouhaud, Kyotaro Horiguchi, Hayato Kuroda, Álvaro Herrera, Zhihong Yu
Discussion: https://www.postgresql.org/message-id/cbfabeb0-cd3c-e951-a572-19b365ed314d%40iki.fi
---
src/backend/access/common/tupdesc.c | 42 +-
src/backend/jit/jit.c | 2 -
src/backend/jit/llvm/llvmjit.c | 46 +-
src/backend/storage/buffer/bufmgr.c | 47 +-
src/backend/storage/buffer/localbuf.c | 4 +-
src/backend/storage/file/fd.c | 44 +-
src/backend/storage/ipc/dsm.c | 44 +-
src/backend/storage/lmgr/lock.c | 2 +-
src/backend/utils/cache/catcache.c | 74 +-
src/backend/utils/cache/plancache.c | 45 +-
src/backend/utils/cache/relcache.c | 41 +-
src/backend/utils/resowner/README | 18 +-
src/backend/utils/resowner/resowner.c | 1327 +++++++------------------
src/backend/utils/time/snapmgr.c | 39 +-
src/common/cryptohash_openssl.c | 48 +-
src/common/hmac_openssl.c | 46 +-
src/include/storage/buf_internals.h | 9 +
src/include/utils/catcache.h | 3 -
src/include/utils/plancache.h | 2 +
src/include/utils/resowner.h | 45 +-
src/include/utils/resowner_private.h | 112 ---
src/tools/pgindent/typedefs.list | 4 +-
22 files changed, 913 insertions(+), 1131 deletions(-)
delete mode 100644 src/include/utils/resowner_private.h
diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c
index b7f918c877b..9636abb4ff7 100644
--- a/src/backend/access/common/tupdesc.c
+++ b/src/backend/access/common/tupdesc.c
@@ -30,9 +30,27 @@
#include "utils/acl.h"
#include "utils/builtins.h"
#include "utils/datum.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
+/* ResourceOwner callbacks to hold tupledesc references */
+static void ResOwnerReleaseTupleDesc(Datum res);
+static void ResOwnerPrintTupleDescLeakWarning(Datum res);
+
+static ResourceOwnerFuncs tupdesc_resowner_funcs =
+{
+ /* relcache references */
+ .name = "tupdesc reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseTupleDesc,
+ .PrintLeakWarning = ResOwnerPrintTupleDescLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberTupleDesc(owner, tupdesc) \
+ ResourceOwnerRemember(owner, PointerGetDatum(tupdesc), &tupdesc_resowner_funcs)
+#define ResourceOwnerForgetTupleDesc(owner, tupdesc) \
+ ResourceOwnerForget(owner, PointerGetDatum(tupdesc), &tupdesc_resowner_funcs)
/*
* CreateTemplateTupleDesc
@@ -367,7 +385,7 @@ IncrTupleDescRefCount(TupleDesc tupdesc)
{
Assert(tupdesc->tdrefcount >= 0);
- ResourceOwnerEnlargeTupleDescs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
tupdesc->tdrefcount++;
ResourceOwnerRememberTupleDesc(CurrentResourceOwner, tupdesc);
}
@@ -919,3 +937,23 @@ BuildDescFromLists(List *names, List *types, List *typmods, List *collations)
return desc;
}
+
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseTupleDesc(Datum res)
+{
+ DecrTupleDescRefCount((TupleDesc) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintTupleDescLeakWarning(Datum res)
+{
+ TupleDesc tupdesc = (TupleDesc) DatumGetPointer(res);
+
+ elog(WARNING,
+ "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
+ tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
+}
diff --git a/src/backend/jit/jit.c b/src/backend/jit/jit.c
index 91a6b2b63ad..9354ba34c3b 100644
--- a/src/backend/jit/jit.c
+++ b/src/backend/jit/jit.c
@@ -26,7 +26,6 @@
#include "jit/jit.h"
#include "miscadmin.h"
#include "utils/fmgrprotos.h"
-#include "utils/resowner_private.h"
/* GUCs */
bool jit_enabled = true;
@@ -140,7 +139,6 @@ jit_release_context(JitContext *context)
if (provider_successfully_loaded)
provider.release_context(context);
- ResourceOwnerForgetJIT(context->resowner, PointerGetDatum(context));
pfree(context);
}
diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index 9aca7fc7a47..f0e15bc6345 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -40,7 +40,7 @@
#include "portability/instr_time.h"
#include "storage/ipc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Handle of a module emitted via ORC JIT */
typedef struct LLVMJitHandle
@@ -121,8 +121,26 @@ static LLVMOrcLLJITRef llvm_create_jit_instance(LLVMTargetMachineRef tm);
static char *llvm_error_message(LLVMErrorRef error);
#endif /* LLVM_VERSION_MAJOR > 11 */
-PG_MODULE_MAGIC;
+/* ResourceOwner callbacks to hold JitContexts */
+static void ResOwnerReleaseJitContext(Datum res);
+static void ResOwnerPrintJitContextLeakWarning(Datum res);
+
+static ResourceOwnerFuncs jit_resowner_funcs =
+{
+ /* relcache references */
+ .name = "LLVM JIT context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseJitContext,
+ .PrintLeakWarning = ResOwnerPrintJitContextLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberJIT(owner, handle) \
+ ResourceOwnerRemember(owner, PointerGetDatum(handle), &jit_resowner_funcs)
+#define ResourceOwnerForgetJIT(owner, handle) \
+ ResourceOwnerForget(owner, PointerGetDatum(handle), &jit_resowner_funcs)
+PG_MODULE_MAGIC;
/*
* Initialize LLVM JIT provider.
@@ -151,7 +169,7 @@ llvm_create_context(int jitFlags)
llvm_session_initialize();
- ResourceOwnerEnlargeJIT(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
context = MemoryContextAllocZero(TopMemoryContext,
sizeof(LLVMJitContext));
@@ -159,7 +177,7 @@ llvm_create_context(int jitFlags)
/* ensure cleanup */
context->base.resowner = CurrentResourceOwner;
- ResourceOwnerRememberJIT(CurrentResourceOwner, PointerGetDatum(context));
+ ResourceOwnerRememberJIT(CurrentResourceOwner, context);
return context;
}
@@ -221,6 +239,8 @@ llvm_release_context(JitContext *context)
}
list_free(llvm_context->handles);
llvm_context->handles = NIL;
+
+ ResourceOwnerForgetJIT(context->resowner, context);
}
/*
@@ -1266,3 +1286,21 @@ llvm_error_message(LLVMErrorRef error)
}
#endif /* LLVM_VERSION_MAJOR > 11 */
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseJitContext(Datum res)
+{
+ jit_release_context((JitContext *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintJitContextLeakWarning(Datum res)
+{
+ /* XXX: We used to not print these. Was that intentional? */
+ JitContext *context = (JitContext *) DatumGetPointer(res);
+
+ elog(WARNING, "JIT context leak: context %p still referenced", context);
+}
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index d12e81097c5..0f0cedc8b39 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -54,7 +54,7 @@
#include "utils/memdebug.h"
#include "utils/ps_status.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/timestamp.h"
@@ -208,6 +208,18 @@ static PrivateRefCountEntry *GetPrivateRefCountEntry(Buffer buffer, bool do_move
static inline int32 GetPrivateRefCount(Buffer buffer);
static void ForgetPrivateRefCountEntry(PrivateRefCountEntry *ref);
+/* ResourceOwner callbacks to hold buffer pins */
+static void ResOwnerReleaseBuffer(Datum res);
+static void ResOwnerPrintBufferLeakWarning(Datum res);
+
+ResourceOwnerFuncs buffer_resowner_funcs =
+{
+ .name = "buffer",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseBuffer,
+ .PrintLeakWarning = ResOwnerPrintBufferLeakWarning
+};
+
/*
* Ensure that the PrivateRefCountArray has sufficient space to store one more
* entry. This has to be called before using NewPrivateRefCountEntry() to fill
@@ -630,7 +642,7 @@ ReadRecentBuffer(RelFileLocator rlocator, ForkNumber forkNum, BlockNumber blockN
Assert(BufferIsValid(recent_buffer));
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ReservePrivateRefCountEntry();
InitBufferTag(&tag, &rlocator, forkNum, blockNum);
@@ -1193,7 +1205,7 @@ BufferAlloc(SMgrRelation smgr, char relpersistence, ForkNumber forkNum,
* pin.
*/
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Select a victim buffer. The buffer is returned with its header
@@ -1704,7 +1716,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
bool result;
PrivateRefCountEntry *ref;
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ref = GetPrivateRefCountEntry(b, true);
@@ -1787,7 +1799,7 @@ PinBuffer(BufferDesc *buf, BufferAccessStrategy strategy)
*
* As this function is called with the spinlock held, the caller has to
* previously call ReservePrivateRefCountEntry() and
- * ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ * ResourceOwnerEnlarge(CurrentResourceOwner);
*
* Currently, no callers of this function want to modify the buffer's
* usage_count at all, so there's no need for a strategy parameter.
@@ -2525,7 +2537,7 @@ SyncOneBuffer(int buf_id, bool skip_recently_used, WritebackContext *wb_context)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Check whether buffer needs writing.
@@ -3571,7 +3583,7 @@ FlushRelationBuffers(Relation rel)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (BufTagMatchesRelFileLocator(&bufHdr->tag, &rel->rd_locator) &&
@@ -3668,7 +3680,7 @@ FlushRelationsAllBuffers(SMgrRelation *smgrs, int nrels)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (BufTagMatchesRelFileLocator(&bufHdr->tag, &srelent->rlocator) &&
@@ -3877,7 +3889,7 @@ FlushDatabaseBuffers(Oid dbid)
/* Make sure we can handle the pin */
ReservePrivateRefCountEntry();
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
buf_state = LockBufHdr(bufHdr);
if (bufHdr->tag.dbOid == dbid &&
@@ -3960,7 +3972,7 @@ void
IncrBufferRefCount(Buffer buffer)
{
Assert(BufferIsPinned(buffer));
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
if (BufferIsLocal(buffer))
LocalRefCount[-buffer - 1]++;
else
@@ -5021,3 +5033,18 @@ TestForOldSnapshot_impl(Snapshot snapshot, Relation relation)
(errcode(ERRCODE_SNAPSHOT_TOO_OLD),
errmsg("snapshot too old")));
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseBuffer(Datum res)
+{
+ ReleaseBuffer(DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintBufferLeakWarning(Datum res)
+{
+ PrintBufferLeakWarning(DatumGetInt32(res));
+}
diff --git a/src/backend/storage/buffer/localbuf.c b/src/backend/storage/buffer/localbuf.c
index 58e7c32c1e3..f83722173ed 100644
--- a/src/backend/storage/buffer/localbuf.c
+++ b/src/backend/storage/buffer/localbuf.c
@@ -22,7 +22,7 @@
#include "storage/bufmgr.h"
#include "utils/guc_hooks.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/*#define LBDEBUG*/
@@ -124,7 +124,7 @@ LocalBufferAlloc(SMgrRelation smgr, ForkNumber forkNum, BlockNumber blockNum,
InitLocalBuffers();
/* Make sure we will have room to remember the buffer pin */
- ResourceOwnerEnlargeBuffers(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/* See if the desired buffer already exists */
hresult = (LocalBufferLookupEnt *)
diff --git a/src/backend/storage/file/fd.c b/src/backend/storage/file/fd.c
index 4151cafec54..41a5386f512 100644
--- a/src/backend/storage/file/fd.c
+++ b/src/backend/storage/file/fd.c
@@ -98,7 +98,7 @@
#include "storage/fd.h"
#include "storage/ipc.h"
#include "utils/guc.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* Define PG_FLUSH_DATA_WORKS if we have an implementation for pg_flush_data */
#if defined(HAVE_SYNC_FILE_RANGE)
@@ -349,6 +349,24 @@ static void unlink_if_exists_fname(const char *fname, bool isdir, int elevel);
static int fsync_parent_path(const char *fname, int elevel);
+/* ResourceOwner callbacks to hold virtual file descriptors */
+static void ResOwnerReleaseFile(Datum res);
+static void ResOwnerPrintFileLeakWarning(Datum res);
+
+static ResourceOwnerFuncs file_resowner_funcs =
+{
+ .name = "File",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseFile,
+ .PrintLeakWarning = ResOwnerPrintFileLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberFile(owner, file) \
+ ResourceOwnerRemember(owner, Int32GetDatum(file), &file_resowner_funcs)
+#define ResourceOwnerForgetFile(owner, file) \
+ ResourceOwnerForget(owner, Int32GetDatum(file), &file_resowner_funcs)
+
/*
* pg_fsync --- do fsync with or without writethrough
*/
@@ -1453,7 +1471,7 @@ ReportTemporaryFileUsage(const char *path, off_t size)
/*
* Called to register a temporary file for automatic close.
- * ResourceOwnerEnlargeFiles(CurrentResourceOwner) must have been called
+ * ResourceOwnerEnlarge(CurrentResourceOwner) must have been called
* before the file was opened.
*/
static void
@@ -1637,7 +1655,7 @@ OpenTemporaryFile(bool interXact)
* open it, if we'll be registering it below.
*/
if (!interXact)
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* If some temp tablespace(s) have been given to us, try to use the next
@@ -1769,7 +1787,7 @@ PathNameCreateTemporaryFile(const char *path, bool error_on_failure)
Assert(temporary_files_allowed); /* check temp file access is up */
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/*
* Open the file. Note: we don't use O_EXCL, in case there is an orphaned
@@ -1809,7 +1827,7 @@ PathNameOpenTemporaryFile(const char *path, int mode)
Assert(temporary_files_allowed); /* check temp file access is up */
- ResourceOwnerEnlargeFiles(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
file = PathNameOpenFile(path, mode | PG_BINARY);
@@ -3737,3 +3755,19 @@ data_sync_elevel(int elevel)
{
return data_sync_retry ? elevel : PANIC;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseFile(Datum res)
+{
+ FileClose((File) DatumGetInt32(res));
+}
+
+static void
+ResOwnerPrintFileLeakWarning(Datum res)
+{
+ elog(WARNING, "temporary file leak: File %d still referenced",
+ DatumGetInt32(res));
+}
diff --git a/src/backend/storage/ipc/dsm.c b/src/backend/storage/ipc/dsm.c
index a360325f864..d1eea705ebc 100644
--- a/src/backend/storage/ipc/dsm.c
+++ b/src/backend/storage/ipc/dsm.c
@@ -38,13 +38,15 @@
#include "miscadmin.h"
#include "port/pg_bitutils.h"
#include "storage/dsm.h"
+#include "storage/fd.h"
#include "storage/ipc.h"
#include "storage/lwlock.h"
#include "storage/pg_shmem.h"
+#include "storage/shmem.h"
#include "utils/freepage.h"
#include "utils/guc.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#define PG_DYNSHMEM_CONTROL_MAGIC 0x9a503d32
@@ -140,6 +142,25 @@ static dsm_control_header *dsm_control;
static Size dsm_control_mapped_size = 0;
static void *dsm_control_impl_private = NULL;
+
+/* ResourceOwner callbacks to hold DSM segments */
+static void ResOwnerReleaseDSM(Datum res);
+static void ResOwnerPrintDSMLeakWarning(Datum res);
+
+static ResourceOwnerFuncs dsm_resowner_funcs =
+{
+ .name = "dynamic shared memory segment",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseDSM,
+ .PrintLeakWarning = ResOwnerPrintDSMLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberDSM(owner, seg) \
+ ResourceOwnerRemember(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+#define ResourceOwnerForgetDSM(owner, seg) \
+ ResourceOwnerForget(owner, PointerGetDatum(seg), &dsm_resowner_funcs)
+
/*
* Start up the dynamic shared memory system.
*
@@ -908,7 +929,7 @@ void
dsm_unpin_mapping(dsm_segment *seg)
{
Assert(seg->resowner == NULL);
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg->resowner = CurrentResourceOwner;
ResourceOwnerRememberDSM(seg->resowner, seg);
}
@@ -1175,7 +1196,7 @@ dsm_create_descriptor(void)
dsm_segment *seg;
if (CurrentResourceOwner)
- ResourceOwnerEnlargeDSMs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
seg = MemoryContextAlloc(TopMemoryContext, sizeof(dsm_segment));
dlist_push_head(&dsm_segment_list, &seg->node);
@@ -1254,3 +1275,20 @@ is_main_region_dsm_handle(dsm_handle handle)
{
return handle & 1;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseDSM(Datum res)
+{
+ dsm_detach((dsm_segment *) DatumGetPointer(res));
+}
+static void
+ResOwnerPrintDSMLeakWarning(Datum res)
+{
+ dsm_segment *seg = (dsm_segment *) res;
+
+ elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
+ dsm_segment_handle(seg));
+}
diff --git a/src/backend/storage/lmgr/lock.c b/src/backend/storage/lmgr/lock.c
index 3d1049cf756..bf363001e0a 100644
--- a/src/backend/storage/lmgr/lock.c
+++ b/src/backend/storage/lmgr/lock.c
@@ -48,7 +48,7 @@
#include "storage/standby.h"
#include "utils/memutils.h"
#include "utils/ps_status.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
/* This configuration variable is used to set the lock table size */
diff --git a/src/backend/utils/cache/catcache.c b/src/backend/utils/cache/catcache.c
index e7b5b20ea29..607ddb6f6f9 100644
--- a/src/backend/utils/cache/catcache.c
+++ b/src/backend/utils/cache/catcache.c
@@ -32,12 +32,13 @@
#endif
#include "storage/lmgr.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/memutils.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/syscache.h"
@@ -105,6 +106,42 @@ static void CatCacheCopyKeys(TupleDesc tupdesc, int nkeys, int *attnos,
* internal support functions
*/
+/* ResourceOwner callbacks to hold catcache references */
+
+static void ResOwnerReleaseCatCache(Datum res);
+static void ResOwnerPrintCatCacheLeakWarning(Datum res);
+static void ResOwnerReleaseCatCacheList(Datum res);
+static void ResOwnerPrintCatCacheListLeakWarning(Datum res);
+
+static ResourceOwnerFuncs catcache_resowner_funcs =
+{
+ /* catcache references */
+ .name = "catcache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCache,
+ .PrintLeakWarning = ResOwnerPrintCatCacheLeakWarning
+};
+
+static ResourceOwnerFuncs catlistref_resowner_funcs =
+{
+ /* catcache-list pins */
+ .name = "catcache list reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCatCacheList,
+ .PrintLeakWarning = ResOwnerPrintCatCacheListLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberCatCacheRef(owner, tuple) \
+ ResourceOwnerRemember(owner, PointerGetDatum(tuple), &catcache_resowner_funcs)
+#define ResourceOwnerForgetCatCacheRef(owner, tuple) \
+ ResourceOwnerForget(owner, PointerGetDatum(tuple), &catcache_resowner_funcs)
+#define ResourceOwnerRememberCatCacheListRef(owner, list) \
+ ResourceOwnerRemember(owner, PointerGetDatum(list), &catlistref_resowner_funcs)
+#define ResourceOwnerForgetCatCacheListRef(owner, list) \
+ ResourceOwnerForget(owner, PointerGetDatum(list), &catlistref_resowner_funcs)
+
+
/*
* Hash and equality functions for system types that are used as cache key
* fields. In some cases, we just call the regular SQL-callable functions for
@@ -1267,7 +1304,7 @@ SearchCatCacheInternal(CatCache *cache,
*/
if (!ct->negative)
{
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
@@ -1368,7 +1405,7 @@ SearchCatCacheMiss(CatCache *cache,
hashValue, hashIndex,
false);
/* immediately set the refcount to 1 */
- ResourceOwnerEnlargeCatCacheRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
ct->refcount++;
ResourceOwnerRememberCatCacheRef(CurrentResourceOwner, &ct->tuple);
break; /* assume only one match */
@@ -1580,7 +1617,7 @@ SearchCatCacheList(CatCache *cache,
dlist_move_head(&cache->cc_lists, &cl->cache_elem);
/* Bump the list's refcount and return it */
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
cl->refcount++;
ResourceOwnerRememberCatCacheListRef(CurrentResourceOwner, cl);
@@ -1692,7 +1729,7 @@ SearchCatCacheList(CatCache *cache,
table_close(relation, AccessShareLock);
/* Make sure the resource owner has room to remember this entry. */
- ResourceOwnerEnlargeCatCacheListRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
/* Now we can build the CatCList entry. */
oldcxt = MemoryContextSwitchTo(CacheMemoryContext);
@@ -2064,14 +2101,19 @@ PrepareToInvalidateCacheTuple(Relation relation,
}
}
-
/*
- * Subroutines for warning about reference leaks. These are exported so
- * that resowner.c can call them.
+ * ResourceOwner callbacks
*/
-void
-PrintCatCacheLeakWarning(HeapTuple tuple)
+static void
+ResOwnerReleaseCatCache(Datum res)
+{
+ ReleaseCatCache((HeapTuple) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheLeakWarning(Datum res)
{
+ HeapTuple tuple = (HeapTuple) DatumGetPointer(res);
CatCTup *ct = (CatCTup *) (((char *) tuple) -
offsetof(CatCTup, tuple));
@@ -2085,9 +2127,17 @@ PrintCatCacheLeakWarning(HeapTuple tuple)
ct->refcount);
}
-void
-PrintCatCacheListLeakWarning(CatCList *list)
+static void
+ResOwnerReleaseCatCacheList(Datum res)
{
+ ReleaseCatCacheList((CatCList *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCatCacheListLeakWarning(Datum res)
+{
+ CatCList *list = (CatCList *) DatumGetPointer(res);
+
elog(WARNING, "cache reference leak: cache %s (%d), list %p has count %d",
list->my_cache->cc_relname, list->my_cache->id,
list, list->refcount);
diff --git a/src/backend/utils/cache/plancache.c b/src/backend/utils/cache/plancache.c
index cc943205d34..319256c99a4 100644
--- a/src/backend/utils/cache/plancache.c
+++ b/src/backend/utils/cache/plancache.c
@@ -69,7 +69,7 @@
#include "tcop/utility.h"
#include "utils/inval.h"
#include "utils/memutils.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/rls.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -115,6 +115,26 @@ static void PlanCacheRelCallback(Datum arg, Oid relid);
static void PlanCacheObjectCallback(Datum arg, int cacheid, uint32 hashvalue);
static void PlanCacheSysCallback(Datum arg, int cacheid, uint32 hashvalue);
+/* ResourceOwner callbacks to track plancache references */
+static void ResOwnerReleaseCachedPlan(Datum res);
+static void ResOwnerPrintPlanCacheLeakWarning(Datum res);
+
+/* this is exported for ResourceOwnerReleaseAllPlanCacheRefs() */
+ResourceOwnerFuncs planref_resowner_funcs =
+{
+ .name = "plancache reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCachedPlan,
+ .PrintLeakWarning = ResOwnerPrintPlanCacheLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberPlanCacheRef(owner, plan) \
+ ResourceOwnerRemember(owner, PointerGetDatum(plan), &planref_resowner_funcs)
+#define ResourceOwnerForgetPlanCacheRef(owner, plan) \
+ ResourceOwnerForget(owner, PointerGetDatum(plan), &planref_resowner_funcs)
+
+
/* GUC parameter */
int plan_cache_mode = PLAN_CACHE_MODE_AUTO;
@@ -1229,7 +1249,7 @@ GetCachedPlan(CachedPlanSource *plansource, ParamListInfo boundParams,
/* Flag the plan as in use by caller */
if (owner)
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
if (owner)
ResourceOwnerRememberPlanCacheRef(owner, plan);
@@ -1392,7 +1412,7 @@ CachedPlanAllowsSimpleValidityCheck(CachedPlanSource *plansource,
/* Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -1451,7 +1471,7 @@ CachedPlanIsSimplyValid(CachedPlanSource *plansource, CachedPlan *plan,
/* It's still good. Bump refcount if requested. */
if (owner)
{
- ResourceOwnerEnlargePlanCacheRefs(owner);
+ ResourceOwnerEnlarge(owner);
plan->refcount++;
ResourceOwnerRememberPlanCacheRef(owner, plan);
}
@@ -2205,3 +2225,20 @@ ResetPlanCache(void)
cexpr->is_valid = false;
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+
+static void
+ResOwnerReleaseCachedPlan(Datum res)
+{
+ ReleaseCachedPlan((CachedPlan *) DatumGetPointer(res), CurrentResourceOwner);
+}
+
+static void
+ResOwnerPrintPlanCacheLeakWarning(Datum res)
+{
+ elog(WARNING, "plancache reference leak: plan %p not closed",
+ DatumGetPointer(res));
+}
diff --git a/src/backend/utils/cache/relcache.c b/src/backend/utils/cache/relcache.c
index bd6cd4e47b5..ab68df1277e 100644
--- a/src/backend/utils/cache/relcache.c
+++ b/src/backend/utils/cache/relcache.c
@@ -80,13 +80,14 @@
#include "storage/smgr.h"
#include "utils/array.h"
#include "utils/builtins.h"
+#include "utils/catcache.h"
#include "utils/datum.h"
#include "utils/fmgroids.h"
#include "utils/inval.h"
#include "utils/lsyscache.h"
#include "utils/memutils.h"
#include "utils/relmapper.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
@@ -2115,6 +2116,24 @@ RelationIdGetRelation(Oid relationId)
* ----------------------------------------------------------------
*/
+/* ResourceOwner callbacks to track relcache references */
+static void ResOwnerReleaseRelation(Datum res);
+static void ResOwnerPrintRelCacheLeakWarning(Datum res);
+
+static ResourceOwnerFuncs relref_resowner_funcs =
+{
+ .name = "relcache reference",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseRelation,
+ .PrintLeakWarning = ResOwnerPrintRelCacheLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberRelationRef(owner, rel) \
+ ResourceOwnerRemember(owner, PointerGetDatum(rel), &relref_resowner_funcs)
+#define ResourceOwnerForgetRelationRef(owner, rel) \
+ ResourceOwnerForget(owner, PointerGetDatum(rel), &relref_resowner_funcs)
+
/*
* RelationIncrementReferenceCount
* Increments relation reference count.
@@ -2126,7 +2145,7 @@ RelationIdGetRelation(Oid relationId)
void
RelationIncrementReferenceCount(Relation rel)
{
- ResourceOwnerEnlargeRelationRefs(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
rel->rd_refcnt += 1;
if (!IsBootstrapProcessingMode())
ResourceOwnerRememberRelationRef(CurrentResourceOwner, rel);
@@ -6776,3 +6795,21 @@ unlink_initfile(const char *initfilename, int elevel)
initfilename)));
}
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerPrintRelCacheLeakWarning(Datum res)
+{
+ Relation rel = (Relation) res;
+
+ elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
+ RelationGetRelationName(rel));
+}
+
+static void
+ResOwnerReleaseRelation(Datum res)
+{
+ RelationClose((Relation) res);
+}
diff --git a/src/backend/utils/resowner/README b/src/backend/utils/resowner/README
index f94c9700df4..e32f7a27384 100644
--- a/src/backend/utils/resowner/README
+++ b/src/backend/utils/resowner/README
@@ -54,12 +54,18 @@ The basic operations on a ResourceOwner are:
* delete a ResourceOwner (including child owner objects); all resources
must have been released beforehand
-This API directly supports the resource types listed in the definition of
-ResourceOwnerData struct in src/backend/utils/resowner/resowner.c.
-Other objects can be associated with a ResourceOwner by recording the address
-of the owning ResourceOwner in such an object. There is an API for other
-modules to get control during ResourceOwner release, so that they can scan
-their own data structures to find the objects that need to be deleted.
+ResourceOwner can record ownership of many different kinds of resources. In
+core PostgreSQL, it is used for buffer pins, lmgr locks, and catalog cache
+references, to name a few examples. ResourceOwner treats all resources the
+same, and extensions can define new kinds of resources by filling in a
+ResourceOwnerFuncs struct with suitable callback functions.
+
+There is also an API for other modules to get control during ResourceOwner
+release, so that they can scan their own data structures to find the objects
+that need to be deleted. See RegisterResourceReleaseCallback function.
+This used to be the only way for extensions to use the resource owner
+mechanism with new kinds of objects; nowadays it easier to write custom
+ResourceOwnerFuncs callbacks.
Locks are handled specially because in non-error situations a lock should
be held until end of transaction, even if it was originally taken by a
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 37b43ee1f8c..3a7f08ea2bf 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -20,76 +20,45 @@
*/
#include "postgres.h"
-#include "common/cryptohash.h"
#include "common/hashfn.h"
-#include "common/hmac.h"
-#include "jit/jit.h"
-#include "storage/bufmgr.h"
#include "storage/ipc.h"
#include "storage/predicate.h"
#include "storage/proc.h"
#include "utils/memutils.h"
-#include "utils/rel.h"
-#include "utils/resowner_private.h"
-#include "utils/snapmgr.h"
-
-
-/*
- * All resource IDs managed by this code are required to fit into a Datum,
- * which is fine since they are generally pointers or integers.
- *
- * Provide Datum conversion macros for a couple of things that are really
- * just "int".
- */
-#define FileGetDatum(file) Int32GetDatum(file)
-#define DatumGetFile(datum) ((File) DatumGetInt32(datum))
-#define BufferGetDatum(buffer) Int32GetDatum(buffer)
-#define DatumGetBuffer(datum) ((Buffer) DatumGetInt32(datum))
+#include "utils/plancache.h"
+#include "utils/resowner.h"
/*
- * ResourceArray is a common structure for storing all types of resource IDs.
- *
- * We manage small sets of resource IDs by keeping them in a simple array:
- * itemsarr[k] holds an ID, for 0 <= k < nitems <= maxitems = capacity.
- *
- * If a set grows large, we switch over to using open-addressing hashing.
- * Then, itemsarr[] is a hash table of "capacity" slots, with each
- * slot holding either an ID or "invalidval". nitems is the number of valid
- * items present; if it would exceed maxitems, we enlarge the array and
- * re-hash. In this mode, maxitems should be rather less than capacity so
- * that we don't waste too much time searching for empty slots.
+ * ResourceElem represents a reference associated with a resource owner.
*
- * In either mode, lastidx remembers the location of the last item inserted
- * or returned by GetAny; this speeds up searches in ResourceArrayRemove.
+ * All objects managed by this code are required to fit into a Datum,
+ * which is fine since they are generally pointers or integers.
*/
-typedef struct ResourceArray
+typedef struct ResourceElem
{
- Datum *itemsarr; /* buffer for storing values */
- Datum invalidval; /* value that is considered invalid */
- uint32 capacity; /* allocated length of itemsarr[] */
- uint32 nitems; /* how many items are stored in items array */
- uint32 maxitems; /* current limit on nitems before enlarging */
- uint32 lastidx; /* index of last item returned by GetAny */
-} ResourceArray;
+ Datum item;
+ ResourceOwnerFuncs *kind; /* NULL indicates a free hash table slot */
+} ResourceElem;
/*
- * Initially allocated size of a ResourceArray. Must be power of two since
- * we'll use (arraysize - 1) as mask for hashing.
+ * Size of the small fixed-size array to hold most-recently remembered resources.
*/
-#define RESARRAY_INIT_SIZE 16
+#define RESOWNER_ARRAY_SIZE 32
/*
- * When to switch to hashing vs. simple array logic in a ResourceArray.
+ * Initially allocated size of a ResourceOwner's hash. Must be power of two since
+ * we'll use (capacity - 1) as mask for hashing.
*/
-#define RESARRAY_MAX_ARRAY 64
-#define RESARRAY_IS_ARRAY(resarr) ((resarr)->capacity <= RESARRAY_MAX_ARRAY)
+#define RESOWNER_HASH_INIT_SIZE 64
/*
- * How many items may be stored in a resource array of given capacity.
+ * How many items may be stored in a hash of given capacity.
* When this number is reached, we must resize.
*/
-#define RESARRAY_MAX_ITEMS(capacity) \
- ((capacity) <= RESARRAY_MAX_ARRAY ? (capacity) : (capacity)/4 * 3)
+#define RESOWNER_HASH_MAX_ITEMS(capacity) ((capacity)/4 * 3)
+
+StaticAssertDecl(RESOWNER_HASH_MAX_ITEMS(RESOWNER_HASH_INIT_SIZE) > RESOWNER_ARRAY_SIZE,
+ "initial hash size too small compared to array size");
/*
* To speed up bulk releasing or reassigning locks from a resource owner to
@@ -119,24 +88,33 @@ typedef struct ResourceOwnerData
ResourceOwner nextchild; /* next child of same parent */
const char *name; /* name (just for debugging) */
- /* We have built-in support for remembering: */
- ResourceArray bufferarr; /* owned buffers */
- ResourceArray catrefarr; /* catcache references */
- ResourceArray catlistrefarr; /* catcache-list pins */
- ResourceArray relrefarr; /* relcache references */
- ResourceArray planrefarr; /* plancache references */
- ResourceArray tupdescarr; /* tupdesc references */
- ResourceArray snapshotarr; /* snapshot references */
- ResourceArray filearr; /* open temporary files */
- ResourceArray dsmarr; /* dynamic shmem segments */
- ResourceArray jitarr; /* JIT contexts */
- ResourceArray cryptohasharr; /* cryptohash contexts */
- ResourceArray hmacarr; /* HMAC contexts */
+ /*
+ * These structs keep track of the objects registered with this owner.
+ *
+ * We manage a small set of references by keeping them in a simple array.
+ * When the array gets full, all the elements in the array are moved to a
+ * hash table. This way, the array always contains a few most recently
+ * remembered references. To find a particular reference, you need to
+ * search both the array and the hash table.
+ */
+ ResourceElem arr[RESOWNER_ARRAY_SIZE];
+ uint32 narr; /* how many items are stored in the array */
+
+ /*
+ * The hash table. Uses open-addressing. 'nhash' is the number of items
+ * present; if it would exceed 'grow_at', we enlarge it and re-hash.
+ * 'grow_at' should be rather less than 'capacity' so that we don't waste
+ * too much time searching for empty slots.
+ */
+ ResourceElem *hash;
+ uint32 nhash; /* how many items are stored in the hash */
+ uint32 capacity; /* allocated length of hash[] */
+ uint32 grow_at; /* grow hash when reach this */
/* We can remember up to MAX_RESOWNER_LOCKS references to local locks. */
int nlocks; /* number of owned locks */
LOCALLOCK *locks[MAX_RESOWNER_LOCKS]; /* list of owned locks */
-} ResourceOwnerData;
+} ResourceOwnerData;
/*****************************************************************************
@@ -148,6 +126,18 @@ ResourceOwner CurTransactionResourceOwner = NULL;
ResourceOwner TopTransactionResourceOwner = NULL;
ResourceOwner AuxProcessResourceOwner = NULL;
+/* #define RESOWNER_STATS */
+/* #define RESOWNER_TRACE */
+
+#ifdef RESOWNER_STATS
+static int narray_lookups = 0;
+static int nhash_lookups = 0;
+#endif
+
+#ifdef RESOWNER_TRACE
+static int resowner_trace_counter = 0;
+#endif
+
/*
* List of add-on callbacks for resource releasing
*/
@@ -162,298 +152,340 @@ static ResourceReleaseCallbackItem *ResourceRelease_callbacks = NULL;
/* Internal routines */
-static void ResourceArrayInit(ResourceArray *resarr, Datum invalidval);
-static void ResourceArrayEnlarge(ResourceArray *resarr);
-static void ResourceArrayAdd(ResourceArray *resarr, Datum value);
-static bool ResourceArrayRemove(ResourceArray *resarr, Datum value);
-static bool ResourceArrayGetAny(ResourceArray *resarr, Datum *value);
-static void ResourceArrayFree(ResourceArray *resarr);
+static inline uint32 hash_resource_elem(Datum value, ResourceOwnerFuncs *kind);
+static void ResourceOwnerAddToHash(ResourceOwner owner, Datum value,
+ ResourceOwnerFuncs *kind);
+static void ResourceOwnerReleaseAll(ResourceOwner owner,
+ ResourceReleasePhase phase,
+ bool printLeakWarnings);
static void ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
-static void PrintRelCacheLeakWarning(Relation rel);
-static void PrintPlanCacheLeakWarning(CachedPlan *plan);
-static void PrintTupleDescLeakWarning(TupleDesc tupdesc);
-static void PrintSnapshotLeakWarning(Snapshot snapshot);
-static void PrintFileLeakWarning(File file);
-static void PrintDSMLeakWarning(dsm_segment *seg);
-static void PrintCryptoHashLeakWarning(Datum handle);
-static void PrintHMACLeakWarning(Datum handle);
/*****************************************************************************
* INTERNAL ROUTINES *
*****************************************************************************/
+static inline uint32
+hash_resource_elem(Datum value, ResourceOwnerFuncs *kind)
+{
+ Datum data[2];
+
+ data[0] = value;
+ data[1] = PointerGetDatum(kind);
+
+ return hash_bytes((unsigned char *) &data, 2 * SIZEOF_DATUM);
+}
/*
- * Initialize a ResourceArray
+ * Adds 'value' of given 'kind' to the ResourceOwner's hash table
*/
static void
-ResourceArrayInit(ResourceArray *resarr, Datum invalidval)
+ResourceOwnerAddToHash(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- /* Assert it's empty */
- Assert(resarr->itemsarr == NULL);
- Assert(resarr->capacity == 0);
- Assert(resarr->nitems == 0);
- Assert(resarr->maxitems == 0);
- /* Remember the appropriate "invalid" value */
- resarr->invalidval = invalidval;
- /* We don't allocate any storage until needed */
+ /* Insert into first free slot at or after hash location. */
+ uint32 mask = owner->capacity - 1;
+ uint32 idx;
+
+ Assert(kind != NULL);
+
+ idx = hash_resource_elem(value, kind) & mask;
+ for (;;)
+ {
+ if (owner->hash[idx].kind == NULL)
+ break; /* found a free slot */
+ idx = (idx + 1) & mask;
+ }
+ owner->hash[idx].item = value;
+ owner->hash[idx].kind = kind;
+ owner->nhash++;
}
/*
- * Make sure there is room for at least one more resource in an array.
- *
- * This is separate from actually inserting a resource because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
+ * Call the ReleaseResource callback on entries with given 'phase'.
*/
static void
-ResourceArrayEnlarge(ResourceArray *resarr)
+ResourceOwnerReleaseAll(ResourceOwner owner, ResourceReleasePhase phase,
+ bool printLeakWarnings)
{
- uint32 i,
- oldcap,
- newcap;
- Datum *olditemsarr;
- Datum *newitemsarr;
+ bool found;
+ int capacity;
- if (resarr->nitems < resarr->maxitems)
- return; /* no work needed */
-
- olditemsarr = resarr->itemsarr;
- oldcap = resarr->capacity;
-
- /* Double the capacity of the array (capacity must stay a power of 2!) */
- newcap = (oldcap > 0) ? oldcap * 2 : RESARRAY_INIT_SIZE;
- newitemsarr = (Datum *) MemoryContextAlloc(TopMemoryContext,
- newcap * sizeof(Datum));
- for (i = 0; i < newcap; i++)
- newitemsarr[i] = resarr->invalidval;
+ /*
+ * First handle all the entries in the array.
+ *
+ * Note that ReleaseResource() will call ResourceOwnerForget) and remove
+ * the entry from our array, so we just have to iterate till there is
+ * nothing left to remove.
+ */
+ do
+ {
+ found = false;
+ for (int i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].kind->phase == phase)
+ {
+ Datum value = owner->arr[i].item;
+ ResourceOwnerFuncs *kind = owner->arr[i].kind;
- /* We assume we can't fail below this point, so OK to scribble on resarr */
- resarr->itemsarr = newitemsarr;
- resarr->capacity = newcap;
- resarr->maxitems = RESARRAY_MAX_ITEMS(newcap);
- resarr->nitems = 0;
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+ found = true;
+ }
+ }
- if (olditemsarr != NULL)
- {
/*
- * Transfer any pre-existing entries into the new array; they don't
- * necessarily go where they were before, so this simple logic is the
- * best way. Note that if we were managing the set as a simple array,
- * the entries after nitems are garbage, but that shouldn't matter
- * because we won't get here unless nitems was equal to oldcap.
+ * If any resources were released, check again because some of the
+ * elements might have been moved by the callbacks. We don't want to
+ * miss them.
*/
- for (i = 0; i < oldcap; i++)
+ } while (found && owner->narr > 0);
+
+ /*
+ * Ok, the array has now been handled. Then the hash. Like with the
+ * array, ReleaseResource() will remove the entry from the hash.
+ */
+ do
+ {
+ capacity = owner->capacity;
+ for (int idx = 0; idx < capacity; idx++)
{
- if (olditemsarr[i] != resarr->invalidval)
- ResourceArrayAdd(resarr, olditemsarr[i]);
+ while (owner->hash[idx].kind != NULL &&
+ owner->hash[idx].kind->phase == phase)
+ {
+ Datum value = owner->hash[idx].item;
+ ResourceOwnerFuncs *kind = owner->hash[idx].kind;
+
+ if (printLeakWarnings)
+ kind->PrintLeakWarning(value);
+ kind->ReleaseResource(value);
+
+ /*
+ * If the same resource is remembered more than once in this
+ * resource owner, the ReleaseResource callback might've
+ * released a different copy of it. Because of that, loop to
+ * check the same index again.
+ */
+ }
}
- /* And release old array. */
- pfree(olditemsarr);
- }
-
- Assert(resarr->nitems < resarr->maxitems);
+ /*
+ * It's possible that the callbacks acquired more resources, causing
+ * the hash table to grow and the existing entries to be moved around.
+ * If that happened, scan the hash table again, so that we don't miss
+ * entries that were moved. (XXX: I'm not sure if any of the
+ * callbacks actually do that, but this is cheap to check, and better
+ * safe than sorry.)
+ */
+ Assert(owner->capacity >= capacity);
+ } while (capacity != owner->capacity);
}
+
+/*****************************************************************************
+ * EXPORTED ROUTINES *
+ *****************************************************************************/
+
+
/*
- * Add a resource to ResourceArray
+ * ResourceOwnerCreate
+ * Create an empty ResourceOwner.
*
- * Caller must have previously done ResourceArrayEnlarge()
+ * All ResourceOwner objects are kept in TopMemoryContext, since they should
+ * only be freed explicitly.
*/
-static void
-ResourceArrayAdd(ResourceArray *resarr, Datum value)
+ResourceOwner
+ResourceOwnerCreate(ResourceOwner parent, const char *name)
{
- uint32 idx;
+ ResourceOwner owner;
- Assert(value != resarr->invalidval);
- Assert(resarr->nitems < resarr->maxitems);
+ owner = (ResourceOwner) MemoryContextAllocZero(TopMemoryContext,
+ sizeof(ResourceOwnerData));
+ owner->name = name;
- if (RESARRAY_IS_ARRAY(resarr))
+ if (parent)
{
- /* Append to linear array. */
- idx = resarr->nitems;
+ owner->parent = parent;
+ owner->nextchild = parent->firstchild;
+ parent->firstchild = owner;
}
- else
- {
- /* Insert into first free slot at or after hash location. */
- uint32 mask = resarr->capacity - 1;
- idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (;;)
- {
- if (resarr->itemsarr[idx] == resarr->invalidval)
- break;
- idx = (idx + 1) & mask;
- }
- }
- resarr->lastidx = idx;
- resarr->itemsarr[idx] = value;
- resarr->nitems++;
+#ifdef RESOWNER_TRACE
+ elog(LOG, "CREATE %d: %p %s",
+ resowner_trace_counter++, owner, name);
+#endif
+
+ return owner;
}
/*
- * Remove a resource from ResourceArray
+ * Make sure there is room for at least one more resource in an array.
*
- * Returns true on success, false if resource was not found.
+ * This is separate from actually inserting a resource because if we run out
+ * of memory, it's critical to do so *before* acquiring the resource.
*
- * Note: if same resource ID appears more than once, one instance is removed.
+ * NB: Make sure there are no unrelated ResourceOwnerRemember() calls between
+ * your ResourceOwnerEnlarge() call and the ResourceOwnerRemember() call that
+ * you reserved the space for!
*/
-static bool
-ResourceArrayRemove(ResourceArray *resarr, Datum value)
+void
+ResourceOwnerEnlarge(ResourceOwner owner)
{
- uint32 i,
- idx,
- lastidx = resarr->lastidx;
-
- Assert(value != resarr->invalidval);
+ if (owner->narr < RESOWNER_ARRAY_SIZE)
+ return; /* no work needed */
- /* Search through all items, but try lastidx first. */
- if (RESARRAY_IS_ARRAY(resarr))
+ /* Is there space in the hash? If not, enlarge it. */
+ if (owner->narr + owner->nhash >= owner->grow_at)
{
- if (lastidx < resarr->nitems &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
- }
- for (i = 0; i < resarr->nitems; i++)
+ uint32 i,
+ oldcap,
+ newcap;
+ ResourceElem *oldhash;
+ ResourceElem *newhash;
+
+ oldhash = owner->hash;
+ oldcap = owner->capacity;
+
+ /* Double the capacity (it must stay a power of 2!) */
+ newcap = (oldcap > 0) ? oldcap * 2 : RESOWNER_HASH_INIT_SIZE;
+ newhash = (ResourceElem *) MemoryContextAllocZero(TopMemoryContext,
+ newcap * sizeof(ResourceElem));
+
+ /*
+ * We assume we can't fail below this point, so OK to scribble on the
+ * owner
+ */
+ owner->hash = newhash;
+ owner->capacity = newcap;
+ owner->grow_at = RESOWNER_HASH_MAX_ITEMS(newcap);
+ owner->nhash = 0;
+
+ if (oldhash != NULL)
{
- if (resarr->itemsarr[i] == value)
+ /*
+ * Transfer any pre-existing entries into the new hash table; they
+ * don't necessarily go where they were before, so this simple
+ * logic is the best way.
+ */
+ for (i = 0; i < oldcap; i++)
{
- resarr->itemsarr[i] = resarr->itemsarr[resarr->nitems - 1];
- resarr->nitems--;
- /* Update lastidx to make reverse-order removals fast. */
- resarr->lastidx = resarr->nitems - 1;
- return true;
+ if (oldhash[i].kind != NULL)
+ ResourceOwnerAddToHash(owner, oldhash[i].item, oldhash[i].kind);
}
+
+ /* And release old hash table. */
+ pfree(oldhash);
}
}
- else
- {
- uint32 mask = resarr->capacity - 1;
- if (lastidx < resarr->capacity &&
- resarr->itemsarr[lastidx] == value)
- {
- resarr->itemsarr[lastidx] = resarr->invalidval;
- resarr->nitems--;
- return true;
- }
- idx = DatumGetUInt32(hash_any((void *) &value, sizeof(value))) & mask;
- for (i = 0; i < resarr->capacity; i++)
- {
- if (resarr->itemsarr[idx] == value)
- {
- resarr->itemsarr[idx] = resarr->invalidval;
- resarr->nitems--;
- return true;
- }
- idx = (idx + 1) & mask;
- }
+ /* Move items from the array to the hash */
+ Assert(owner->narr == RESOWNER_ARRAY_SIZE);
+ for (int i = 0; i < owner->narr; i++)
+ {
+ ResourceOwnerAddToHash(owner, owner->arr[i].item, owner->arr[i].kind);
}
+ owner->narr = 0;
- return false;
+ Assert(owner->nhash < owner->grow_at);
}
/*
- * Get any convenient entry in a ResourceArray.
- *
- * "Convenient" is defined as "easy for ResourceArrayRemove to remove";
- * we help that along by setting lastidx to match. This avoids O(N^2) cost
- * when removing all ResourceArray items during ResourceOwner destruction.
+ * Remember that an object is owner by a ReourceOwner
*
- * Returns true if we found an element, or false if the array is empty.
+ * Caller must have previously done ResourceOwnerEnlarge()
*/
-static bool
-ResourceArrayGetAny(ResourceArray *resarr, Datum *value)
+void
+ResourceOwnerRemember(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- if (resarr->nitems == 0)
- return false;
+ uint32 idx;
- if (RESARRAY_IS_ARRAY(resarr))
- {
- /* Linear array: just return the first element. */
- resarr->lastidx = 0;
- }
- else
- {
- /* Hash: search forward from wherever we were last. */
- uint32 mask = resarr->capacity - 1;
+#ifdef RESOWNER_TRACE
+ elog(LOG, "REMEMBER %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
- for (;;)
- {
- resarr->lastidx &= mask;
- if (resarr->itemsarr[resarr->lastidx] != resarr->invalidval)
- break;
- resarr->lastidx++;
- }
+ if (owner->narr >= RESOWNER_ARRAY_SIZE)
+ {
+ /* forgot to call ResourceOwnerEnlarge? */
+ elog(ERROR, "ResourceOwnerRemember called but array was full");
}
- *value = resarr->itemsarr[resarr->lastidx];
- return true;
+ /* Append to linear array. */
+ idx = owner->narr;
+ owner->arr[idx].item = value;
+ owner->arr[idx].kind = kind;
+ owner->narr++;
}
/*
- * Trash a ResourceArray (we don't care about its state after this)
+ * Forget that an object is owned by a ResourceOwner
+ *
+ * Note: if same resource ID is associated with the ResourceOwner more than once,
+ * one instance is removed.
*/
-static void
-ResourceArrayFree(ResourceArray *resarr)
+void
+ResourceOwnerForget(ResourceOwner owner, Datum value, ResourceOwnerFuncs *kind)
{
- if (resarr->itemsarr)
- pfree(resarr->itemsarr);
-}
-
+ uint32 i,
+ idx;
-/*****************************************************************************
- * EXPORTED ROUTINES *
- *****************************************************************************/
+#ifdef RESOWNER_TRACE
+ elog(LOG, "FORGET %d: owner %p value " UINT64_FORMAT ", kind: %s",
+ resowner_trace_counter++, owner, DatumGetUInt64(value), kind->name);
+#endif
+ /* Search through all items, but check the array first. */
+ for (i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].item == value &&
+ owner->arr[i].kind == kind)
+ {
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
-/*
- * ResourceOwnerCreate
- * Create an empty ResourceOwner.
- *
- * All ResourceOwner objects are kept in TopMemoryContext, since they should
- * only be freed explicitly.
- */
-ResourceOwner
-ResourceOwnerCreate(ResourceOwner parent, const char *name)
-{
- ResourceOwner owner;
+#ifdef RESOWNER_STATS
+ narray_lookups++;
+#endif
- owner = (ResourceOwner) MemoryContextAllocZero(TopMemoryContext,
- sizeof(ResourceOwnerData));
- owner->name = name;
+ return;
+ }
+ }
- if (parent)
+ /* Search hash */
+ if (owner->nhash > 0)
{
- owner->parent = parent;
- owner->nextchild = parent->firstchild;
- parent->firstchild = owner;
- }
+ uint32 mask = owner->capacity - 1;
- ResourceArrayInit(&(owner->bufferarr), BufferGetDatum(InvalidBuffer));
- ResourceArrayInit(&(owner->catrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->catlistrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->relrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->planrefarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->tupdescarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->snapshotarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->filearr), FileGetDatum(-1));
- ResourceArrayInit(&(owner->dsmarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->jitarr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->cryptohasharr), PointerGetDatum(NULL));
- ResourceArrayInit(&(owner->hmacarr), PointerGetDatum(NULL));
+ idx = hash_resource_elem(value, kind) & mask;
+ for (i = 0; i < owner->capacity; i++)
+ {
+ if (owner->hash[idx].item == value &&
+ owner->hash[idx].kind == kind)
+ {
+ owner->hash[idx].item = (Datum) 0;
+ owner->hash[idx].kind = NULL;
+ owner->nhash--;
+
+#ifdef RESOWNER_STATS
+ nhash_lookups++;
+#endif
+ return;
+ }
+ idx = (idx + 1) & mask;
+ }
+ }
- return owner;
+ /*
+ * Use %p to print the reference, since most objects tracked by a resource
+ * owner are pointers. It's a bit misleading if it's not a pointer, but
+ * this is a programmer error, anyway.
+ */
+ elog(ERROR, "%s %p is not owned by resource owner %s",
+ kind->name, DatumGetPointer(value), owner->name);
}
/*
@@ -490,6 +522,15 @@ ResourceOwnerRelease(ResourceOwner owner,
{
/* There's not currently any setup needed before recursing */
ResourceOwnerReleaseInternal(owner, phase, isCommit, isTopLevel);
+
+#ifdef RESOWNER_STATS
+ if (isTopLevel)
+ {
+ elog(LOG, "RESOWNER STATS: lookups: array %d, hash %d", narray_lookups, nhash_lookups);
+ narray_lookups = 0;
+ nhash_lookups = 0;
+ }
+#endif
}
static void
@@ -502,7 +543,6 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
ResourceOwner save;
ResourceReleaseCallbackItem *item;
ResourceReleaseCallbackItem *next;
- Datum foundres;
/* Recurse to handle descendants */
for (child = owner->firstchild; child != NULL; child = child->nextchild)
@@ -518,71 +558,13 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
if (phase == RESOURCE_RELEASE_BEFORE_LOCKS)
{
/*
- * Release buffer pins. Note that ReleaseBuffer will remove the
- * buffer entry from our array, so we just have to iterate till there
- * are none.
+ * Release all resources that need to be released before the locks.
*
- * During a commit, there shouldn't be any remaining pins --- that
- * would indicate failure to clean up the executor correctly --- so
- * issue warnings. In the abort case, just clean up quietly.
+ * During a commit, there shouldn't be any remaining resources ---
+ * that would indicate failure to clean up the executor correctly ---
+ * so issue warnings. In the abort case, just clean up quietly.
*/
- while (ResourceArrayGetAny(&(owner->bufferarr), &foundres))
- {
- Buffer res = DatumGetBuffer(foundres);
-
- if (isCommit)
- PrintBufferLeakWarning(res);
- ReleaseBuffer(res);
- }
-
- /* Ditto for relcache references */
- while (ResourceArrayGetAny(&(owner->relrefarr), &foundres))
- {
- Relation res = (Relation) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintRelCacheLeakWarning(res);
- RelationClose(res);
- }
-
- /* Ditto for dynamic shared memory segments */
- while (ResourceArrayGetAny(&(owner->dsmarr), &foundres))
- {
- dsm_segment *res = (dsm_segment *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintDSMLeakWarning(res);
- dsm_detach(res);
- }
-
- /* Ditto for JIT contexts */
- while (ResourceArrayGetAny(&(owner->jitarr), &foundres))
- {
- JitContext *context = (JitContext *) DatumGetPointer(foundres);
-
- jit_release_context(context);
- }
-
- /* Ditto for cryptohash contexts */
- while (ResourceArrayGetAny(&(owner->cryptohasharr), &foundres))
- {
- pg_cryptohash_ctx *context =
- (pg_cryptohash_ctx *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCryptoHashLeakWarning(foundres);
- pg_cryptohash_free(context);
- }
-
- /* Ditto for HMAC contexts */
- while (ResourceArrayGetAny(&(owner->hmacarr), &foundres))
- {
- pg_hmac_ctx *context = (pg_hmac_ctx *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintHMACLeakWarning(foundres);
- pg_hmac_free(context);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
else if (phase == RESOURCE_RELEASE_LOCKS)
{
@@ -635,70 +617,9 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
else if (phase == RESOURCE_RELEASE_AFTER_LOCKS)
{
/*
- * Release catcache references. Note that ReleaseCatCache will remove
- * the catref entry from our array, so we just have to iterate till
- * there are none.
- *
- * As with buffer pins, warn if any are left at commit time.
+ * Release all resources that need to be released after the locks.
*/
- while (ResourceArrayGetAny(&(owner->catrefarr), &foundres))
- {
- HeapTuple res = (HeapTuple) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheLeakWarning(res);
- ReleaseCatCache(res);
- }
-
- /* Ditto for catcache lists */
- while (ResourceArrayGetAny(&(owner->catlistrefarr), &foundres))
- {
- CatCList *res = (CatCList *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintCatCacheListLeakWarning(res);
- ReleaseCatCacheList(res);
- }
-
- /* Ditto for plancache references */
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
- {
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintPlanCacheLeakWarning(res);
- ReleaseCachedPlan(res, owner);
- }
-
- /* Ditto for tupdesc references */
- while (ResourceArrayGetAny(&(owner->tupdescarr), &foundres))
- {
- TupleDesc res = (TupleDesc) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintTupleDescLeakWarning(res);
- DecrTupleDescRefCount(res);
- }
-
- /* Ditto for snapshot references */
- while (ResourceArrayGetAny(&(owner->snapshotarr), &foundres))
- {
- Snapshot res = (Snapshot) DatumGetPointer(foundres);
-
- if (isCommit)
- PrintSnapshotLeakWarning(res);
- UnregisterSnapshot(res);
- }
-
- /* Ditto for temporary files */
- while (ResourceArrayGetAny(&(owner->filearr), &foundres))
- {
- File res = DatumGetFile(foundres);
-
- if (isCommit)
- PrintFileLeakWarning(res);
- FileClose(res);
- }
+ ResourceOwnerReleaseAll(owner, phase, isCommit);
}
/* Let add-on modules get a chance too */
@@ -722,13 +643,42 @@ ResourceOwnerReleaseInternal(ResourceOwner owner,
void
ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner)
{
- Datum foundres;
+ /* array first */
+ for (int i = 0; i < owner->narr; i++)
+ {
+ if (owner->arr[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->arr[i].item);
+
+ owner->arr[i] = owner->arr[owner->narr - 1];
+ owner->narr--;
+ i--;
+
+ /*
+ * pass 'NULL' because we already removed the entry from the
+ * resowner
+ */
+ ReleaseCachedPlan(planref, NULL);
+ }
+ }
- while (ResourceArrayGetAny(&(owner->planrefarr), &foundres))
+ /* Then hash */
+ for (int i = 0; i < owner->capacity; i++)
{
- CachedPlan *res = (CachedPlan *) DatumGetPointer(foundres);
+ if (owner->hash[i].kind == &planref_resowner_funcs)
+ {
+ CachedPlan *planref = (CachedPlan *) DatumGetPointer(owner->hash[i].item);
+
+ owner->hash[i].item = (Datum) 0;
+ owner->hash[i].kind = NULL;
+ owner->nhash--;
- ReleaseCachedPlan(res, owner);
+ /*
+ * pass 'NULL' because we already removed the entry from the
+ * resowner
+ */
+ ReleaseCachedPlan(planref, NULL);
+ }
}
}
@@ -745,20 +695,15 @@ ResourceOwnerDelete(ResourceOwner owner)
Assert(owner != CurrentResourceOwner);
/* And it better not own any resources, either */
- Assert(owner->bufferarr.nitems == 0);
- Assert(owner->catrefarr.nitems == 0);
- Assert(owner->catlistrefarr.nitems == 0);
- Assert(owner->relrefarr.nitems == 0);
- Assert(owner->planrefarr.nitems == 0);
- Assert(owner->tupdescarr.nitems == 0);
- Assert(owner->snapshotarr.nitems == 0);
- Assert(owner->filearr.nitems == 0);
- Assert(owner->dsmarr.nitems == 0);
- Assert(owner->jitarr.nitems == 0);
- Assert(owner->cryptohasharr.nitems == 0);
- Assert(owner->hmacarr.nitems == 0);
+ Assert(owner->narr == 0);
+ Assert(owner->nhash == 0);
Assert(owner->nlocks == 0 || owner->nlocks == MAX_RESOWNER_LOCKS + 1);
+#ifdef RESOWNER_TRACE
+ elog(LOG, "DELETE %d: %p %s",
+ resowner_trace_counter++, owner, owner->name);
+#endif
+
/*
* Delete children. The recursive call will delink the child from me, so
* just iterate as long as there is a child.
@@ -774,19 +719,8 @@ ResourceOwnerDelete(ResourceOwner owner)
ResourceOwnerNewParent(owner, NULL);
/* And free the object. */
- ResourceArrayFree(&(owner->bufferarr));
- ResourceArrayFree(&(owner->catrefarr));
- ResourceArrayFree(&(owner->catlistrefarr));
- ResourceArrayFree(&(owner->relrefarr));
- ResourceArrayFree(&(owner->planrefarr));
- ResourceArrayFree(&(owner->tupdescarr));
- ResourceArrayFree(&(owner->snapshotarr));
- ResourceArrayFree(&(owner->filearr));
- ResourceArrayFree(&(owner->dsmarr));
- ResourceArrayFree(&(owner->jitarr));
- ResourceArrayFree(&(owner->cryptohasharr));
- ResourceArrayFree(&(owner->hmacarr));
-
+ if (owner->hash)
+ pfree(owner->hash);
pfree(owner);
}
@@ -844,11 +778,10 @@ ResourceOwnerNewParent(ResourceOwner owner,
/*
* Register or deregister callback functions for resource cleanup
*
- * These functions are intended for use by dynamically loaded modules.
- * For built-in modules we generally just hardwire the appropriate calls.
- *
- * Note that the callback occurs post-commit or post-abort, so the callback
- * functions can only do noncritical cleanup.
+ * These functions can be used by dynamically loaded modules. These used
+ * to be the only way for an extension to register custom resource types
+ * with a resource owner, but nowadays it is easier to define a new
+ * ResourceOwnerFuncs instance with custom callbacks.
*/
void
RegisterResourceReleaseCallback(ResourceReleaseCallback callback, void *arg)
@@ -938,58 +871,20 @@ ReleaseAuxProcessResourcesCallback(int code, Datum arg)
ReleaseAuxProcessResources(isCommit);
}
-
/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * buffer array.
+ * Remember that a Local Lock is owned by a ResourceOwner
*
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
+ * This is different from the other Remember functions in that the list of
+ * locks is only a lossy cache. It can hold up to MAX_RESOWNER_LOCKS entries,
+ * and when it overflows, we stop tracking locks. The point of only remembering
+ * only up to MAX_RESOWNER_LOCKS entries is that if a lot of locks are held,
+ * ResourceOwnerForgetLock doesn't need to scan through a large array to find
+ * the entry.
*/
void
-ResourceOwnerEnlargeBuffers(ResourceOwner owner)
+ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock)
{
- /* We used to allow pinning buffers without a resowner, but no more */
- Assert(owner != NULL);
- ResourceArrayEnlarge(&(owner->bufferarr));
-}
-
-/*
- * Remember that a buffer pin is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeBuffers()
- */
-void
-ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer)
-{
- ResourceArrayAdd(&(owner->bufferarr), BufferGetDatum(buffer));
-}
-
-/*
- * Forget that a buffer pin is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer)
-{
- if (!ResourceArrayRemove(&(owner->bufferarr), BufferGetDatum(buffer)))
- elog(ERROR, "buffer %d is not owned by resource owner %s",
- buffer, owner->name);
-}
-
-/*
- * Remember that a Local Lock is owned by a ResourceOwner
- *
- * This is different from the other Remember functions in that the list of
- * locks is only a lossy cache. It can hold up to MAX_RESOWNER_LOCKS entries,
- * and when it overflows, we stop tracking locks. The point of only remembering
- * only up to MAX_RESOWNER_LOCKS entries is that if a lot of locks are held,
- * ResourceOwnerForgetLock doesn't need to scan through a large array to find
- * the entry.
- */
-void
-ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock)
-{
- Assert(locallock != NULL);
+ Assert(locallock != NULL);
if (owner->nlocks > MAX_RESOWNER_LOCKS)
return; /* we have already overflowed */
@@ -1027,469 +922,3 @@ ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock)
elog(ERROR, "lock reference %p is not owned by resource owner %s",
locallock, owner->name);
}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catrefarr));
-}
-
-/*
- * Remember that a catcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheRefs()
- */
-void
-ResourceOwnerRememberCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- ResourceArrayAdd(&(owner->catrefarr), PointerGetDatum(tuple));
-}
-
-/*
- * Forget that a catcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheRef(ResourceOwner owner, HeapTuple tuple)
-{
- if (!ResourceArrayRemove(&(owner->catrefarr), PointerGetDatum(tuple)))
- elog(ERROR, "catcache reference %p is not owned by resource owner %s",
- tuple, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * catcache-list reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->catlistrefarr));
-}
-
-/*
- * Remember that a catcache-list reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCatCacheListRefs()
- */
-void
-ResourceOwnerRememberCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- ResourceArrayAdd(&(owner->catlistrefarr), PointerGetDatum(list));
-}
-
-/*
- * Forget that a catcache-list reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCatCacheListRef(ResourceOwner owner, CatCList *list)
-{
- if (!ResourceArrayRemove(&(owner->catlistrefarr), PointerGetDatum(list)))
- elog(ERROR, "catcache list reference %p is not owned by resource owner %s",
- list, owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * relcache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeRelationRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->relrefarr));
-}
-
-/*
- * Remember that a relcache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeRelationRefs()
- */
-void
-ResourceOwnerRememberRelationRef(ResourceOwner owner, Relation rel)
-{
- ResourceArrayAdd(&(owner->relrefarr), PointerGetDatum(rel));
-}
-
-/*
- * Forget that a relcache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetRelationRef(ResourceOwner owner, Relation rel)
-{
- if (!ResourceArrayRemove(&(owner->relrefarr), PointerGetDatum(rel)))
- elog(ERROR, "relcache reference %s is not owned by resource owner %s",
- RelationGetRelationName(rel), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintRelCacheLeakWarning(Relation rel)
-{
- elog(WARNING, "relcache reference leak: relation \"%s\" not closed",
- RelationGetRelationName(rel));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * plancache reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->planrefarr));
-}
-
-/*
- * Remember that a plancache reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargePlanCacheRefs()
- */
-void
-ResourceOwnerRememberPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- ResourceArrayAdd(&(owner->planrefarr), PointerGetDatum(plan));
-}
-
-/*
- * Forget that a plancache reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetPlanCacheRef(ResourceOwner owner, CachedPlan *plan)
-{
- if (!ResourceArrayRemove(&(owner->planrefarr), PointerGetDatum(plan)))
- elog(ERROR, "plancache reference %p is not owned by resource owner %s",
- plan, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintPlanCacheLeakWarning(CachedPlan *plan)
-{
- elog(WARNING, "plancache reference leak: plan %p not closed", plan);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * tupdesc reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeTupleDescs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->tupdescarr));
-}
-
-/*
- * Remember that a tupdesc reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeTupleDescs()
- */
-void
-ResourceOwnerRememberTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- ResourceArrayAdd(&(owner->tupdescarr), PointerGetDatum(tupdesc));
-}
-
-/*
- * Forget that a tupdesc reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetTupleDesc(ResourceOwner owner, TupleDesc tupdesc)
-{
- if (!ResourceArrayRemove(&(owner->tupdescarr), PointerGetDatum(tupdesc)))
- elog(ERROR, "tupdesc reference %p is not owned by resource owner %s",
- tupdesc, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintTupleDescLeakWarning(TupleDesc tupdesc)
-{
- elog(WARNING,
- "TupleDesc reference leak: TupleDesc %p (%u,%d) still referenced",
- tupdesc, tupdesc->tdtypeid, tupdesc->tdtypmod);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * snapshot reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeSnapshots(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->snapshotarr));
-}
-
-/*
- * Remember that a snapshot reference is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeSnapshots()
- */
-void
-ResourceOwnerRememberSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- ResourceArrayAdd(&(owner->snapshotarr), PointerGetDatum(snapshot));
-}
-
-/*
- * Forget that a snapshot reference is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetSnapshot(ResourceOwner owner, Snapshot snapshot)
-{
- if (!ResourceArrayRemove(&(owner->snapshotarr), PointerGetDatum(snapshot)))
- elog(ERROR, "snapshot reference %p is not owned by resource owner %s",
- snapshot, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintSnapshotLeakWarning(Snapshot snapshot)
-{
- elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
- snapshot);
-}
-
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * files reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeFiles(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->filearr));
-}
-
-/*
- * Remember that a temporary file is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeFiles()
- */
-void
-ResourceOwnerRememberFile(ResourceOwner owner, File file)
-{
- ResourceArrayAdd(&(owner->filearr), FileGetDatum(file));
-}
-
-/*
- * Forget that a temporary file is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetFile(ResourceOwner owner, File file)
-{
- if (!ResourceArrayRemove(&(owner->filearr), FileGetDatum(file)))
- elog(ERROR, "temporary file %d is not owned by resource owner %s",
- file, owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintFileLeakWarning(File file)
-{
- elog(WARNING, "temporary file leak: File %d still referenced",
- file);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * dynamic shmem segment reference array.
- *
- * This is separate from actually inserting an entry because if we run out
- * of memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeDSMs(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->dsmarr));
-}
-
-/*
- * Remember that a dynamic shmem segment is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeDSMs()
- */
-void
-ResourceOwnerRememberDSM(ResourceOwner owner, dsm_segment *seg)
-{
- ResourceArrayAdd(&(owner->dsmarr), PointerGetDatum(seg));
-}
-
-/*
- * Forget that a dynamic shmem segment is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetDSM(ResourceOwner owner, dsm_segment *seg)
-{
- if (!ResourceArrayRemove(&(owner->dsmarr), PointerGetDatum(seg)))
- elog(ERROR, "dynamic shared memory segment %u is not owned by resource owner %s",
- dsm_segment_handle(seg), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintDSMLeakWarning(dsm_segment *seg)
-{
- elog(WARNING, "dynamic shared memory leak: segment %u still referenced",
- dsm_segment_handle(seg));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * JIT context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeJIT(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->jitarr));
-}
-
-/*
- * Remember that a JIT context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeJIT()
- */
-void
-ResourceOwnerRememberJIT(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->jitarr), handle);
-}
-
-/*
- * Forget that a JIT context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetJIT(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->jitarr), handle))
- elog(ERROR, "JIT context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * cryptohash context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeCryptoHash(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->cryptohasharr));
-}
-
-/*
- * Remember that a cryptohash context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeCryptoHash()
- */
-void
-ResourceOwnerRememberCryptoHash(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->cryptohasharr), handle);
-}
-
-/*
- * Forget that a cryptohash context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetCryptoHash(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->cryptohasharr), handle))
- elog(ERROR, "cryptohash context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintCryptoHashLeakWarning(Datum handle)
-{
- elog(WARNING, "cryptohash context reference leak: context %p still referenced",
- DatumGetPointer(handle));
-}
-
-/*
- * Make sure there is room for at least one more entry in a ResourceOwner's
- * hmac context reference array.
- *
- * This is separate from actually inserting an entry because if we run out of
- * memory, it's critical to do so *before* acquiring the resource.
- */
-void
-ResourceOwnerEnlargeHMAC(ResourceOwner owner)
-{
- ResourceArrayEnlarge(&(owner->hmacarr));
-}
-
-/*
- * Remember that a HMAC context is owned by a ResourceOwner
- *
- * Caller must have previously done ResourceOwnerEnlargeHMAC()
- */
-void
-ResourceOwnerRememberHMAC(ResourceOwner owner, Datum handle)
-{
- ResourceArrayAdd(&(owner->hmacarr), handle);
-}
-
-/*
- * Forget that a HMAC context is owned by a ResourceOwner
- */
-void
-ResourceOwnerForgetHMAC(ResourceOwner owner, Datum handle)
-{
- if (!ResourceArrayRemove(&(owner->hmacarr), handle))
- elog(ERROR, "HMAC context %p is not owned by resource owner %s",
- DatumGetPointer(handle), owner->name);
-}
-
-/*
- * Debugging subroutine
- */
-static void
-PrintHMACLeakWarning(Datum handle)
-{
- elog(WARNING, "HMAC context reference leak: context %p still referenced",
- DatumGetPointer(handle));
-}
diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c
index f1f2ddac17c..ceff63a3ecf 100644
--- a/src/backend/utils/time/snapmgr.c
+++ b/src/backend/utils/time/snapmgr.c
@@ -57,6 +57,7 @@
#include "lib/pairingheap.h"
#include "miscadmin.h"
#include "port/pg_lfind.h"
+#include "storage/fd.h"
#include "storage/predicate.h"
#include "storage/proc.h"
#include "storage/procarray.h"
@@ -67,7 +68,7 @@
#include "utils/memutils.h"
#include "utils/old_snapshot.h"
#include "utils/rel.h"
-#include "utils/resowner_private.h"
+#include "utils/resowner.h"
#include "utils/snapmgr.h"
#include "utils/syscache.h"
#include "utils/timestamp.h"
@@ -175,6 +176,24 @@ static Snapshot CopySnapshot(Snapshot snapshot);
static void FreeSnapshot(Snapshot snapshot);
static void SnapshotResetXmin(void);
+/* ResourceOwner callbacks to track snapshot references */
+static void ResOwnerReleaseSnapshot(Datum res);
+static void ResOwnerPrintSnapshotLeakWarning(Datum res);
+
+static ResourceOwnerFuncs snapshot_resowner_funcs =
+{
+ .name = "snapshot reference",
+ .phase = RESOURCE_RELEASE_AFTER_LOCKS,
+ .ReleaseResource = ResOwnerReleaseSnapshot,
+ .PrintLeakWarning = ResOwnerPrintSnapshotLeakWarning
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberSnapshot(owner, snap) \
+ ResourceOwnerRemember(owner, PointerGetDatum(snap), &snapshot_resowner_funcs)
+#define ResourceOwnerForgetSnapshot(owner, snap) \
+ ResourceOwnerForget(owner, PointerGetDatum(snap), &snapshot_resowner_funcs)
+
/*
* Snapshot fields to be serialized.
*
@@ -850,7 +869,7 @@ RegisterSnapshotOnOwner(Snapshot snapshot, ResourceOwner owner)
snap = snapshot->copied ? snapshot : CopySnapshot(snapshot);
/* and tell resowner.c about it */
- ResourceOwnerEnlargeSnapshots(owner);
+ ResourceOwnerEnlarge(owner);
snap->regd_count++;
ResourceOwnerRememberSnapshot(owner, snap);
@@ -2379,3 +2398,19 @@ XidInMVCCSnapshot(TransactionId xid, Snapshot snapshot)
return false;
}
+
+/*
+ * ResourceOwner callbacks
+ */
+static void
+ResOwnerReleaseSnapshot(Datum res)
+{
+ UnregisterSnapshot((Snapshot) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintSnapshotLeakWarning(Datum res)
+{
+ elog(WARNING, "Snapshot reference leak: Snapshot %p still referenced",
+ DatumGetPointer(res));
+}
diff --git a/src/common/cryptohash_openssl.c b/src/common/cryptohash_openssl.c
index 2fb29b1f855..5ed746e7d69 100644
--- a/src/common/cryptohash_openssl.c
+++ b/src/common/cryptohash_openssl.c
@@ -31,7 +31,6 @@
#ifndef FRONTEND
#include "utils/memutils.h"
#include "utils/resowner.h"
-#include "utils/resowner_private.h"
#endif
/*
@@ -74,6 +73,27 @@ struct pg_cryptohash_ctx
#endif
};
+/* ResourceOwner callbacks to hold cryptohash contexts */
+#ifndef FRONTEND
+static void ResOwnerReleaseCryptoHash(Datum res);
+static void ResOwnerPrintCryptoHashLeakWarning(Datum res);
+
+static ResourceOwnerFuncs cryptohash_resowner_funcs =
+{
+ /* relcache references */
+ .name = "OpenSSL cryptohash context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseCryptoHash,
+ .PrintLeakWarning = ResOwnerPrintCryptoHashLeakWarning,
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberCryptoHash(owner, ctx) \
+ ResourceOwnerRemember(owner, PointerGetDatum(ctx), &cryptohash_resowner_funcs)
+#define ResourceOwnerForgetCryptoHash(owner, ctx) \
+ ResourceOwnerForget(owner, PointerGetDatum(ctx), &cryptohash_resowner_funcs)
+#endif
+
static const char *
SSLerrmessage(unsigned long ecode)
{
@@ -104,7 +124,7 @@ pg_cryptohash_create(pg_cryptohash_type type)
* allocation to avoid leaking.
*/
#ifndef FRONTEND
- ResourceOwnerEnlargeCryptoHash(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
#endif
ctx = ALLOC(sizeof(pg_cryptohash_ctx));
@@ -138,8 +158,7 @@ pg_cryptohash_create(pg_cryptohash_type type)
#ifndef FRONTEND
ctx->resowner = CurrentResourceOwner;
- ResourceOwnerRememberCryptoHash(CurrentResourceOwner,
- PointerGetDatum(ctx));
+ ResourceOwnerRememberCryptoHash(CurrentResourceOwner, ctx);
#endif
return ctx;
@@ -307,8 +326,7 @@ pg_cryptohash_free(pg_cryptohash_ctx *ctx)
EVP_MD_CTX_destroy(ctx->evpctx);
#ifndef FRONTEND
- ResourceOwnerForgetCryptoHash(ctx->resowner,
- PointerGetDatum(ctx));
+ ResourceOwnerForgetCryptoHash(ctx->resowner, ctx);
#endif
explicit_bzero(ctx, sizeof(pg_cryptohash_ctx));
@@ -351,3 +369,21 @@ pg_cryptohash_error(pg_cryptohash_ctx *ctx)
Assert(false); /* cannot be reached */
return _("success");
}
+
+/*
+ * ResourceOwner callbacks
+ */
+#ifndef FRONTEND
+static void
+ResOwnerReleaseCryptoHash(Datum res)
+{
+ pg_cryptohash_free((pg_cryptohash_ctx *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintCryptoHashLeakWarning(Datum res)
+{
+ elog(WARNING, "cryptohash context reference leak: context %p still referenced",
+ DatumGetPointer(res));
+}
+#endif
diff --git a/src/common/hmac_openssl.c b/src/common/hmac_openssl.c
index 8874d6a240c..dfe617327ed 100644
--- a/src/common/hmac_openssl.c
+++ b/src/common/hmac_openssl.c
@@ -31,7 +31,6 @@
#ifndef FRONTEND
#include "utils/memutils.h"
#include "utils/resowner.h"
-#include "utils/resowner_private.h"
#endif
/*
@@ -73,6 +72,27 @@ struct pg_hmac_ctx
#endif
};
+/* ResourceOwner callbacks to hold HMAC contexts */
+#ifndef FRONTEND
+static void ResOwnerReleaseHMAC(Datum res);
+static void ResOwnerPrintHMACLeakWarning(Datum res);
+
+static ResourceOwnerFuncs hmac_resowner_funcs =
+{
+ /* relcache references */
+ .name = "OpenSSL HMAC context",
+ .phase = RESOURCE_RELEASE_BEFORE_LOCKS,
+ .ReleaseResource = ResOwnerReleaseHMAC,
+ .PrintLeakWarning = ResOwnerPrintHMACLeakWarning,
+};
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberHMAC(owner, ctx) \
+ ResourceOwnerRemember(owner, PointerGetDatum(ctx), &hmac_resowner_funcs)
+#define ResourceOwnerForgetHMAC(owner, ctx) \
+ ResourceOwnerForget(owner, PointerGetDatum(ctx), &hmac_resowner_funcs)
+#endif
+
static const char *
SSLerrmessage(unsigned long ecode)
{
@@ -115,7 +135,7 @@ pg_hmac_create(pg_cryptohash_type type)
ERR_clear_error();
#ifdef HAVE_HMAC_CTX_NEW
#ifndef FRONTEND
- ResourceOwnerEnlargeHMAC(CurrentResourceOwner);
+ ResourceOwnerEnlarge(CurrentResourceOwner);
#endif
ctx->hmacctx = HMAC_CTX_new();
#else
@@ -137,7 +157,7 @@ pg_hmac_create(pg_cryptohash_type type)
#ifdef HAVE_HMAC_CTX_NEW
#ifndef FRONTEND
ctx->resowner = CurrentResourceOwner;
- ResourceOwnerRememberHMAC(CurrentResourceOwner, PointerGetDatum(ctx));
+ ResourceOwnerRememberHMAC(CurrentResourceOwner, ctx);
#endif
#else
memset(ctx->hmacctx, 0, sizeof(HMAC_CTX));
@@ -303,7 +323,7 @@ pg_hmac_free(pg_hmac_ctx *ctx)
#ifdef HAVE_HMAC_CTX_FREE
HMAC_CTX_free(ctx->hmacctx);
#ifndef FRONTEND
- ResourceOwnerForgetHMAC(ctx->resowner, PointerGetDatum(ctx));
+ ResourceOwnerForgetHMAC(ctx->resowner, ctx);
#endif
#else
explicit_bzero(ctx->hmacctx, sizeof(HMAC_CTX));
@@ -346,3 +366,21 @@ pg_hmac_error(pg_hmac_ctx *ctx)
Assert(false); /* cannot be reached */
return _("success");
}
+
+/*
+ * ResourceOwner callbacks
+ */
+#ifndef FRONTEND
+static void
+ResOwnerReleaseHMAC(Datum res)
+{
+ pg_hmac_free((pg_hmac_ctx *) DatumGetPointer(res));
+}
+
+static void
+ResOwnerPrintHMACLeakWarning(Datum res)
+{
+ elog(WARNING, "HMAC context reference leak: context %p still referenced",
+ DatumGetPointer(res));
+}
+#endif
diff --git a/src/include/storage/buf_internals.h b/src/include/storage/buf_internals.h
index 406db6be783..a4ca7c9ee6e 100644
--- a/src/include/storage/buf_internals.h
+++ b/src/include/storage/buf_internals.h
@@ -382,6 +382,15 @@ typedef struct CkptSortItem
extern PGDLLIMPORT CkptSortItem *CkptBufferIds;
+/* ResourceOwner callbacks to hold buffer pins */
+extern ResourceOwnerFuncs buffer_resowner_funcs;
+
+/* Convenience wrappers over ResourceOwnerRemember/Forget */
+#define ResourceOwnerRememberBuffer(owner, buffer) \
+ ResourceOwnerRemember(owner, Int32GetDatum(buffer), &buffer_resowner_funcs)
+#define ResourceOwnerForgetBuffer(owner, buffer) \
+ ResourceOwnerForget(owner, Int32GetDatum(buffer), &buffer_resowner_funcs)
+
/*
* Internal buffer management routines
*/
diff --git a/src/include/utils/catcache.h b/src/include/utils/catcache.h
index d81e6fabb73..4b945576655 100644
--- a/src/include/utils/catcache.h
+++ b/src/include/utils/catcache.h
@@ -225,7 +225,4 @@ extern void PrepareToInvalidateCacheTuple(Relation relation,
HeapTuple newtuple,
void (*function) (int, uint32, Oid));
-extern void PrintCatCacheLeakWarning(HeapTuple tuple);
-extern void PrintCatCacheListLeakWarning(CatCList *list);
-
#endif /* CATCACHE_H */
diff --git a/src/include/utils/plancache.h b/src/include/utils/plancache.h
index 0499635f594..5e4e1526904 100644
--- a/src/include/utils/plancache.h
+++ b/src/include/utils/plancache.h
@@ -233,4 +233,6 @@ extern bool CachedPlanIsSimplyValid(CachedPlanSource *plansource,
extern CachedExpression *GetCachedExpression(Node *expr);
extern void FreeCachedExpression(CachedExpression *cexpr);
+extern ResourceOwnerFuncs planref_resowner_funcs;
+
#endif /* PLANCACHE_H */
diff --git a/src/include/utils/resowner.h b/src/include/utils/resowner.h
index 4aff7015f92..a3612b848c2 100644
--- a/src/include/utils/resowner.h
+++ b/src/include/utils/resowner.h
@@ -50,6 +50,36 @@ typedef enum
RESOURCE_RELEASE_AFTER_LOCKS
} ResourceReleasePhase;
+/*
+ * In order to track an object, resowner.c needs a few callbacks for it.
+ * The callbacks for resources of a specific kind are encapsulated in
+ * ResourceOwnerFuncs.
+ *
+ * Note that the callback occurs post-commit or post-abort, so these callback
+ * functions can only do noncritical cleanup.
+ */
+typedef struct ResourceOwnerFuncs
+{
+ const char *name; /* name for the object kind, for debugging */
+
+ ResourceReleasePhase phase; /* when are these objects released? */
+
+ /*
+ * Release resource.
+ *
+ * NOTE: this must call ResourceOwnerForget to disassociate it with the
+ * resource owner.
+ */
+ void (*ReleaseResource) (Datum res);
+
+ /*
+ * Print a warning, when a resource has not been properly released before
+ * commit.
+ */
+ void (*PrintLeakWarning) (Datum res);
+
+} ResourceOwnerFuncs;
+
/*
* Dynamically loaded modules can get control during ResourceOwnerRelease
* by providing a callback of this form.
@@ -71,16 +101,29 @@ extern void ResourceOwnerRelease(ResourceOwner owner,
ResourceReleasePhase phase,
bool isCommit,
bool isTopLevel);
-extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
extern void ResourceOwnerDelete(ResourceOwner owner);
extern ResourceOwner ResourceOwnerGetParent(ResourceOwner owner);
extern void ResourceOwnerNewParent(ResourceOwner owner,
ResourceOwner newparent);
+
+extern void ResourceOwnerEnlarge(ResourceOwner owner);
+extern void ResourceOwnerRemember(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+extern void ResourceOwnerForget(ResourceOwner owner, Datum res, ResourceOwnerFuncs *kind);
+
extern void RegisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
extern void UnregisterResourceReleaseCallback(ResourceReleaseCallback callback,
void *arg);
+
extern void CreateAuxProcessResourceOwner(void);
extern void ReleaseAuxProcessResources(bool isCommit);
+/* special support for local lock management */
+struct LOCALLOCK;
+extern void ResourceOwnerRememberLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+extern void ResourceOwnerForgetLock(ResourceOwner owner, struct LOCALLOCK *locallock);
+
+/* special function to relase all plancache references */
+extern void ResourceOwnerReleaseAllPlanCacheRefs(ResourceOwner owner);
+
#endif /* RESOWNER_H */
diff --git a/src/include/utils/resowner_private.h b/src/include/utils/resowner_private.h
deleted file mode 100644
index d01cccc27c1..00000000000
--- a/src/include/utils/resowner_private.h
+++ /dev/null
@@ -1,112 +0,0 @@
-/*-------------------------------------------------------------------------
- *
- * resowner_private.h
- * POSTGRES resource owner private definitions.
- *
- * See utils/resowner/README for more info.
- *
- *
- * Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
- * Portions Copyright (c) 1994, Regents of the University of California
- *
- * src/include/utils/resowner_private.h
- *
- *-------------------------------------------------------------------------
- */
-#ifndef RESOWNER_PRIVATE_H
-#define RESOWNER_PRIVATE_H
-
-#include "storage/dsm.h"
-#include "storage/fd.h"
-#include "storage/lock.h"
-#include "utils/catcache.h"
-#include "utils/plancache.h"
-#include "utils/resowner.h"
-#include "utils/snapshot.h"
-
-
-/* support for buffer refcount management */
-extern void ResourceOwnerEnlargeBuffers(ResourceOwner owner);
-extern void ResourceOwnerRememberBuffer(ResourceOwner owner, Buffer buffer);
-extern void ResourceOwnerForgetBuffer(ResourceOwner owner, Buffer buffer);
-
-/* support for local lock management */
-extern void ResourceOwnerRememberLock(ResourceOwner owner, LOCALLOCK *locallock);
-extern void ResourceOwnerForgetLock(ResourceOwner owner, LOCALLOCK *locallock);
-
-/* support for catcache refcount management */
-extern void ResourceOwnerEnlargeCatCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerForgetCatCacheRef(ResourceOwner owner,
- HeapTuple tuple);
-extern void ResourceOwnerEnlargeCatCacheListRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-extern void ResourceOwnerForgetCatCacheListRef(ResourceOwner owner,
- CatCList *list);
-
-/* support for relcache refcount management */
-extern void ResourceOwnerEnlargeRelationRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberRelationRef(ResourceOwner owner,
- Relation rel);
-extern void ResourceOwnerForgetRelationRef(ResourceOwner owner,
- Relation rel);
-
-/* support for plancache refcount management */
-extern void ResourceOwnerEnlargePlanCacheRefs(ResourceOwner owner);
-extern void ResourceOwnerRememberPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-extern void ResourceOwnerForgetPlanCacheRef(ResourceOwner owner,
- CachedPlan *plan);
-
-/* support for tupledesc refcount management */
-extern void ResourceOwnerEnlargeTupleDescs(ResourceOwner owner);
-extern void ResourceOwnerRememberTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-extern void ResourceOwnerForgetTupleDesc(ResourceOwner owner,
- TupleDesc tupdesc);
-
-/* support for snapshot refcount management */
-extern void ResourceOwnerEnlargeSnapshots(ResourceOwner owner);
-extern void ResourceOwnerRememberSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-extern void ResourceOwnerForgetSnapshot(ResourceOwner owner,
- Snapshot snapshot);
-
-/* support for temporary file management */
-extern void ResourceOwnerEnlargeFiles(ResourceOwner owner);
-extern void ResourceOwnerRememberFile(ResourceOwner owner,
- File file);
-extern void ResourceOwnerForgetFile(ResourceOwner owner,
- File file);
-
-/* support for dynamic shared memory management */
-extern void ResourceOwnerEnlargeDSMs(ResourceOwner owner);
-extern void ResourceOwnerRememberDSM(ResourceOwner owner,
- dsm_segment *);
-extern void ResourceOwnerForgetDSM(ResourceOwner owner,
- dsm_segment *);
-
-/* support for JITContext management */
-extern void ResourceOwnerEnlargeJIT(ResourceOwner owner);
-extern void ResourceOwnerRememberJIT(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetJIT(ResourceOwner owner,
- Datum handle);
-
-/* support for cryptohash context management */
-extern void ResourceOwnerEnlargeCryptoHash(ResourceOwner owner);
-extern void ResourceOwnerRememberCryptoHash(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetCryptoHash(ResourceOwner owner,
- Datum handle);
-
-/* support for HMAC context management */
-extern void ResourceOwnerEnlargeHMAC(ResourceOwner owner);
-extern void ResourceOwnerRememberHMAC(ResourceOwner owner,
- Datum handle);
-extern void ResourceOwnerForgetHMAC(ResourceOwner owner,
- Datum handle);
-
-#endif /* RESOWNER_PRIVATE_H */
diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list
index 2f02cc8f422..a0ca1f20ac4 100644
--- a/src/tools/pgindent/typedefs.list
+++ b/src/tools/pgindent/typedefs.list
@@ -2320,8 +2320,10 @@ ReplicationStateOnDisk
ResTarget
ReservoirState
ReservoirStateData
-ResourceArray
+ResourceElem
ResourceOwner
+ResourceOwnerData
+ResourceOwnerFuncs
ResourceReleaseCallback
ResourceReleaseCallbackItem
ResourceReleasePhase
--
2.30.2
[text/x-patch] v12-0003-Use-a-faster-hash-function-in-resource-owners.patch (2.6K, ../../[email protected]/4-v12-0003-Use-a-faster-hash-function-in-resource-owners.patch)
download | inline diff:
From 12cc7e568ae97ffd6fc57f4024b81d5da480ab4f Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas <[email protected]>
Date: Mon, 31 Oct 2022 10:49:06 +0100
Subject: [PATCH v12 3/3] Use a faster hash function in resource owners.
This buys back some of the performance loss that we otherwise saw from the
previous commit.
Reviewed-by: Aleksander Alekseev, Michael Paquier, Hayato Kuroda, Zhihong Yu
Discussion: https://www.postgresql.org/message-id/d746cead-a1ef-7efe-fb47-933311e876a3%40iki.fi
---
src/backend/utils/resowner/resowner.c | 24 ++++++++++++++++++------
src/include/common/hashfn.h | 15 +++++++++++++++
2 files changed, 33 insertions(+), 6 deletions(-)
diff --git a/src/backend/utils/resowner/resowner.c b/src/backend/utils/resowner/resowner.c
index 3a7f08ea2bf..c640ab0d227 100644
--- a/src/backend/utils/resowner/resowner.c
+++ b/src/backend/utils/resowner/resowner.c
@@ -169,15 +169,27 @@ static void ReleaseAuxProcessResourcesCallback(int code, Datum arg);
* INTERNAL ROUTINES *
*****************************************************************************/
+/*
+ * Hash function for value+kind combination.
+ */
static inline uint32
hash_resource_elem(Datum value, ResourceOwnerFuncs *kind)
{
- Datum data[2];
-
- data[0] = value;
- data[1] = PointerGetDatum(kind);
-
- return hash_bytes((unsigned char *) &data, 2 * SIZEOF_DATUM);
+ /*
+ * Most resource kinds store a pointer in 'value', and pointers are unique
+ * all on their own. But some resources store plain integers (Files and
+ * Buffers as of this writing), so we want to incorporate the 'kind' in
+ * the hash too, otherwise those resources will collide a lot. But
+ * because there are only a few resource kinds like that - and only a few
+ * resource kinds to begin with - we don't need to work too hard to mix
+ * 'kind' into the hash. Just add it with hash_combine(), it perturbs the
+ * result enough for our purposes.
+ */
+#if SIZEOF_DATUM == 8
+ return hash_combine64(murmurhash64((uint64) value), (uint64) kind);
+#else
+ return hash_combine(murmurhash32((uint32) value), (uint32) kind);
+#endif
}
/*
diff --git a/src/include/common/hashfn.h b/src/include/common/hashfn.h
index 8d539c0a994..c44b4e5f864 100644
--- a/src/include/common/hashfn.h
+++ b/src/include/common/hashfn.h
@@ -101,4 +101,19 @@ murmurhash32(uint32 data)
return h;
}
+/* 64-bit variant */
+static inline uint64
+murmurhash64(uint64 data)
+{
+ uint64 h = data;
+
+ h ^= h >> 33;
+ h *= 0xff51afd7ed558ccd;
+ h ^= h >> 33;
+ h *= 0xc4ceb9fe1a85ec53;
+ h ^= h >> 33;
+
+ return h;
+}
+
#endif /* HASHFN_H */
--
2.30.2
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2022-10-31 12:49 Aleksander Alekseev <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Aleksander Alekseev @ 2022-10-31 12:49 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Heikki Linnakangas <[email protected]>; Julien Rouhaud <[email protected]>
Hi Heikki,
> Rebased version attached. Given that Aleksander marked this as Ready for
> Committer earlier, I'll add this to the next commitfest in that state,
> and will commit in the next few days, barring any new objections.
Thanks for resurrecting this patch.
While taking a fresh look at the code I noticed a few things.
In 0002 we have:
```
+ .name = "buffer"
...
+ .name = "File",
```
Not sure why "File" starts with an uppercase letter while "buffer"
starts with a lowercase one. This is naturally not a big deal but
could be worth changing for consistency.
In 0003:
```
+#if SIZEOF_DATUM == 8
+ return hash_combine64(murmurhash64((uint64) value), (uint64) kind);
+#else
+ return hash_combine(murmurhash32((uint32) value), (uint32) kind);
+#endif
```
Maybe it's worth using PointerGetDatum() + DatumGetInt32() /
DatumGetInt64() inline functions instead of casting Datums and
pointers directly.
These are arguably nitpicks though and shouldn't stop you from merging
the patches as is.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2022-10-31 15:51 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Aleksander Alekseev @ 2022-10-31 15:51 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Heikki Linnakangas <[email protected]>; Julien Rouhaud <[email protected]>
Hi hackers,
> > Rebased version attached. Given that Aleksander marked this as Ready for
> > Committer earlier, I'll add this to the next commitfest in that state,
> > and will commit in the next few days, barring any new objections.
>
> Thanks for resurrecting this patch.
Additionally I decided to run `make installcheck-world` on Raspberry
Pi 3. It just terminated successfully.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2022-11-01 00:15 Andres Freund <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 1 reply; 86+ messages in thread
From: Andres Freund @ 2022-11-01 00:15 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-10-31 10:51:36 +0100, Heikki Linnakangas wrote:
> These are functions where quite a lot of things happen between the
> ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
> there are no unrelated ResourceOwnerRemember() calls in the code in
> between, otherwise the entry reserved by the ResourceOwnerEnlarge() call
> might be used up by the intervening ResourceOwnerRemember() and not be
> available at the intended ResourceOwnerRemember() call anymore. The longer
> the code path between them is, the harder it is to verify that.
This seems to work towards a future where only one kind of resource can be
reserved ahead of time. That doesn't strike me as great.
> Instead of having a separate array/hash for each resource kind, use a
> single array and hash to hold all kinds of resources. This makes it
> possible to introduce new resource "kinds" without having to modify the
> ResourceOwnerData struct. In particular, this makes it possible for
> extensions to register custom resource kinds.
As a goal I like this.
However, I'm not quite sold on the implementation. Two main worries:
1) As far as I can tell, the way ResourceOwnerReleaseAll() now works seems to
assume that within a phase the reset order does not matter. I don't think
that's a good assumption. I e.g. have a patch to replace InProgressBuf with
resowner handling, and in-progress IO errors should be processed before
before pins are released.
2) There's quite a few resource types where we actually don't need an entry in
an array, because we can instead add a dlist_node to the resource -
avoiding memory overhead and making removal cheap. I have a few pending
patches that use that approach, and this doesn't really provide a path for
that anymore.
I did try out the benchmark from
https://postgr.es/m/20221029200025.w7bvlgvamjfo6z44%40awork3.anarazel.de and
the patches performed well, slightly better than my approach of allocating
some initial memory for each resarray.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2022-11-01 10:39 Heikki Linnakangas <[email protected]>
parent: Andres Freund <[email protected]>
0 siblings, 2 replies; 86+ messages in thread
From: Heikki Linnakangas @ 2022-11-01 10:39 UTC (permalink / raw)
To: Andres Freund <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
On 01/11/2022 02:15, Andres Freund wrote:
> On 2022-10-31 10:51:36 +0100, Heikki Linnakangas wrote:
>> These are functions where quite a lot of things happen between the
>> ResourceOwnerEnlarge and ResourceOwnerRemember calls. It's important that
>> there are no unrelated ResourceOwnerRemember() calls in the code in
>> between, otherwise the entry reserved by the ResourceOwnerEnlarge() call
>> might be used up by the intervening ResourceOwnerRemember() and not be
>> available at the intended ResourceOwnerRemember() call anymore. The longer
>> the code path between them is, the harder it is to verify that.
>
> This seems to work towards a future where only one kind of resource can be
> reserved ahead of time. That doesn't strike me as great.
True. It is enough for all the current callers AFAICS, though. I don't
remember wanting to reserve multiple resources at the same time.
Usually, the distance between the Enlarge and Remember calls is very
short. If it's not, it's error-prone anyway, so we should try to keep
the distance short.
While we're at it, who says that it's enough to reserve space for only
one resource of a kind either? For example, what if you want to pin two
buffers?
If we really need to support that, I propose something like this:
/*
* Reserve a slot in the resource owner.
*
* This is separate from actually inserting an entry because if we run out
* of memory, it's critical to do so *before* acquiring the resource.
*/
ResOwnerReservation *
ResourceOwnerReserve(ResourceOwner owner)
/*
* Remember that an object is owner by a ResourceOwner
*
* 'reservation' is a slot in the resource owner that was pre-reserved
* by ResOwnerReservation.
*/
void
ResOwnerRemember(ResOwnerReservaton *reservation, Datum value,
ResourceOwnerFuncs *kind)
>> Instead of having a separate array/hash for each resource kind, use a
>> single array and hash to hold all kinds of resources. This makes it
>> possible to introduce new resource "kinds" without having to modify the
>> ResourceOwnerData struct. In particular, this makes it possible for
>> extensions to register custom resource kinds.
>
> As a goal I like this.
>
> However, I'm not quite sold on the implementation. Two main worries:
>
> 1) As far as I can tell, the way ResourceOwnerReleaseAll() now works seems to
> assume that within a phase the reset order does not matter. I don't think
> that's a good assumption. I e.g. have a patch to replace InProgressBuf with
> resowner handling, and in-progress IO errors should be processed before
> before pins are released.
Hmm. Currently, you're not supposed to hold any resources at commit. You
get warnings about resource leaks if a resource owner is not empty on
ResourceOwnerReleaseAll(). On abort, does the order matter? I'm not
familiar with your InProgressBuf patch, but I guess you could handle the
in-progress IO errors in ReleaseBuffer().
If we do need to worry about release order, perhaps add a "priority" or
"phase" to each resource kind, and release them in priority order. We
already have before- and after-locks as phases, but we could generalize
that.
However, I feel that trying to enforce a particular order moves the
goalposts. If we need that, let's add it as a separate patch later.
> 2) There's quite a few resource types where we actually don't need an entry in
> an array, because we can instead add a dlist_node to the resource -
> avoiding memory overhead and making removal cheap. I have a few pending
> patches that use that approach, and this doesn't really provide a path for
> that anymore.
Is that materially better than using the array? The fast path with an
array is very fast. If it is better, perhaps we should bite the bullet
and require a dlist node and use that mechanism for all resource types?
> I did try out the benchmark from
> https://postgr.es/m/20221029200025.w7bvlgvamjfo6z44%40awork3.anarazel.de and
> the patches performed well, slightly better than my approach of allocating
> some initial memory for each resarray.
Thank you, glad to hear that!
- Heikki
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2022-11-01 12:43 Robert Haas <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Robert Haas @ 2022-11-01 12:43 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Andres Freund <[email protected]>; Julien Rouhaud <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
On Tue, Nov 1, 2022 at 6:39 AM Heikki Linnakangas <[email protected]> wrote:
> However, I feel that trying to enforce a particular order moves the
> goalposts. If we need that, let's add it as a separate patch later.
I don't really see it that way, because with the current
implementation, we do all resources of a particular type together,
before moving on to the next type. That seems like a valuable property
to preserve, and I think we should.
--
Robert Haas
EDB: http://www.enterprisedb.com
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: ResourceOwner refactoring
@ 2022-11-01 15:42 Andres Freund <[email protected]>
parent: Heikki Linnakangas <[email protected]>
1 sibling, 0 replies; 86+ messages in thread
From: Andres Freund @ 2022-11-01 15:42 UTC (permalink / raw)
To: Heikki Linnakangas <[email protected]>; +Cc: Julien Rouhaud <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers
Hi,
On 2022-11-01 12:39:39 +0200, Heikki Linnakangas wrote:
> > 1) As far as I can tell, the way ResourceOwnerReleaseAll() now works seems to
> > assume that within a phase the reset order does not matter. I don't think
> > that's a good assumption. I e.g. have a patch to replace InProgressBuf with
> > resowner handling, and in-progress IO errors should be processed before
> > before pins are released.
>
> Hmm. Currently, you're not supposed to hold any resources at commit. You get
> warnings about resource leaks if a resource owner is not empty on
> ResourceOwnerReleaseAll(). On abort, does the order matter? I'm not familiar
> with your InProgressBuf patch, but I guess you could handle the in-progress
> IO errors in ReleaseBuffer().
I was thinking about doing that as well, but it's not really trivial to know
about the in-progress IO at that time, without additional tracking (which
isn't free).
> If we do need to worry about release order, perhaps add a "priority" or
> "phase" to each resource kind, and release them in priority order. We
> already have before- and after-locks as phases, but we could generalize
> that.
>
> However, I feel that trying to enforce a particular order moves the
> goalposts. If we need that, let's add it as a separate patch later.
Like Robert, I think that the patch is moving the goalpost...
> > 2) There's quite a few resource types where we actually don't need an entry in
> > an array, because we can instead add a dlist_node to the resource -
> > avoiding memory overhead and making removal cheap. I have a few pending
> > patches that use that approach, and this doesn't really provide a path for
> > that anymore.
>
> Is that materially better than using the array?
It's safe in critical sections. I have a, not really baked but promising,
patch to make WAL writes use AIO. There's no way to know the number of
"asynchronous IOs" needed before entering the critical section.
> The fast path with an array is very fast. If it is better, perhaps we should
> bite the bullet and require a dlist node and use that mechanism for all
> resource types?
I don't think it's suitable for all - you need an exclusively owned region of
memory to embed a list in. That works nicely for some things, but not others
(e.g. buffer pins).
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 86+ messages in thread
* Changing types of block and chunk sizes in memory contexts
@ 2023-06-26 14:59 Melih Mutlu <[email protected]>
0 siblings, 1 reply; 86+ messages in thread
From: Melih Mutlu @ 2023-06-26 14:59 UTC (permalink / raw)
To: pgsql-hackers
Hi hackers,
In memory contexts, block and chunk sizes are likely to be limited by
some upper bounds. Some examples of those bounds can be
MEMORYCHUNK_MAX_BLOCKOFFSET and MEMORYCHUNK_MAX_VALUE. Both values are
only 1 less than 1GB.
This makes memory contexts to have blocks/chunks with sizes less than
1GB. Such sizes can be stored in 32-bits. Currently, "Size" type,
which is 64-bit, is used, but 32-bit integers should be enough to
store any value less than 1GB.
Attached patch is an attempt to change the types of some fields to
uint32 from Size in aset, slab and generation memory contexts.
I tried to find most of the places that needed to be changed to
uint32, but I probably missed some. I can add more places if you feel
like it.
I would appreciate any feedback.
Thanks,
--
Melih Mutlu
Microsoft
Attachments:
[application/octet-stream] 0001-Change-memory-context-fields-to-uint32.patch (6.6K, ../../CAGPVpCSOW3uJ1QJmsMR9_oE3X7fG_z4q0AoU4R_w+2RzvroPFg@mail.gmail.com/2-0001-Change-memory-context-fields-to-uint32.patch)
download | inline diff:
From a11795b222b98ea51afc6c09b4ad8859d6b4284e Mon Sep 17 00:00:00 2001
From: Melih Mutlu <[email protected]>
Date: Mon, 12 Jun 2023 08:50:28 +0300
Subject: [PATCH] Change memory context fields to uint32
Block sizes and chunk sizes can be upper bounded by
MEMORYCHUNK_MAX_BLOCKOFFSET and MEMORYCHUNK_MAX_VALUE respectively.
Values of both bounds correspond to 1 less than 1GB. This allows us
to store the sizes of any block or chunk size limited by those bounds
in 32 bits.
This patch changes types of such fields that represents block or chunk sizes
from 64-bit Size to 32-bit unsigned integers.
---
src/backend/utils/mmgr/aset.c | 16 ++++++++--------
src/backend/utils/mmgr/generation.c | 18 +++++++++---------
src/backend/utils/mmgr/slab.c | 8 ++++----
src/include/utils/memutils.h | 14 +++++++-------
4 files changed, 28 insertions(+), 28 deletions(-)
diff --git a/src/backend/utils/mmgr/aset.c b/src/backend/utils/mmgr/aset.c
index 0bbbf93672..62fb7e1f90 100644
--- a/src/backend/utils/mmgr/aset.c
+++ b/src/backend/utils/mmgr/aset.c
@@ -156,10 +156,10 @@ typedef struct AllocSetContext
AllocBlock blocks; /* head of list of blocks in this set */
MemoryChunk *freelist[ALLOCSET_NUM_FREELISTS]; /* free chunk lists */
/* Allocation parameters for this context: */
- Size initBlockSize; /* initial block size */
- Size maxBlockSize; /* maximum block size */
- Size nextBlockSize; /* next block size to allocate */
- Size allocChunkLimit; /* effective chunk size limit */
+ uint32 initBlockSize; /* initial block size */
+ uint32 maxBlockSize; /* maximum block size */
+ uint32 nextBlockSize; /* next block size to allocate */
+ uint32 allocChunkLimit; /* effective chunk size limit */
AllocBlock keeper; /* keep this block over resets */
/* freelist this context could be put in, or -1 if not a candidate: */
int freeListIndex; /* index in context_freelists[], or -1 */
@@ -340,12 +340,12 @@ AllocSetFreeIndex(Size size)
MemoryContext
AllocSetContextCreateInternal(MemoryContext parent,
const char *name,
- Size minContextSize,
- Size initBlockSize,
- Size maxBlockSize)
+ uint32 minContextSize,
+ uint32 initBlockSize,
+ uint32 maxBlockSize)
{
int freeListIndex;
- Size firstBlockSize;
+ uint32 firstBlockSize;
AllocSet set;
AllocBlock block;
diff --git a/src/backend/utils/mmgr/generation.c b/src/backend/utils/mmgr/generation.c
index 4fb8663cd6..a27d113af8 100644
--- a/src/backend/utils/mmgr/generation.c
+++ b/src/backend/utils/mmgr/generation.c
@@ -61,10 +61,10 @@ typedef struct GenerationContext
MemoryContextData header; /* Standard memory-context fields */
/* Generational context parameters */
- Size initBlockSize; /* initial block size */
- Size maxBlockSize; /* maximum block size */
- Size nextBlockSize; /* next block size to allocate */
- Size allocChunkLimit; /* effective chunk size limit */
+ uint32 initBlockSize; /* initial block size */
+ uint32 maxBlockSize; /* maximum block size */
+ uint32 nextBlockSize; /* next block size to allocate */
+ uint32 allocChunkLimit; /* effective chunk size limit */
GenerationBlock *block; /* current (most recently allocated) block, or
* NULL if we've just freed the most recent
@@ -149,12 +149,12 @@ static inline void GenerationBlockFree(GenerationContext *set,
MemoryContext
GenerationContextCreate(MemoryContext parent,
const char *name,
- Size minContextSize,
- Size initBlockSize,
- Size maxBlockSize)
+ uint32 minContextSize,
+ uint32 initBlockSize,
+ uint32 maxBlockSize)
{
- Size firstBlockSize;
- Size allocSize;
+ uint32 firstBlockSize;
+ uint32 allocSize;
GenerationContext *set;
GenerationBlock *block;
diff --git a/src/backend/utils/mmgr/slab.c b/src/backend/utils/mmgr/slab.c
index 718dd2ba03..43e01a480f 100644
--- a/src/backend/utils/mmgr/slab.c
+++ b/src/backend/utils/mmgr/slab.c
@@ -104,7 +104,7 @@ typedef struct SlabContext
{
MemoryContextData header; /* Standard memory-context fields */
/* Allocation parameters for this context: */
- Size chunkSize; /* the requested (non-aligned) chunk size */
+ uint32 chunkSize; /* the requested (non-aligned) chunk size */
Size fullChunkSize; /* chunk size with chunk header and alignment */
Size blockSize; /* the size to make each block of chunks */
int32 chunksPerBlock; /* number of chunks that fit in 1 block */
@@ -320,7 +320,7 @@ MemoryContext
SlabContextCreate(MemoryContext parent,
const char *name,
Size blockSize,
- Size chunkSize)
+ uint32 chunkSize)
{
int chunksPerBlock;
Size fullChunkSize;
@@ -352,7 +352,7 @@ SlabContextCreate(MemoryContext parent,
/* Make sure the block can store at least one chunk. */
if (chunksPerBlock == 0)
- elog(ERROR, "block size %zu for slab is too small for %zu-byte chunks",
+ elog(ERROR, "block size %zu for slab is too small for %u-byte chunks",
blockSize, chunkSize);
@@ -506,7 +506,7 @@ SlabAlloc(MemoryContext context, Size size)
/* make sure we only allow correct request size */
if (unlikely(size != slab->chunkSize))
- elog(ERROR, "unexpected alloc chunk size %zu (expected %zu)",
+ elog(ERROR, "unexpected alloc chunk size %zu (expected %u)",
size, slab->chunkSize);
/*
diff --git a/src/include/utils/memutils.h b/src/include/utils/memutils.h
index 21640d62a6..c0c3f102a5 100644
--- a/src/include/utils/memutils.h
+++ b/src/include/utils/memutils.h
@@ -111,9 +111,9 @@ extern void ProcessLogMemoryContextInterrupt(void);
/* aset.c */
extern MemoryContext AllocSetContextCreateInternal(MemoryContext parent,
const char *name,
- Size minContextSize,
- Size initBlockSize,
- Size maxBlockSize);
+ uint32 minContextSize,
+ uint32 initBlockSize,
+ uint32 maxBlockSize);
/*
* This wrapper macro exists to check for non-constant strings used as context
@@ -134,14 +134,14 @@ extern MemoryContext AllocSetContextCreateInternal(MemoryContext parent,
extern MemoryContext SlabContextCreate(MemoryContext parent,
const char *name,
Size blockSize,
- Size chunkSize);
+ uint32 chunkSize);
/* generation.c */
extern MemoryContext GenerationContextCreate(MemoryContext parent,
const char *name,
- Size minContextSize,
- Size initBlockSize,
- Size maxBlockSize);
+ uint32 minContextSize,
+ uint32 initBlockSize,
+ uint32 maxBlockSize);
/*
* Recommended default alloc parameters, suitable for "ordinary" contexts
--
2.25.1
^ permalink raw reply [nested|flat] 86+ messages in thread
* Re: Changing types of block and chunk sizes in memory contexts
@ 2023-06-28 08:13 Peter Eisentraut <[email protected]>
parent: Melih Mutlu <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Peter Eisentraut @ 2023-06-28 08:13 UTC (permalink / raw)
To: Melih Mutlu <[email protected]>; pgsql-hackers
> In memory contexts, block and chunk sizes are likely to be limited by
> some upper bounds. Some examples of those bounds can be
> MEMORYCHUNK_MAX_BLOCKOFFSET and MEMORYCHUNK_MAX_VALUE. Both values are
> only 1 less than 1GB.
> This makes memory contexts to have blocks/chunks with sizes less than
> 1GB. Such sizes can be stored in 32-bits. Currently, "Size" type,
> which is 64-bit, is used, but 32-bit integers should be enough to
> store any value less than 1GB.
size_t (= Size) is the correct type in C to store the size of an object
in memory. This is partially a self-documentation issue: If I see
size_t in a function signature, I know what is intended; if I see
uint32, I have to wonder what the intent was.
You could make an argument that using shorter types would save space for
some internal structs, but then you'd have to show some more information
where and why that would be beneficial. (But again, self-documentation:
If one were to do that, I would argue for introducing a custom type like
pg_short_size_t.)
Absent any strong performance argument, I don't see the benefit of this
change. People might well want to experiment with MEMORYCHUNK_...
settings larger than 1GB.
^ permalink raw reply [nested|flat] 86+ messages in thread
* [PATCH v4 1/7] Row pattern recognition patch for raw parser.
@ 2023-08-09 07:56 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 86+ messages in thread
From: Tatsuo Ishii @ 2023-08-09 07:56 UTC (permalink / raw)
---
src/backend/parser/gram.y | 216 +++++++++++++++++++++++++++++---
src/include/nodes/parsenodes.h | 56 +++++++++
src/include/parser/kwlist.h | 8 ++
src/include/parser/parse_node.h | 1 +
4 files changed, 267 insertions(+), 14 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 15ece871a0..62c1919538 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -251,6 +251,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
DefElem *defelt;
SortBy *sortby;
WindowDef *windef;
+ RPCommonSyntax *rpcom;
+ RPSubsetItem *rpsubset;
JoinExpr *jexpr;
IndexElem *ielem;
StatsElem *selem;
@@ -453,8 +455,12 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
drop_option_list pub_obj_list
-
-%type <node> opt_routine_body
+ row_pattern_measure_list row_pattern_definition_list
+ opt_row_pattern_subset_clause
+ row_pattern_subset_list row_pattern_subset_rhs
+ row_pattern
+%type <rpsubset> row_pattern_subset_item
+%type <node> opt_routine_body row_pattern_term
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
@@ -551,6 +557,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> relation_expr_opt_alias
%type <node> tablesample_clause opt_repeatable_clause
%type <target> target_el set_target insert_column_item
+ row_pattern_measure_item row_pattern_definition
%type <str> generic_option_name
%type <node> generic_option_arg
@@ -633,6 +640,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
+%type <rpcom> opt_row_pattern_common_syntax opt_row_pattern_skip_to
+%type <boolean> opt_row_pattern_initial_or_seek
+%type <list> opt_row_pattern_measures
%type <ival> opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
@@ -659,7 +669,6 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
json_object_constructor_null_clause_opt
json_array_constructor_null_clause_opt
-
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
* They must be listed first so that their numeric codes do not depend on
@@ -702,7 +711,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE
DATA_P DATABASE DAY_P DEALLOCATE DEC DECIMAL_P DECLARE DEFAULT DEFAULTS
- DEFERRABLE DEFERRED DEFINER DELETE_P DELIMITER DELIMITERS DEPENDS DEPTH DESC
+ DEFERRABLE DEFERRED DEFINE DEFINER DELETE_P DELIMITER DELIMITERS DEPENDS DEPTH DESC
DETACH DICTIONARY DISABLE_P DISCARD DISTINCT DO DOCUMENT_P DOMAIN_P
DOUBLE_P DROP
@@ -718,7 +727,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIAL INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -731,7 +740,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL
LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED
- MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MERGE METHOD
+ MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MEASURES MERGE METHOD
MINUTE_P MINVALUE MODE MONTH_P MOVE
NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NFC NFD NFKC NFKD NO NONE
@@ -743,8 +752,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
ORDER ORDINALITY OTHERS OUT_P OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
- PARALLEL PARAMETER PARSER PARTIAL PARTITION PASSING PASSWORD
- PLACING PLANS POLICY
+ PARALLEL PARAMETER PARSER PARTIAL PARTITION PASSING PASSWORD PAST
+ PATTERN PERMUTE PLACING PLANS POLICY
POSITION PRECEDING PRECISION PRESERVE PREPARE PREPARED PRIMARY
PRIOR PRIVILEGES PROCEDURAL PROCEDURE PROCEDURES PROGRAM PUBLICATION
@@ -755,12 +764,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SEEK SELECT
SEQUENCE SEQUENCES
+
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
START STATEMENT STATISTICS STDIN STDOUT STORAGE STORED STRICT_P STRIP_P
- SUBSCRIPTION SUBSTRING SUPPORT SYMMETRIC SYSID SYSTEM_P SYSTEM_USER
+ SUBSCRIPTION SUBSET SUBSTRING SUPPORT SYMMETRIC SYSID SYSTEM_P SYSTEM_USER
TABLE TABLES TABLESAMPLE TABLESPACE TEMP TEMPLATE TEMPORARY TEXT_P THEN
TIES TIME TIMESTAMP TO TRAILING TRANSACTION TRANSFORM
@@ -853,6 +863,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
*/
%nonassoc UNBOUNDED /* ideally would have same precedence as IDENT */
%nonassoc IDENT PARTITION RANGE ROWS GROUPS PRECEDING FOLLOWING CUBE ROLLUP
+%nonassoc MEASURES AFTER INITIAL SEEK PATTERN
%left Op OPERATOR /* multi-character ops and user-defined operators */
%left '+' '-'
%left '*' '/' '%'
@@ -15832,7 +15843,8 @@ over_clause: OVER window_specification
;
window_specification: '(' opt_existing_window_name opt_partition_clause
- opt_sort_clause opt_frame_clause ')'
+ opt_sort_clause opt_row_pattern_measures opt_frame_clause
+ opt_row_pattern_common_syntax ')'
{
WindowDef *n = makeNode(WindowDef);
@@ -15840,10 +15852,12 @@ window_specification: '(' opt_existing_window_name opt_partition_clause
n->refname = $2;
n->partitionClause = $3;
n->orderClause = $4;
+ n->rowPatternMeasures = $5;
/* copy relevant fields of opt_frame_clause */
- n->frameOptions = $5->frameOptions;
- n->startOffset = $5->startOffset;
- n->endOffset = $5->endOffset;
+ n->frameOptions = $6->frameOptions;
+ n->startOffset = $6->startOffset;
+ n->endOffset = $6->endOffset;
+ n->rpCommonSyntax = $7;
n->location = @1;
$$ = n;
}
@@ -15867,6 +15881,31 @@ opt_partition_clause: PARTITION BY expr_list { $$ = $3; }
| /*EMPTY*/ { $$ = NIL; }
;
+/*
+ * ROW PATTERN MEASURES
+ */
+opt_row_pattern_measures: MEASURES row_pattern_measure_list { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_measure_list:
+ row_pattern_measure_item
+ { $$ = list_make1($1); }
+ | row_pattern_measure_list ',' row_pattern_measure_item
+ { $$ = lappend($1, $3); }
+ ;
+
+row_pattern_measure_item:
+ a_expr AS ColLabel
+ {
+ $$ = makeNode(ResTarget);
+ $$->name = $3;
+ $$->indirection = NIL;
+ $$->val = (Node *) $1;
+ $$->location = @1;
+ }
+ ;
+
/*
* For frame clauses, we return a WindowDef, but only some fields are used:
* frameOptions, startOffset, and endOffset.
@@ -16026,6 +16065,139 @@ opt_window_exclusion_clause:
| /*EMPTY*/ { $$ = 0; }
;
+opt_row_pattern_common_syntax:
+opt_row_pattern_skip_to opt_row_pattern_initial_or_seek
+ PATTERN '(' row_pattern ')'
+ opt_row_pattern_subset_clause
+ DEFINE row_pattern_definition_list
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = $1->rpSkipTo;
+ n->rpSkipVariable = $1->rpSkipVariable;
+ n->initial = $2;
+ n->rpPatterns = $5;
+ n->rpSubsetClause = $7;
+ n->rpDefs = $9;
+ $$ = n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+opt_row_pattern_skip_to:
+ AFTER MATCH SKIP TO NEXT ROW
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_NEXT_ROW;
+ n->rpSkipVariable = NULL;
+ $$ = n;
+ }
+ | AFTER MATCH SKIP PAST LAST_P ROW
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_PAST_LAST_ROW;
+ n->rpSkipVariable = NULL;
+ $$ = n;
+ }
+/*
+ | AFTER MATCH SKIP TO FIRST_P ColId %prec FIRST_P
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_FIRST_VARIABLE;
+ n->rpSkipVariable = $6;
+ $$ = n;
+ }
+ | AFTER MATCH SKIP TO LAST_P ColId %prec LAST_P
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_LAST_VARIABLE;
+ n->rpSkipVariable = $6;
+ $$ = n;
+ }
+ * Shift/reduce
+ | AFTER MATCH SKIP TO ColId
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_VARIABLE;
+ n->rpSkipVariable = $5;
+ $$ = n;
+ }
+*/
+ | /*EMPTY*/
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ /* temporary set default to ST_NEXT_ROW */
+ n->rpSkipTo = ST_PAST_LAST_ROW;
+ n->rpSkipVariable = NULL;
+ $$ = n;
+ }
+ ;
+
+opt_row_pattern_initial_or_seek:
+ INITIAL { $$ = true; }
+ | SEEK
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("SEEK is not supported"),
+ errhint("Use INITIAL."),
+ parser_errposition(@1)));
+ }
+ | /*EMPTY*/ { $$ = true; }
+ ;
+
+row_pattern:
+ row_pattern_term { $$ = list_make1($1); }
+ | row_pattern row_pattern_term { $$ = lappend($1, $2); }
+ ;
+
+row_pattern_term:
+ ColId { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "", (Node *)makeString($1), NULL, @1); }
+ | ColId '*' { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "*", (Node *)makeString($1), NULL, @1); }
+ | ColId '+' { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "+", (Node *)makeString($1), NULL, @1); }
+ | ColId '?' { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "?", (Node *)makeString($1), NULL, @1); }
+ ;
+
+opt_row_pattern_subset_clause:
+ SUBSET row_pattern_subset_list { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_subset_list:
+ row_pattern_subset_item { $$ = list_make1($1); }
+ | row_pattern_subset_list ',' row_pattern_subset_item { $$ = lappend($1, $3); }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_subset_item: ColId '=' '(' row_pattern_subset_rhs ')'
+ {
+ RPSubsetItem *n = makeNode(RPSubsetItem);
+ n->name = $1;
+ n->rhsVariable = $4;
+ $$ = n;
+ }
+ ;
+
+row_pattern_subset_rhs:
+ ColId { $$ = list_make1(makeStringConst($1, @1)); }
+ | row_pattern_subset_rhs ',' ColId { $$ = lappend($1, makeStringConst($3, @1)); }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_definition_list:
+ row_pattern_definition { $$ = list_make1($1); }
+ | row_pattern_definition_list ',' row_pattern_definition { $$ = lappend($1, $3); }
+ ;
+
+row_pattern_definition:
+ ColId AS a_expr
+ {
+ $$ = makeNode(ResTarget);
+ $$->name = $1;
+ $$->indirection = NIL;
+ $$->val = (Node *) $3;
+ $$->location = @1;
+ }
+ ;
/*
* Supporting nonterminals for expressions.
@@ -17121,6 +17293,7 @@ unreserved_keyword:
| INDEXES
| INHERIT
| INHERITS
+ | INITIAL
| INLINE_P
| INPUT_P
| INSENSITIVE
@@ -17148,6 +17321,7 @@ unreserved_keyword:
| MATCHED
| MATERIALIZED
| MAXVALUE
+ | MEASURES
| MERGE
| METHOD
| MINUTE_P
@@ -17190,6 +17364,9 @@ unreserved_keyword:
| PARTITION
| PASSING
| PASSWORD
+ | PAST
+ | PATTERN
+ | PERMUTE
| PLANS
| POLICY
| PRECEDING
@@ -17240,6 +17417,7 @@ unreserved_keyword:
| SEARCH
| SECOND_P
| SECURITY
+ | SEEK
| SEQUENCE
| SEQUENCES
| SERIALIZABLE
@@ -17265,6 +17443,7 @@ unreserved_keyword:
| STRICT_P
| STRIP_P
| SUBSCRIPTION
+ | SUBSET
| SUPPORT
| SYSID
| SYSTEM_P
@@ -17452,6 +17631,7 @@ reserved_keyword:
| CURRENT_USER
| DEFAULT
| DEFERRABLE
+ | DEFINE
| DESC
| DISTINCT
| DO
@@ -17614,6 +17794,7 @@ bare_label_keyword:
| DEFAULTS
| DEFERRABLE
| DEFERRED
+ | DEFINE
| DEFINER
| DELETE_P
| DELIMITER
@@ -17689,6 +17870,7 @@ bare_label_keyword:
| INDEXES
| INHERIT
| INHERITS
+ | INITIAL
| INITIALLY
| INLINE_P
| INNER_P
@@ -17738,6 +17920,7 @@ bare_label_keyword:
| MATCHED
| MATERIALIZED
| MAXVALUE
+ | MEASURES
| MERGE
| METHOD
| MINVALUE
@@ -17791,6 +17974,9 @@ bare_label_keyword:
| PARTITION
| PASSING
| PASSWORD
+ | PAST
+ | PATTERN
+ | PERMUTE
| PLACING
| PLANS
| POLICY
@@ -17847,6 +18033,7 @@ bare_label_keyword:
| SCROLL
| SEARCH
| SECURITY
+ | SEEK
| SELECT
| SEQUENCE
| SEQUENCES
@@ -17878,6 +18065,7 @@ bare_label_keyword:
| STRICT_P
| STRIP_P
| SUBSCRIPTION
+ | SUBSET
| SUBSTRING
| SUPPORT
| SYMMETRIC
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index fe003ded50..c78bd849f9 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -547,6 +547,44 @@ typedef struct SortBy
int location; /* operator location, or -1 if none/unknown */
} SortBy;
+/*
+ * AFTER MATCH row pattern skip to types in row pattern common syntax
+ */
+typedef enum RPSkipTo
+{
+ ST_NONE, /* AFTER MATCH omitted */
+ ST_NEXT_ROW, /* SKIP TO NEXT ROW */
+ ST_PAST_LAST_ROW, /* SKIP TO PAST LAST ROW */
+ ST_FIRST_VARIABLE, /* SKIP TO FIRST variable name */
+ ST_LAST_VARIABLE, /* SKIP TO LAST variable name */
+ ST_VARIABLE /* SKIP TO variable name */
+} RPSkipTo;
+
+/*
+ * Row Pattern SUBSET clause item
+ */
+typedef struct RPSubsetItem
+{
+ NodeTag type;
+ char *name; /* Row Pattern SUBSET clause variable name */
+ List *rhsVariable; /* Row Pattern SUBSET rhs variables (list of char *string) */
+} RPSubsetItem;
+
+/*
+ * RowPatternCommonSyntax - raw representation of row pattern common syntax
+ *
+ */
+typedef struct RPCommonSyntax
+{
+ NodeTag type;
+ RPSkipTo rpSkipTo; /* Row Pattern AFTER MATCH SKIP type */
+ char *rpSkipVariable; /* Row Pattern Skip To variable name, if any */
+ bool initial; /* true if <row pattern initial or seek> is initial */
+ List *rpPatterns; /* PATTERN variables (list of A_Expr) */
+ List *rpSubsetClause; /* row pattern subset clause (list of RPSubsetItem), if any */
+ List *rpDefs; /* row pattern definitions clause (list of ResTarget) */
+} RPCommonSyntax;
+
/*
* WindowDef - raw representation of WINDOW and OVER clauses
*
@@ -562,6 +600,8 @@ typedef struct WindowDef
char *refname; /* referenced window name, if any */
List *partitionClause; /* PARTITION BY expression list */
List *orderClause; /* ORDER BY (list of SortBy) */
+ List *rowPatternMeasures; /* row pattern measures (list of ResTarget) */
+ RPCommonSyntax *rpCommonSyntax; /* row pattern common syntax */
int frameOptions; /* frame_clause options, see below */
Node *startOffset; /* expression for starting bound, if any */
Node *endOffset; /* expression for ending bound, if any */
@@ -1483,6 +1523,11 @@ typedef struct GroupingSet
* the orderClause might or might not be copied (see copiedOrder); the framing
* options are never copied, per spec.
*
+ * "defineClause" is Row Pattern Recognition DEFINE clause (list of
+ * TargetEntry). TargetEntry.resname represents row pattern definition
+ * variable name. "patternVariable" and "patternRegexp" represents PATTERN
+ * clause.
+ *
* The information relevant for the query jumbling is the partition clause
* type and its bounds.
*/
@@ -1514,6 +1559,17 @@ typedef struct WindowClause
Index winref; /* ID referenced by window functions */
/* did we copy orderClause from refname? */
bool copiedOrder pg_node_attr(query_jumble_ignore);
+ /* Row Pattern AFTER MACH SKIP clause */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ bool initial; /* true if <row pattern initial or seek> is initial */
+ /* Row Pattern DEFINE clause (list of TargetEntry) */
+ List *defineClause;
+ /* Row Pattern DEFINE variable initial names (list of String) */
+ List *defineInitial;
+ /* Row Pattern PATTERN variable name (list of String) */
+ List *patternVariable;
+ /* Row Pattern PATTERN regular expression quantifier ('+' or ''. list of String) */
+ List *patternRegexp;
} WindowClause;
/*
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa4b..431e0625f1 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -128,6 +128,7 @@ PG_KEYWORD("default", DEFAULT, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("defaults", DEFAULTS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("deferrable", DEFERRABLE, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("deferred", DEFERRED, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("define", DEFINE, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("definer", DEFINER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("delete", DELETE_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("delimiter", DELIMITER, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -212,6 +213,7 @@ PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherits", INHERITS, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("initial", INITIAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("initially", INITIALLY, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inline", INLINE_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inner", INNER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
@@ -265,6 +267,7 @@ PG_KEYWORD("match", MATCH, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("matched", MATCHED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("materialized", MATERIALIZED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("maxvalue", MAXVALUE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("measures", MEASURES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("merge", MERGE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("method", METHOD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("minute", MINUTE_P, UNRESERVED_KEYWORD, AS_LABEL)
@@ -326,6 +329,9 @@ PG_KEYWORD("partial", PARTIAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("partition", PARTITION, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("passing", PASSING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("password", PASSWORD, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("past", PAST, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("pattern", PATTERN, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("permute", PERMUTE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("placing", PLACING, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("plans", PLANS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("policy", POLICY, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -385,6 +391,7 @@ PG_KEYWORD("scroll", SCROLL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("search", SEARCH, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("second", SECOND_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("security", SECURITY, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("seek", SEEK, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("select", SELECT, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("sequence", SEQUENCE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("sequences", SEQUENCES, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -416,6 +423,7 @@ PG_KEYWORD("stored", STORED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("strict", STRICT_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("strip", STRIP_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("subscription", SUBSCRIPTION, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("subset", SUBSET, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("substring", SUBSTRING, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("support", SUPPORT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("symmetric", SYMMETRIC, RESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index f589112d5e..6640090910 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -51,6 +51,7 @@ typedef enum ParseExprKind
EXPR_KIND_WINDOW_FRAME_RANGE, /* window frame clause with RANGE */
EXPR_KIND_WINDOW_FRAME_ROWS, /* window frame clause with ROWS */
EXPR_KIND_WINDOW_FRAME_GROUPS, /* window frame clause with GROUPS */
+ EXPR_KIND_RPR_DEFINE, /* DEFINE */
EXPR_KIND_SELECT_TARGET, /* SELECT target list item */
EXPR_KIND_INSERT_TARGET, /* INSERT target list item */
EXPR_KIND_UPDATE_SOURCE, /* UPDATE assignment source item */
--
2.25.1
----Next_Part(Wed_Aug__9_17_41_12_2023_134)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v4-0002-Row-pattern-recognition-patch-parse-analysis.patch"
^ permalink raw reply [nested|flat] 86+ messages in thread
end of thread, other threads:[~2023-08-09 07:56 UTC | newest]
Thread overview: 86+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-12-21 12:20 [PATCH v20 2/3] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2017-12-21 12:20 [PATCH v19 2/3] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2017-12-21 12:20 [PATCH 1/6] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2017-12-21 12:20 [PATCH 1/6] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2017-12-21 12:20 [PATCH v20 2/3] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2017-12-21 12:20 [PATCH 1/6] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2017-12-21 12:20 [PATCH 1/6] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2018-12-19 03:43 [PATCH v21 1/2] Add WAL relief vent for replication slots Kyotaro Horiguchi <[email protected]>
2019-12-20 01:05 [PATCH v30 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v27 1/9] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v38 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v28 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v26 01/10] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v29 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v32 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v39 1/8] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v29 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v39 1/8] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v38 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v39 1/8] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v25 01/15] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v23 01/15] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v30 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v27 1/9] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v37 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v24 01/15] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v38 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v29 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v37 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v37 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v30 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v24 01/15] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2019-12-20 01:05 [PATCH v31 01/11] Add a syntax to create Incrementally Maintainable Materialized Views Yugo Nagata <[email protected]>
2020-11-17 14:21 ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2020-11-18 08:06 ` Re: ResourceOwner refactoring Michael Paquier <[email protected]>
2020-11-18 08:50 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2020-11-19 02:16 ` Re: ResourceOwner refactoring Michael Paquier <[email protected]>
2020-11-19 09:27 ` Re: ResourceOwner refactoring Julien Rouhaud <[email protected]>
2020-11-26 08:52 ` Re: ResourceOwner refactoring Kyotaro Horiguchi <[email protected]>
2020-12-16 15:46 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-13 01:55 ` RE: ResourceOwner refactoring [email protected] <[email protected]>
2021-01-13 07:18 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-13 07:22 ` Re: ResourceOwner refactoring Michael Paquier <[email protected]>
2021-01-13 07:39 ` RE: ResourceOwner refactoring [email protected] <[email protected]>
2021-01-14 10:15 ` RE: ResourceOwner refactoring [email protected] <[email protected]>
2021-01-14 13:34 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-15 09:10 ` RE: ResourceOwner refactoring [email protected] <[email protected]>
2021-01-18 07:49 ` RE: ResourceOwner refactoring [email protected] <[email protected]>
2021-01-18 12:22 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-18 14:34 ` Re: ResourceOwner refactoring Alvaro Herrera <[email protected]>
2021-01-18 15:19 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-18 16:11 ` Re: ResourceOwner refactoring Robert Haas <[email protected]>
2021-01-18 16:11 ` Re: ResourceOwner refactoring Robert Haas <[email protected]>
2021-01-19 09:09 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-20 22:11 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-21 02:22 ` RE: ResourceOwner refactoring [email protected] <[email protected]>
2021-01-21 04:14 ` Re: ResourceOwner refactoring Michael Paquier <[email protected]>
2021-01-21 10:14 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-25 17:14 ` Re: ResourceOwner refactoring Robert Haas <[email protected]>
2021-03-08 16:47 ` Re: ResourceOwner refactoring Ibrar Ahmed <[email protected]>
2021-03-09 12:40 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-07-14 12:14 ` Re: ResourceOwner refactoring vignesh C <[email protected]>
2021-07-14 14:07 ` Re: ResourceOwner refactoring Alvaro Herrera <[email protected]>
2021-07-14 14:40 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-07-14 15:18 ` Re: ResourceOwner refactoring Zhihong Yu <[email protected]>
2021-07-14 15:26 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-07-14 15:41 ` Re: ResourceOwner refactoring Zhihong Yu <[email protected]>
2021-09-08 10:19 ` Re: ResourceOwner refactoring Aleksander Alekseev <[email protected]>
2021-10-01 07:18 ` Re: ResourceOwner refactoring Michael Paquier <[email protected]>
2021-10-18 11:17 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-11-23 15:37 ` Re: ResourceOwner refactoring Aleksander Alekseev <[email protected]>
2021-11-26 12:40 ` Re: ResourceOwner refactoring Aleksander Alekseev <[email protected]>
2022-10-31 09:51 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2022-10-31 12:49 ` Re: ResourceOwner refactoring Aleksander Alekseev <[email protected]>
2022-10-31 15:51 ` Re: ResourceOwner refactoring Aleksander Alekseev <[email protected]>
2022-11-01 00:15 ` Re: ResourceOwner refactoring Andres Freund <[email protected]>
2022-11-01 10:39 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2022-11-01 12:43 ` Re: ResourceOwner refactoring Robert Haas <[email protected]>
2022-11-01 15:42 ` Re: ResourceOwner refactoring Andres Freund <[email protected]>
2021-01-19 14:45 ` Re: ResourceOwner refactoring Heikki Linnakangas <[email protected]>
2021-01-19 06:48 ` Re: ResourceOwner refactoring Michael Paquier <[email protected]>
2020-11-19 10:40 ` Re: ResourceOwner refactoring Craig Ringer <[email protected]>
2020-11-26 07:29 ` Re: ResourceOwner refactoring Michael Paquier <[email protected]>
2023-06-26 14:59 Changing types of block and chunk sizes in memory contexts Melih Mutlu <[email protected]>
2023-06-28 08:13 ` Re: Changing types of block and chunk sizes in memory contexts Peter Eisentraut <[email protected]>
2023-08-09 07:56 [PATCH v4 1/7] Row pattern recognition patch for raw parser. Tatsuo Ishii <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox