public inbox for [email protected]
help / color / mirror / Atom feed[PATCH] Refactor the fsync mechanism to support future SMGR implementations.
4+ messages / 4 participants
[nested] [flat]
* [PATCH] Refactor the fsync mechanism to support future SMGR implementations.
@ 2019-02-27 18:58 Shawn Debnath <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Shawn Debnath @ 2019-02-27 18:58 UTC (permalink / raw)
In anticipation of proposed block storage managers alongside md.c that
map bufmgr.c blocks to files optimised for different usage patterns:
1. Move the system for requesting and processing fsyncs out of md.c
into storage/sync/sync.c with definitions in include/storage/sync.h.
ProcessSyncRequests() is now responsible for processing the sync
requests during checkpoint.
2. Removed the need for specific storage managers to implement pre and
post checkpoint callbacks. These are now executed by the sync mechanism.
3. We now embed the fork number and the segment number as part of the
hash key for the pending ops table. This eliminates the bitmapset based
segment tracking for each relfilenode during fsync as not all storage
managers may map their segments from zero.
4. Each sync request now must include a type: sync, forget, forget
hierarchy, or unlink, and the owner who will be responsible for
generating paths or matching forget requests.
5. For cancelling relation sync requests, we now must send a forget
request for each fork and segment in the relation.
6. We do not rely on smgr to provide the file descriptor we use to
issue fsync. Instead, we generate the full path based on the FileTag
in the sync request and use PathNameOpenFile to get the file descriptor.
Author: Shawn Debnath, Thomas Munro
Reviewed-by:
Discussion:
https://postgr.es/m/CAEepm=2gTANm=e3ARnJT=n0h8hf88wqmaZxk0JYkxw+b21fNrw@mail.gmail.com
---
src/backend/access/transam/xlog.c | 7 +-
src/backend/commands/dbcommands.c | 6 +-
src/backend/postmaster/checkpointer.c | 52 +--
src/backend/storage/Makefile | 2 +-
src/backend/storage/buffer/bufmgr.c | 2 +-
src/backend/storage/smgr/md.c | 791 ++++------------------------------
src/backend/storage/smgr/smgr.c | 54 ---
src/backend/storage/sync/Makefile | 17 +
src/backend/storage/sync/sync.c | 590 +++++++++++++++++++++++++
src/backend/utils/init/postinit.c | 2 +
src/include/postmaster/bgwriter.h | 8 +-
src/include/storage/fd.h | 12 +
src/include/storage/segment.h | 28 ++
src/include/storage/smgr.h | 17 +-
src/include/storage/sync.h | 108 +++++
15 files changed, 890 insertions(+), 806 deletions(-)
create mode 100644 src/backend/storage/sync/Makefile
create mode 100644 src/backend/storage/sync/sync.c
create mode 100644 src/include/storage/segment.h
create mode 100644 src/include/storage/sync.h
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index ecd12fc53a..a04f993e3e 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -65,6 +65,7 @@
#include "storage/reinit.h"
#include "storage/smgr.h"
#include "storage/spin.h"
+#include "storage/sync.h"
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/memutils.h"
@@ -6986,7 +6987,7 @@ StartupXLOG(void)
if (ArchiveRecoveryRequested && IsUnderPostmaster)
{
PublishStartupProcessInformation();
- SetForwardFsyncRequests();
+ EnableSyncRequestForwarding();
SendPostmasterSignal(PMSIGNAL_RECOVERY_STARTED);
bgwriterLaunched = true;
}
@@ -8616,7 +8617,7 @@ CreateCheckPoint(int flags)
* the REDO pointer. Note that smgr must not do anything that'd have to
* be undone if we decide no checkpoint is needed.
*/
- smgrpreckpt();
+ PreCheckpoint();
/* Begin filling in the checkpoint WAL record */
MemSet(&checkPoint, 0, sizeof(checkPoint));
@@ -8912,7 +8913,7 @@ CreateCheckPoint(int flags)
/*
* Let smgr do post-checkpoint cleanup (eg, deleting old files).
*/
- smgrpostckpt();
+ PostCheckpoint();
/*
* Update the average distance between checkpoints if the prior checkpoint
diff --git a/src/backend/commands/dbcommands.c b/src/backend/commands/dbcommands.c
index d207cd899f..825e44de26 100644
--- a/src/backend/commands/dbcommands.c
+++ b/src/backend/commands/dbcommands.c
@@ -940,11 +940,11 @@ dropdb(const char *dbname, bool missing_ok)
* worse, it will delete files that belong to a newly created database
* with the same OID.
*/
- ForgetDatabaseFsyncRequests(db_id);
+ ForgetDatabaseSyncRequests(db_id);
/*
* Force a checkpoint to make sure the checkpointer has received the
- * message sent by ForgetDatabaseFsyncRequests. On Windows, this also
+ * message sent by ForgetDatabaseSyncRequests. On Windows, this also
* ensures that background procs don't hold any open files, which would
* cause rmdir() to fail.
*/
@@ -2149,7 +2149,7 @@ dbase_redo(XLogReaderState *record)
DropDatabaseBuffers(xlrec->db_id);
/* Also, clean out any fsync requests that might be pending in md.c */
- ForgetDatabaseFsyncRequests(xlrec->db_id);
+ ForgetDatabaseSyncRequests(xlrec->db_id);
/* Clean out the xlog relcache too */
XLogDropDatabase(xlrec->db_id);
diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c
index fe96c41359..6fb22246a6 100644
--- a/src/backend/postmaster/checkpointer.c
+++ b/src/backend/postmaster/checkpointer.c
@@ -106,14 +106,6 @@
* the requests fields are protected by CheckpointerCommLock.
*----------
*/
-typedef struct
-{
- RelFileNode rnode;
- ForkNumber forknum;
- BlockNumber segno; /* see md.c for special values */
- /* might add a real request-type field later; not needed yet */
-} CheckpointerRequest;
-
typedef struct
{
pid_t checkpointer_pid; /* PID (0 if not started) */
@@ -131,7 +123,7 @@ typedef struct
int num_requests; /* current # of requests */
int max_requests; /* allocated array size */
- CheckpointerRequest requests[FLEXIBLE_ARRAY_MEMBER];
+ SyncRequest requests[FLEXIBLE_ARRAY_MEMBER];
} CheckpointerShmemStruct;
static CheckpointerShmemStruct *CheckpointerShmem;
@@ -347,7 +339,7 @@ CheckpointerMain(void)
/*
* Process any requests or signals received recently.
*/
- AbsorbFsyncRequests();
+ AbsorbSyncRequests();
if (got_SIGHUP)
{
@@ -676,7 +668,7 @@ CheckpointWriteDelay(int flags, double progress)
UpdateSharedMemoryConfig();
}
- AbsorbFsyncRequests();
+ AbsorbSyncRequests();
absorb_counter = WRITES_PER_ABSORB;
CheckArchiveTimeout();
@@ -701,7 +693,7 @@ CheckpointWriteDelay(int flags, double progress)
* operations even when we don't sleep, to prevent overflow of the
* fsync request queue.
*/
- AbsorbFsyncRequests();
+ AbsorbSyncRequests();
absorb_counter = WRITES_PER_ABSORB;
}
}
@@ -885,7 +877,7 @@ CheckpointerShmemSize(void)
* NBuffers. This may prove too large or small ...
*/
size = offsetof(CheckpointerShmemStruct, requests);
- size = add_size(size, mul_size(NBuffers, sizeof(CheckpointerRequest)));
+ size = add_size(size, mul_size(NBuffers, sizeof(SyncRequest)));
return size;
}
@@ -1063,7 +1055,7 @@ RequestCheckpoint(int flags)
}
/*
- * ForwardFsyncRequest
+ * ForwardSyncRequest
* Forward a file-fsync request from a backend to the checkpointer
*
* Whenever a backend is compelled to write directly to a relation
@@ -1092,9 +1084,9 @@ RequestCheckpoint(int flags)
* let the backend know by returning false.
*/
bool
-ForwardFsyncRequest(RelFileNode rnode, ForkNumber forknum, BlockNumber segno)
+ForwardSyncRequest(FileTag ftag, SyncRequestType type, SyncRequestOwner owner)
{
- CheckpointerRequest *request;
+ SyncRequest *request;
bool too_full;
if (!IsUnderPostmaster)
@@ -1130,9 +1122,9 @@ ForwardFsyncRequest(RelFileNode rnode, ForkNumber forknum, BlockNumber segno)
/* OK, insert request */
request = &CheckpointerShmem->requests[CheckpointerShmem->num_requests++];
- request->rnode = rnode;
- request->forknum = forknum;
- request->segno = segno;
+ request->owner = owner;
+ request->type = type;
+ request->ftag = ftag;
/* If queue is more than half full, nudge the checkpointer to empty it */
too_full = (CheckpointerShmem->num_requests >=
@@ -1168,7 +1160,7 @@ CompactCheckpointerRequestQueue(void)
{
struct CheckpointerSlotMapping
{
- CheckpointerRequest request;
+ SyncRequest request;
int slot;
};
@@ -1187,7 +1179,7 @@ CompactCheckpointerRequestQueue(void)
/* Initialize temporary hash table */
MemSet(&ctl, 0, sizeof(ctl));
- ctl.keysize = sizeof(CheckpointerRequest);
+ ctl.keysize = sizeof(SyncRequest);
ctl.entrysize = sizeof(struct CheckpointerSlotMapping);
ctl.hcxt = CurrentMemoryContext;
@@ -1211,7 +1203,7 @@ CompactCheckpointerRequestQueue(void)
*/
for (n = 0; n < CheckpointerShmem->num_requests; n++)
{
- CheckpointerRequest *request;
+ SyncRequest *request;
struct CheckpointerSlotMapping *slotmap;
bool found;
@@ -1263,8 +1255,8 @@ CompactCheckpointerRequestQueue(void)
}
/*
- * AbsorbFsyncRequests
- * Retrieve queued fsync requests and pass them to local smgr.
+ * AbsorbSyncRequests
+ * Retrieve queued sync requests and pass them to sync mechanism.
*
* This is exported because it must be called during CreateCheckPoint;
* we have to be sure we have accepted all pending requests just before
@@ -1272,10 +1264,10 @@ CompactCheckpointerRequestQueue(void)
* non-checkpointer processes, do nothing if not checkpointer.
*/
void
-AbsorbFsyncRequests(void)
+AbsorbSyncRequests(void)
{
- CheckpointerRequest *requests = NULL;
- CheckpointerRequest *request;
+ SyncRequest *requests = NULL;
+ SyncRequest *request;
int n;
if (!AmCheckpointerProcess())
@@ -1303,8 +1295,8 @@ AbsorbFsyncRequests(void)
n = CheckpointerShmem->num_requests;
if (n > 0)
{
- requests = (CheckpointerRequest *) palloc(n * sizeof(CheckpointerRequest));
- memcpy(requests, CheckpointerShmem->requests, n * sizeof(CheckpointerRequest));
+ requests = (SyncRequest *) palloc(n * sizeof(SyncRequest));
+ memcpy(requests, CheckpointerShmem->requests, n * sizeof(SyncRequest));
}
START_CRIT_SECTION();
@@ -1314,7 +1306,7 @@ AbsorbFsyncRequests(void)
LWLockRelease(CheckpointerCommLock);
for (request = requests; n > 0; request++, n--)
- RememberFsyncRequest(request->rnode, request->forknum, request->segno);
+ RememberSyncRequest(request->ftag, request->type, request->owner);
END_CRIT_SECTION();
diff --git a/src/backend/storage/Makefile b/src/backend/storage/Makefile
index bd2d272c6e..8376cdfca2 100644
--- a/src/backend/storage/Makefile
+++ b/src/backend/storage/Makefile
@@ -8,6 +8,6 @@ subdir = src/backend/storage
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
-SUBDIRS = buffer file freespace ipc large_object lmgr page smgr
+SUBDIRS = buffer file freespace ipc large_object lmgr page smgr sync
include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 273e2f385f..887023fc8a 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -2584,7 +2584,7 @@ CheckPointBuffers(int flags)
BufferSync(flags);
CheckpointStats.ckpt_sync_t = GetCurrentTimestamp();
TRACE_POSTGRESQL_BUFFER_CHECKPOINT_SYNC_START();
- smgrsync();
+ ProcessSyncRequests();
CheckpointStats.ckpt_sync_end_t = GetCurrentTimestamp();
TRACE_POSTGRESQL_BUFFER_CHECKPOINT_DONE();
}
diff --git a/src/backend/storage/smgr/md.c b/src/backend/storage/smgr/md.c
index 2aba2dfe91..c5c17dc69d 100644
--- a/src/backend/storage/smgr/md.c
+++ b/src/backend/storage/smgr/md.c
@@ -29,45 +29,17 @@
#include "access/xlogutils.h"
#include "access/xlog.h"
#include "pgstat.h"
-#include "portability/instr_time.h"
#include "postmaster/bgwriter.h"
#include "storage/fd.h"
#include "storage/bufmgr.h"
#include "storage/relfilenode.h"
+#include "storage/segment.h"
#include "storage/smgr.h"
+#include "storage/sync.h"
#include "utils/hsearch.h"
#include "utils/memutils.h"
#include "pg_trace.h"
-
-/* intervals for calling AbsorbFsyncRequests in mdsync and mdpostckpt */
-#define FSYNCS_PER_ABSORB 10
-#define UNLINKS_PER_ABSORB 10
-
-/*
- * Special values for the segno arg to RememberFsyncRequest.
- *
- * Note that CompactCheckpointerRequestQueue assumes that it's OK to remove an
- * fsync request from the queue if an identical, subsequent request is found.
- * See comments there before making changes here.
- */
-#define FORGET_RELATION_FSYNC (InvalidBlockNumber)
-#define FORGET_DATABASE_FSYNC (InvalidBlockNumber-1)
-#define UNLINK_RELATION_REQUEST (InvalidBlockNumber-2)
-
-/*
- * On Windows, we have to interpret EACCES as possibly meaning the same as
- * ENOENT, because if a file is unlinked-but-not-yet-gone on that platform,
- * that's what you get. Ugh. This code is designed so that we don't
- * actually believe these cases are okay without further evidence (namely,
- * a pending fsync request getting canceled ... see mdsync).
- */
-#ifndef WIN32
-#define FILE_POSSIBLY_DELETED(err) ((err) == ENOENT)
-#else
-#define FILE_POSSIBLY_DELETED(err) ((err) == ENOENT || (err) == EACCES)
-#endif
-
/*
* The magnetic disk storage manager keeps track of open file
* descriptors in its own descriptor pool. This is done to make it
@@ -114,53 +86,27 @@ typedef struct _MdfdVec
static MemoryContext MdCxt; /* context for all MdfdVec objects */
+/* local routines */
+static void mdunlinkfork(RelFileNodeBackend rnode, ForkNumber forkNum,
+ bool isRedo);
+static MdfdVec *mdopen(SMgrRelation reln, ForkNumber forknum, int behavior);
+static void register_dirty_segment(SMgrRelation reln, ForkNumber forknum,
+ MdfdVec *seg);
+static void register_unlink(RelFileNodeBackend rnode);
+static void _fdvec_resize(SMgrRelation reln,
+ ForkNumber forknum,
+ int nseg);
+static MdfdVec *_mdfd_openseg(SMgrRelation reln, ForkNumber forkno,
+ BlockNumber segno, int oflags);
+static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forkno,
+ BlockNumber blkno, bool skipFsync, int behavior);
+static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
+ MdfdVec *seg);
+
/*
- * In some contexts (currently, standalone backends and the checkpointer)
- * we keep track of pending fsync operations: we need to remember all relation
- * segments that have been written since the last checkpoint, so that we can
- * fsync them down to disk before completing the next checkpoint. This hash
- * table remembers the pending operations. We use a hash table mostly as
- * a convenient way of merging duplicate requests.
- *
- * We use a similar mechanism to remember no-longer-needed files that can
- * be deleted after the next checkpoint, but we use a linked list instead of
- * a hash table, because we don't expect there to be any duplicate requests.
- *
- * These mechanisms are only used for non-temp relations; we never fsync
- * temp rels, nor do we need to postpone their deletion (see comments in
- * mdunlink).
- *
- * (Regular backends do not track pending operations locally, but forward
- * them to the checkpointer.)
+ * Segment handling behaviors
*/
-typedef uint16 CycleCtr; /* can be any convenient integer size */
-
-typedef struct
-{
- RelFileNode rnode; /* hash table key (must be first!) */
- CycleCtr cycle_ctr; /* mdsync_cycle_ctr of oldest request */
- /* requests[f] has bit n set if we need to fsync segment n of fork f */
- Bitmapset *requests[MAX_FORKNUM + 1];
- /* canceled[f] is true if we canceled fsyncs for fork "recently" */
- bool canceled[MAX_FORKNUM + 1];
-} PendingOperationEntry;
-
-typedef struct
-{
- RelFileNode rnode; /* the dead relation to delete */
- CycleCtr cycle_ctr; /* mdckpt_cycle_ctr when request was made */
-} PendingUnlinkEntry;
-
-static HTAB *pendingOpsTable = NULL;
-static List *pendingUnlinks = NIL;
-static MemoryContext pendingOpsCxt; /* context for the above */
-
-static CycleCtr mdsync_cycle_ctr = 0;
-static CycleCtr mdckpt_cycle_ctr = 0;
-
-
-/*** behavior for mdopen & _mdfd_getseg ***/
/* ereport if segment not present */
#define EXTENSION_FAIL (1 << 0)
/* return NULL if segment not present */
@@ -179,26 +125,6 @@ static CycleCtr mdckpt_cycle_ctr = 0;
#define EXTENSION_DONT_CHECK_SIZE (1 << 4)
-/* local routines */
-static void mdunlinkfork(RelFileNodeBackend rnode, ForkNumber forkNum,
- bool isRedo);
-static MdfdVec *mdopen(SMgrRelation reln, ForkNumber forknum, int behavior);
-static void register_dirty_segment(SMgrRelation reln, ForkNumber forknum,
- MdfdVec *seg);
-static void register_unlink(RelFileNodeBackend rnode);
-static void _fdvec_resize(SMgrRelation reln,
- ForkNumber forknum,
- int nseg);
-static char *_mdfd_segpath(SMgrRelation reln, ForkNumber forknum,
- BlockNumber segno);
-static MdfdVec *_mdfd_openseg(SMgrRelation reln, ForkNumber forkno,
- BlockNumber segno, int oflags);
-static MdfdVec *_mdfd_getseg(SMgrRelation reln, ForkNumber forkno,
- BlockNumber blkno, bool skipFsync, int behavior);
-static BlockNumber _mdnblocks(SMgrRelation reln, ForkNumber forknum,
- MdfdVec *seg);
-
-
/*
* mdinit() -- Initialize private state for magnetic disk storage manager.
*/
@@ -208,64 +134,6 @@ mdinit(void)
MdCxt = AllocSetContextCreate(TopMemoryContext,
"MdSmgr",
ALLOCSET_DEFAULT_SIZES);
-
- /*
- * Create pending-operations hashtable if we need it. Currently, we need
- * it if we are standalone (not under a postmaster) or if we are a startup
- * or checkpointer auxiliary process.
- */
- if (!IsUnderPostmaster || AmStartupProcess() || AmCheckpointerProcess())
- {
- HASHCTL hash_ctl;
-
- /*
- * XXX: The checkpointer needs to add entries to the pending ops table
- * when absorbing fsync requests. That is done within a critical
- * section, which isn't usually allowed, but we make an exception. It
- * means that there's a theoretical possibility that you run out of
- * memory while absorbing fsync requests, which leads to a PANIC.
- * Fortunately the hash table is small so that's unlikely to happen in
- * practice.
- */
- pendingOpsCxt = AllocSetContextCreate(MdCxt,
- "Pending ops context",
- ALLOCSET_DEFAULT_SIZES);
- MemoryContextAllowInCriticalSection(pendingOpsCxt, true);
-
- MemSet(&hash_ctl, 0, sizeof(hash_ctl));
- hash_ctl.keysize = sizeof(RelFileNode);
- hash_ctl.entrysize = sizeof(PendingOperationEntry);
- hash_ctl.hcxt = pendingOpsCxt;
- pendingOpsTable = hash_create("Pending Ops Table",
- 100L,
- &hash_ctl,
- HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
- pendingUnlinks = NIL;
- }
-}
-
-/*
- * In archive recovery, we rely on checkpointer to do fsyncs, but we will have
- * already created the pendingOpsTable during initialization of the startup
- * process. Calling this function drops the local pendingOpsTable so that
- * subsequent requests will be forwarded to checkpointer.
- */
-void
-SetForwardFsyncRequests(void)
-{
- /* Perform any pending fsyncs we may have queued up, then drop table */
- if (pendingOpsTable)
- {
- mdsync();
- hash_destroy(pendingOpsTable);
- }
- pendingOpsTable = NULL;
-
- /*
- * We should not have any pending unlink requests, since mdunlink doesn't
- * queue unlink requests when isRedo.
- */
- Assert(pendingUnlinks == NIL);
}
/*
@@ -380,16 +248,6 @@ mdcreate(SMgrRelation reln, ForkNumber forkNum, bool isRedo)
void
mdunlink(RelFileNodeBackend rnode, ForkNumber forkNum, bool isRedo)
{
- /*
- * We have to clean out any pending fsync requests for the doomed
- * relation, else the next mdsync() will fail. There can't be any such
- * requests for a temp relation, though. We can send just one request
- * even when deleting multiple forks, since the fsync queuing code accepts
- * the "InvalidForkNumber = all forks" convention.
- */
- if (!RelFileNodeBackendIsTemp(rnode))
- ForgetRelationFsyncRequests(rnode.node, forkNum);
-
/* Now do the per-fork work */
if (forkNum == InvalidForkNumber)
{
@@ -408,6 +266,10 @@ mdunlinkfork(RelFileNodeBackend rnode, ForkNumber forkNum, bool isRedo)
path = relpath(rnode, forkNum);
+ /* Forget any pending sync requests for the first segment */
+ if (!RelFileNodeBackendIsTemp(rnode))
+ ForgetRelationSyncRequests(rnode.node, forkNum, InvalidSegmentNumber);
+
/*
* Delete or truncate the first segment.
*/
@@ -459,6 +321,10 @@ mdunlinkfork(RelFileNodeBackend rnode, ForkNumber forkNum, bool isRedo)
*/
for (segno = 1;; segno++)
{
+ /* Forget any pending sync requests for the rest of the segments */
+ if (!RelFileNodeBackendIsTemp(rnode))
+ ForgetRelationSyncRequests(rnode.node, forkNum, segno);
+
sprintf(segpath, "%s.%u", path, segno);
if (unlink(segpath) < 0)
{
@@ -1004,385 +870,50 @@ mdimmedsync(SMgrRelation reln, ForkNumber forknum)
}
/*
- * mdsync() -- Sync previous writes to stable storage.
+ * mdfilepath()
+ *
+ * Return the filename for the specified segment of the relation. The
+ * returned string is palloc'd.
*/
-void
-mdsync(void)
+char *
+mdfilepath(FileTag ftag)
{
- static bool mdsync_in_progress = false;
-
- HASH_SEQ_STATUS hstat;
- PendingOperationEntry *entry;
- int absorb_counter;
-
- /* Statistics on sync times */
- int processed = 0;
- instr_time sync_start,
- sync_end,
- sync_diff;
- uint64 elapsed;
- uint64 longest = 0;
- uint64 total_elapsed = 0;
-
- /*
- * This is only called during checkpoints, and checkpoints should only
- * occur in processes that have created a pendingOpsTable.
- */
- if (!pendingOpsTable)
- elog(ERROR, "cannot sync without a pendingOpsTable");
+ char *path,
+ *fullpath;
/*
- * If we are in the checkpointer, the sync had better include all fsync
- * requests that were queued by backends up to this point. The tightest
- * race condition that could occur is that a buffer that must be written
- * and fsync'd for the checkpoint could have been dumped by a backend just
- * before it was visited by BufferSync(). We know the backend will have
- * queued an fsync request before clearing the buffer's dirtybit, so we
- * are safe as long as we do an Absorb after completing BufferSync().
+ * We can safely pass InvalidBackendId as we never expect to sync
+ * any segments for temporary relations.
*/
- AbsorbFsyncRequests();
+ path = GetRelationPath(ftag.rnode.dbNode, ftag.rnode.spcNode,
+ ftag.rnode.relNode, InvalidBackendId, ftag.forknum);
- /*
- * To avoid excess fsync'ing (in the worst case, maybe a never-terminating
- * checkpoint), we want to ignore fsync requests that are entered into the
- * hashtable after this point --- they should be processed next time,
- * instead. We use mdsync_cycle_ctr to tell old entries apart from new
- * ones: new ones will have cycle_ctr equal to the incremented value of
- * mdsync_cycle_ctr.
- *
- * In normal circumstances, all entries present in the table at this point
- * will have cycle_ctr exactly equal to the current (about to be old)
- * value of mdsync_cycle_ctr. However, if we fail partway through the
- * fsync'ing loop, then older values of cycle_ctr might remain when we
- * come back here to try again. Repeated checkpoint failures would
- * eventually wrap the counter around to the point where an old entry
- * might appear new, causing us to skip it, possibly allowing a checkpoint
- * to succeed that should not have. To forestall wraparound, any time the
- * previous mdsync() failed to complete, run through the table and
- * forcibly set cycle_ctr = mdsync_cycle_ctr.
- *
- * Think not to merge this loop with the main loop, as the problem is
- * exactly that that loop may fail before having visited all the entries.
- * From a performance point of view it doesn't matter anyway, as this path
- * will never be taken in a system that's functioning normally.
- */
- if (mdsync_in_progress)
+ if (ftag.segno > 0 && ftag.segno != InvalidSegmentNumber)
{
- /* prior try failed, so update any stale cycle_ctr values */
- hash_seq_init(&hstat, pendingOpsTable);
- while ((entry = (PendingOperationEntry *) hash_seq_search(&hstat)) != NULL)
- {
- entry->cycle_ctr = mdsync_cycle_ctr;
- }
+ fullpath = psprintf("%s.%u", path, ftag.segno);
+ pfree(path);
}
+ else
+ fullpath = path;
- /* Advance counter so that new hashtable entries are distinguishable */
- mdsync_cycle_ctr++;
-
- /* Set flag to detect failure if we don't reach the end of the loop */
- mdsync_in_progress = true;
-
- /* Now scan the hashtable for fsync requests to process */
- absorb_counter = FSYNCS_PER_ABSORB;
- hash_seq_init(&hstat, pendingOpsTable);
- while ((entry = (PendingOperationEntry *) hash_seq_search(&hstat)) != NULL)
- {
- ForkNumber forknum;
-
- /*
- * If the entry is new then don't process it this time; it might
- * contain multiple fsync-request bits, but they are all new. Note
- * "continue" bypasses the hash-remove call at the bottom of the loop.
- */
- if (entry->cycle_ctr == mdsync_cycle_ctr)
- continue;
-
- /* Else assert we haven't missed it */
- Assert((CycleCtr) (entry->cycle_ctr + 1) == mdsync_cycle_ctr);
-
- /*
- * Scan over the forks and segments represented by the entry.
- *
- * The bitmap manipulations are slightly tricky, because we can call
- * AbsorbFsyncRequests() inside the loop and that could result in
- * bms_add_member() modifying and even re-palloc'ing the bitmapsets.
- * So we detach it, but if we fail we'll merge it with any new
- * requests that have arrived in the meantime.
- */
- for (forknum = 0; forknum <= MAX_FORKNUM; forknum++)
- {
- Bitmapset *requests = entry->requests[forknum];
- int segno;
-
- entry->requests[forknum] = NULL;
- entry->canceled[forknum] = false;
-
- segno = -1;
- while ((segno = bms_next_member(requests, segno)) >= 0)
- {
- int failures;
-
- /*
- * If fsync is off then we don't have to bother opening the
- * file at all. (We delay checking until this point so that
- * changing fsync on the fly behaves sensibly.)
- */
- if (!enableFsync)
- continue;
-
- /*
- * If in checkpointer, we want to absorb pending requests
- * every so often to prevent overflow of the fsync request
- * queue. It is unspecified whether newly-added entries will
- * be visited by hash_seq_search, but we don't care since we
- * don't need to process them anyway.
- */
- if (--absorb_counter <= 0)
- {
- AbsorbFsyncRequests();
- absorb_counter = FSYNCS_PER_ABSORB;
- }
-
- /*
- * The fsync table could contain requests to fsync segments
- * that have been deleted (unlinked) by the time we get to
- * them. Rather than just hoping an ENOENT (or EACCES on
- * Windows) error can be ignored, what we do on error is
- * absorb pending requests and then retry. Since mdunlink()
- * queues a "cancel" message before actually unlinking, the
- * fsync request is guaranteed to be marked canceled after the
- * absorb if it really was this case. DROP DATABASE likewise
- * has to tell us to forget fsync requests before it starts
- * deletions.
- */
- for (failures = 0;; failures++) /* loop exits at "break" */
- {
- SMgrRelation reln;
- MdfdVec *seg;
- char *path;
- int save_errno;
-
- /*
- * Find or create an smgr hash entry for this relation.
- * This may seem a bit unclean -- md calling smgr? But
- * it's really the best solution. It ensures that the
- * open file reference isn't permanently leaked if we get
- * an error here. (You may say "but an unreferenced
- * SMgrRelation is still a leak!" Not really, because the
- * only case in which a checkpoint is done by a process
- * that isn't about to shut down is in the checkpointer,
- * and it will periodically do smgrcloseall(). This fact
- * justifies our not closing the reln in the success path
- * either, which is a good thing since in non-checkpointer
- * cases we couldn't safely do that.)
- */
- reln = smgropen(entry->rnode, InvalidBackendId);
-
- /* Attempt to open and fsync the target segment */
- seg = _mdfd_getseg(reln, forknum,
- (BlockNumber) segno * (BlockNumber) RELSEG_SIZE,
- false,
- EXTENSION_RETURN_NULL
- | EXTENSION_DONT_CHECK_SIZE);
-
- INSTR_TIME_SET_CURRENT(sync_start);
-
- if (seg != NULL &&
- FileSync(seg->mdfd_vfd, WAIT_EVENT_DATA_FILE_SYNC) >= 0)
- {
- /* Success; update statistics about sync timing */
- INSTR_TIME_SET_CURRENT(sync_end);
- sync_diff = sync_end;
- INSTR_TIME_SUBTRACT(sync_diff, sync_start);
- elapsed = INSTR_TIME_GET_MICROSEC(sync_diff);
- if (elapsed > longest)
- longest = elapsed;
- total_elapsed += elapsed;
- processed++;
- requests = bms_del_member(requests, segno);
- if (log_checkpoints)
- elog(DEBUG1, "checkpoint sync: number=%d file=%s time=%.3f msec",
- processed,
- FilePathName(seg->mdfd_vfd),
- (double) elapsed / 1000);
-
- break; /* out of retry loop */
- }
-
- /* Compute file name for use in message */
- save_errno = errno;
- path = _mdfd_segpath(reln, forknum, (BlockNumber) segno);
- errno = save_errno;
-
- /*
- * It is possible that the relation has been dropped or
- * truncated since the fsync request was entered.
- * Therefore, allow ENOENT, but only if we didn't fail
- * already on this file. This applies both for
- * _mdfd_getseg() and for FileSync, since fd.c might have
- * closed the file behind our back.
- *
- * XXX is there any point in allowing more than one retry?
- * Don't see one at the moment, but easy to change the
- * test here if so.
- */
- if (!FILE_POSSIBLY_DELETED(errno) ||
- failures > 0)
- {
- Bitmapset *new_requests;
-
- /*
- * We need to merge these unsatisfied requests with
- * any others that have arrived since we started.
- */
- new_requests = entry->requests[forknum];
- entry->requests[forknum] =
- bms_join(new_requests, requests);
-
- errno = save_errno;
- ereport(data_sync_elevel(ERROR),
- (errcode_for_file_access(),
- errmsg("could not fsync file \"%s\": %m",
- path)));
- }
- else
- ereport(DEBUG1,
- (errcode_for_file_access(),
- errmsg("could not fsync file \"%s\" but retrying: %m",
- path)));
- pfree(path);
-
- /*
- * Absorb incoming requests and check to see if a cancel
- * arrived for this relation fork.
- */
- AbsorbFsyncRequests();
- absorb_counter = FSYNCS_PER_ABSORB; /* might as well... */
-
- if (entry->canceled[forknum])
- break;
- } /* end retry loop */
- }
- bms_free(requests);
- }
-
- /*
- * We've finished everything that was requested before we started to
- * scan the entry. If no new requests have been inserted meanwhile,
- * remove the entry. Otherwise, update its cycle counter, as all the
- * requests now in it must have arrived during this cycle.
- */
- for (forknum = 0; forknum <= MAX_FORKNUM; forknum++)
- {
- if (entry->requests[forknum] != NULL)
- break;
- }
- if (forknum <= MAX_FORKNUM)
- entry->cycle_ctr = mdsync_cycle_ctr;
- else
- {
- /* Okay to remove it */
- if (hash_search(pendingOpsTable, &entry->rnode,
- HASH_REMOVE, NULL) == NULL)
- elog(ERROR, "pendingOpsTable corrupted");
- }
- } /* end loop over hashtable entries */
-
- /* Return sync performance metrics for report at checkpoint end */
- CheckpointStats.ckpt_sync_rels = processed;
- CheckpointStats.ckpt_longest_sync = longest;
- CheckpointStats.ckpt_agg_sync_time = total_elapsed;
-
- /* Flag successful completion of mdsync */
- mdsync_in_progress = false;
-}
-
-/*
- * mdpreckpt() -- Do pre-checkpoint work
- *
- * To distinguish unlink requests that arrived before this checkpoint
- * started from those that arrived during the checkpoint, we use a cycle
- * counter similar to the one we use for fsync requests. That cycle
- * counter is incremented here.
- *
- * This must be called *before* the checkpoint REDO point is determined.
- * That ensures that we won't delete files too soon.
- *
- * Note that we can't do anything here that depends on the assumption
- * that the checkpoint will be completed.
- */
-void
-mdpreckpt(void)
-{
- /*
- * Any unlink requests arriving after this point will be assigned the next
- * cycle counter, and won't be unlinked until next checkpoint.
- */
- mdckpt_cycle_ctr++;
+ return fullpath;
}
/*
- * mdpostckpt() -- Do post-checkpoint work
+ * mdtagmatches()
*
- * Remove any lingering files that can now be safely removed.
+ * Returns true if the predicate tag matches with the file tag.
*/
-void
-mdpostckpt(void)
+bool
+mdtagmatches(FileTag ftag, FileTag predicate, SyncRequestType type)
{
- int absorb_counter;
-
- absorb_counter = UNLINKS_PER_ABSORB;
- while (pendingUnlinks != NIL)
- {
- PendingUnlinkEntry *entry = (PendingUnlinkEntry *) linitial(pendingUnlinks);
- char *path;
+ /* Today, we only do matching for hierarchy (forget database) requests */
+ Assert(type == FORGET_HIERARCHY_REQUEST);
- /*
- * New entries are appended to the end, so if the entry is new we've
- * reached the end of old entries.
- *
- * Note: if just the right number of consecutive checkpoints fail, we
- * could be fooled here by cycle_ctr wraparound. However, the only
- * consequence is that we'd delay unlinking for one more checkpoint,
- * which is perfectly tolerable.
- */
- if (entry->cycle_ctr == mdckpt_cycle_ctr)
- break;
-
- /* Unlink the file */
- path = relpathperm(entry->rnode, MAIN_FORKNUM);
- if (unlink(path) < 0)
- {
- /*
- * There's a race condition, when the database is dropped at the
- * same time that we process the pending unlink requests. If the
- * DROP DATABASE deletes the file before we do, we will get ENOENT
- * here. rmtree() also has to ignore ENOENT errors, to deal with
- * the possibility that we delete the file first.
- */
- if (errno != ENOENT)
- ereport(WARNING,
- (errcode_for_file_access(),
- errmsg("could not remove file \"%s\": %m", path)));
- }
- pfree(path);
+ if (type == FORGET_HIERARCHY_REQUEST)
+ return ftag.rnode.dbNode == predicate.rnode.dbNode;
- /* And remove the list entry */
- pendingUnlinks = list_delete_first(pendingUnlinks);
- pfree(entry);
-
- /*
- * As in mdsync, we don't want to stop absorbing fsync requests for a
- * long time when there are many deletions to be done. We can safely
- * call AbsorbFsyncRequests() at this point in the loop (note it might
- * try to delete list entries).
- */
- if (--absorb_counter <= 0)
- {
- AbsorbFsyncRequests();
- absorb_counter = UNLINKS_PER_ABSORB;
- }
- }
+ return false;
}
/*
@@ -1397,17 +928,22 @@ mdpostckpt(void)
static void
register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
{
+ FileTag tag;
+ tag.rnode = reln->smgr_rnode.node;
+ tag.forknum = forknum;
+ tag.segno = seg->mdfd_segno;
+
/* Temp relations should never be fsync'd */
Assert(!SmgrIsTemp(reln));
- if (pendingOpsTable)
+ if (IsSyncManagedLocally())
{
/* push it into local pending-ops table */
- RememberFsyncRequest(reln->smgr_rnode.node, forknum, seg->mdfd_segno);
+ RememberSyncRequest(tag, SYNC_REQUEST, SYNC_MD);
}
else
{
- if (ForwardFsyncRequest(reln->smgr_rnode.node, forknum, seg->mdfd_segno))
+ if (ForwardSyncRequest(tag, SYNC_REQUEST, SYNC_MD))
return; /* passed it off successfully */
ereport(DEBUG1,
@@ -1433,14 +969,18 @@ register_dirty_segment(SMgrRelation reln, ForkNumber forknum, MdfdVec *seg)
static void
register_unlink(RelFileNodeBackend rnode)
{
+ FileTag tag;
+ tag.rnode = rnode.node;
+ tag.forknum = MAIN_FORKNUM;
+ tag.segno = InvalidSegmentNumber;
+
/* Should never be used with temp relations */
Assert(!RelFileNodeBackendIsTemp(rnode));
- if (pendingOpsTable)
+ if (IsSyncManagedLocally())
{
/* push it into local pending-ops table */
- RememberFsyncRequest(rnode.node, MAIN_FORKNUM,
- UNLINK_RELATION_REQUEST);
+ RememberSyncRequest(tag, UNLINK_REQUEST, SYNC_MD);
}
else
{
@@ -1452,165 +992,11 @@ register_unlink(RelFileNodeBackend rnode)
* XXX should we just leave the file orphaned instead?
*/
Assert(IsUnderPostmaster);
- while (!ForwardFsyncRequest(rnode.node, MAIN_FORKNUM,
- UNLINK_RELATION_REQUEST))
+ while (!ForwardSyncRequest(tag, UNLINK_REQUEST, SYNC_MD))
pg_usleep(10000L); /* 10 msec seems a good number */
}
}
-/*
- * RememberFsyncRequest() -- callback from checkpointer side of fsync request
- *
- * We stuff fsync requests into the local hash table for execution
- * during the checkpointer's next checkpoint. UNLINK requests go into a
- * separate linked list, however, because they get processed separately.
- *
- * The range of possible segment numbers is way less than the range of
- * BlockNumber, so we can reserve high values of segno for special purposes.
- * We define three:
- * - FORGET_RELATION_FSYNC means to cancel pending fsyncs for a relation,
- * either for one fork, or all forks if forknum is InvalidForkNumber
- * - FORGET_DATABASE_FSYNC means to cancel pending fsyncs for a whole database
- * - UNLINK_RELATION_REQUEST is a request to delete the file after the next
- * checkpoint.
- * Note also that we're assuming real segment numbers don't exceed INT_MAX.
- *
- * (Handling FORGET_DATABASE_FSYNC requests is a tad slow because the hash
- * table has to be searched linearly, but dropping a database is a pretty
- * heavyweight operation anyhow, so we'll live with it.)
- */
-void
-RememberFsyncRequest(RelFileNode rnode, ForkNumber forknum, BlockNumber segno)
-{
- Assert(pendingOpsTable);
-
- if (segno == FORGET_RELATION_FSYNC)
- {
- /* Remove any pending requests for the relation (one or all forks) */
- PendingOperationEntry *entry;
-
- entry = (PendingOperationEntry *) hash_search(pendingOpsTable,
- &rnode,
- HASH_FIND,
- NULL);
- if (entry)
- {
- /*
- * We can't just delete the entry since mdsync could have an
- * active hashtable scan. Instead we delete the bitmapsets; this
- * is safe because of the way mdsync is coded. We also set the
- * "canceled" flags so that mdsync can tell that a cancel arrived
- * for the fork(s).
- */
- if (forknum == InvalidForkNumber)
- {
- /* remove requests for all forks */
- for (forknum = 0; forknum <= MAX_FORKNUM; forknum++)
- {
- bms_free(entry->requests[forknum]);
- entry->requests[forknum] = NULL;
- entry->canceled[forknum] = true;
- }
- }
- else
- {
- /* remove requests for single fork */
- bms_free(entry->requests[forknum]);
- entry->requests[forknum] = NULL;
- entry->canceled[forknum] = true;
- }
- }
- }
- else if (segno == FORGET_DATABASE_FSYNC)
- {
- /* Remove any pending requests for the entire database */
- HASH_SEQ_STATUS hstat;
- PendingOperationEntry *entry;
- ListCell *cell,
- *prev,
- *next;
-
- /* Remove fsync requests */
- hash_seq_init(&hstat, pendingOpsTable);
- while ((entry = (PendingOperationEntry *) hash_seq_search(&hstat)) != NULL)
- {
- if (entry->rnode.dbNode == rnode.dbNode)
- {
- /* remove requests for all forks */
- for (forknum = 0; forknum <= MAX_FORKNUM; forknum++)
- {
- bms_free(entry->requests[forknum]);
- entry->requests[forknum] = NULL;
- entry->canceled[forknum] = true;
- }
- }
- }
-
- /* Remove unlink requests */
- prev = NULL;
- for (cell = list_head(pendingUnlinks); cell; cell = next)
- {
- PendingUnlinkEntry *entry = (PendingUnlinkEntry *) lfirst(cell);
-
- next = lnext(cell);
- if (entry->rnode.dbNode == rnode.dbNode)
- {
- pendingUnlinks = list_delete_cell(pendingUnlinks, cell, prev);
- pfree(entry);
- }
- else
- prev = cell;
- }
- }
- else if (segno == UNLINK_RELATION_REQUEST)
- {
- /* Unlink request: put it in the linked list */
- MemoryContext oldcxt = MemoryContextSwitchTo(pendingOpsCxt);
- PendingUnlinkEntry *entry;
-
- /* PendingUnlinkEntry doesn't store forknum, since it's always MAIN */
- Assert(forknum == MAIN_FORKNUM);
-
- entry = palloc(sizeof(PendingUnlinkEntry));
- entry->rnode = rnode;
- entry->cycle_ctr = mdckpt_cycle_ctr;
-
- pendingUnlinks = lappend(pendingUnlinks, entry);
-
- MemoryContextSwitchTo(oldcxt);
- }
- else
- {
- /* Normal case: enter a request to fsync this segment */
- MemoryContext oldcxt = MemoryContextSwitchTo(pendingOpsCxt);
- PendingOperationEntry *entry;
- bool found;
-
- entry = (PendingOperationEntry *) hash_search(pendingOpsTable,
- &rnode,
- HASH_ENTER,
- &found);
- /* if new entry, initialize it */
- if (!found)
- {
- entry->cycle_ctr = mdsync_cycle_ctr;
- MemSet(entry->requests, 0, sizeof(entry->requests));
- MemSet(entry->canceled, 0, sizeof(entry->canceled));
- }
-
- /*
- * NB: it's intentional that we don't change cycle_ctr if the entry
- * already exists. The cycle_ctr must represent the oldest fsync
- * request that could be in the entry.
- */
-
- entry->requests[forknum] = bms_add_member(entry->requests[forknum],
- (int) segno);
-
- MemoryContextSwitchTo(oldcxt);
- }
-}
-
/*
* ForgetRelationFsyncRequests -- forget any fsyncs for a relation fork
*
@@ -1618,12 +1004,19 @@ RememberFsyncRequest(RelFileNode rnode, ForkNumber forknum, BlockNumber segno)
* actually know that, since it's just forwarding the request elsewhere.
*/
void
-ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum)
+ForgetRelationSyncRequests(RelFileNode rnode, ForkNumber forknum,
+ SegmentNumber segno)
{
- if (pendingOpsTable)
+ FileTag tag;
+
+ tag.rnode = rnode;
+ tag.forknum = forknum;
+ tag.segno = segno;
+
+ if (IsSyncManagedLocally())
{
/* standalone backend or startup process: fsync state is local */
- RememberFsyncRequest(rnode, forknum, FORGET_RELATION_FSYNC);
+ RememberSyncRequest(tag, FORGET_REQUEST, SYNC_MD);
}
else if (IsUnderPostmaster)
{
@@ -1637,7 +1030,7 @@ ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum)
* which would be bad, so I'm inclined to assume that the checkpointer
* will always empty the queue soon.
*/
- while (!ForwardFsyncRequest(rnode, forknum, FORGET_RELATION_FSYNC))
+ while (!ForwardSyncRequest(tag, FORGET_REQUEST, SYNC_MD))
pg_usleep(10000L); /* 10 msec seems a good number */
/*
@@ -1651,24 +1044,26 @@ ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum)
* ForgetDatabaseFsyncRequests -- forget any fsyncs and unlinks for a DB
*/
void
-ForgetDatabaseFsyncRequests(Oid dbid)
+ForgetDatabaseSyncRequests(Oid dbid)
{
- RelFileNode rnode;
+ FileTag tag;
- rnode.dbNode = dbid;
- rnode.spcNode = 0;
- rnode.relNode = 0;
+ tag.rnode.dbNode = dbid;
+ tag.rnode.spcNode = 0;
+ tag.rnode.relNode = 0;
+ tag.forknum = InvalidForkNumber;
+ tag.segno = InvalidSegmentNumber;
- if (pendingOpsTable)
+ if (IsSyncManagedLocally())
{
/* standalone backend or startup process: fsync state is local */
- RememberFsyncRequest(rnode, InvalidForkNumber, FORGET_DATABASE_FSYNC);
+ RememberSyncRequest(tag, FORGET_HIERARCHY_REQUEST, SYNC_MD);
+
}
else if (IsUnderPostmaster)
{
/* see notes in ForgetRelationFsyncRequests */
- while (!ForwardFsyncRequest(rnode, InvalidForkNumber,
- FORGET_DATABASE_FSYNC))
+ while (!ForwardSyncRequest(tag, FORGET_HIERARCHY_REQUEST, SYNC_MD))
pg_usleep(10000L); /* 10 msec seems a good number */
}
}
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 0c0bba4ab3..e10ad826aa 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -59,12 +59,8 @@ typedef struct f_smgr
void (*smgr_truncate) (SMgrRelation reln, ForkNumber forknum,
BlockNumber nblocks);
void (*smgr_immedsync) (SMgrRelation reln, ForkNumber forknum);
- void (*smgr_pre_ckpt) (void); /* may be NULL */
- void (*smgr_sync) (void); /* may be NULL */
- void (*smgr_post_ckpt) (void); /* may be NULL */
} f_smgr;
-
static const f_smgr smgrsw[] = {
/* magnetic disk */
{
@@ -82,15 +78,11 @@ static const f_smgr smgrsw[] = {
.smgr_nblocks = mdnblocks,
.smgr_truncate = mdtruncate,
.smgr_immedsync = mdimmedsync,
- .smgr_pre_ckpt = mdpreckpt,
- .smgr_sync = mdsync,
- .smgr_post_ckpt = mdpostckpt
}
};
static const int NSmgr = lengthof(smgrsw);
-
/*
* Each backend has a hashtable that stores all extant SMgrRelation objects.
* In addition, "unowned" SMgrRelation objects are chained together in a list.
@@ -751,52 +743,6 @@ smgrimmedsync(SMgrRelation reln, ForkNumber forknum)
smgrsw[reln->smgr_which].smgr_immedsync(reln, forknum);
}
-
-/*
- * smgrpreckpt() -- Prepare for checkpoint.
- */
-void
-smgrpreckpt(void)
-{
- int i;
-
- for (i = 0; i < NSmgr; i++)
- {
- if (smgrsw[i].smgr_pre_ckpt)
- smgrsw[i].smgr_pre_ckpt();
- }
-}
-
-/*
- * smgrsync() -- Sync files to disk during checkpoint.
- */
-void
-smgrsync(void)
-{
- int i;
-
- for (i = 0; i < NSmgr; i++)
- {
- if (smgrsw[i].smgr_sync)
- smgrsw[i].smgr_sync();
- }
-}
-
-/*
- * smgrpostckpt() -- Post-checkpoint cleanup.
- */
-void
-smgrpostckpt(void)
-{
- int i;
-
- for (i = 0; i < NSmgr; i++)
- {
- if (smgrsw[i].smgr_post_ckpt)
- smgrsw[i].smgr_post_ckpt();
- }
-}
-
/*
* AtEOXact_SMgr
*
diff --git a/src/backend/storage/sync/Makefile b/src/backend/storage/sync/Makefile
new file mode 100644
index 0000000000..cfc60cadb4
--- /dev/null
+++ b/src/backend/storage/sync/Makefile
@@ -0,0 +1,17 @@
+#-------------------------------------------------------------------------
+#
+# Makefile--
+# Makefile for storage/sync
+#
+# IDENTIFICATION
+# src/backend/storage/sync/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/backend/storage/sync
+top_builddir = ../../../..
+include $(top_builddir)/src/Makefile.global
+
+OBJS = sync.o
+
+include $(top_srcdir)/src/backend/common.mk
diff --git a/src/backend/storage/sync/sync.c b/src/backend/storage/sync/sync.c
new file mode 100644
index 0000000000..584b6c60d5
--- /dev/null
+++ b/src/backend/storage/sync/sync.c
@@ -0,0 +1,590 @@
+/*-------------------------------------------------------------------------
+ *
+ * sync.c
+ * File synchronization management code.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ *
+ * IDENTIFICATION
+ * src/backend/storage/sync/sync.c
+ *
+ *-------------------------------------------------------------------------
+ */
+#include "postgres.h"
+
+#include <unistd.h>
+#include <fcntl.h>
+#include <sys/file.h>
+
+#include "miscadmin.h"
+#include "pgstat.h"
+#include "access/xlogutils.h"
+#include "access/xlog.h"
+#include "commands/tablespace.h"
+#include "portability/instr_time.h"
+#include "postmaster/bgwriter.h"
+#include "storage/bufmgr.h"
+#include "storage/ipc.h"
+#include "utils/hsearch.h"
+#include "utils/memutils.h"
+#include "utils/inval.h"
+
+static MemoryContext pendingOpsCxt; /* context for the pending ops state */
+
+/*
+ * In some contexts (currently, standalone backends and the checkpointer)
+ * we keep track of pending fsync operations: we need to remember all relation
+ * segments that have been written since the last checkpoint, so that we can
+ * fsync them down to disk before completing the next checkpoint. This hash
+ * table remembers the pending operations. We use a hash table mostly as
+ * a convenient way of merging duplicate requests.
+ *
+ * We use a similar mechanism to remember no-longer-needed files that can
+ * be deleted after the next checkpoint, but we use a linked list instead of
+ * a hash table, because we don't expect there to be any duplicate requests.
+ *
+ * These mechanisms are only used for non-temp relations; we never fsync
+ * temp rels, nor do we need to postpone their deletion (see comments in
+ * mdunlink).
+ *
+ * (Regular backends do not track pending operations locally, but forward
+ * them to the checkpointer.)
+ */
+typedef uint16 CycleCtr; /* can be any convenient integer size */
+
+typedef struct
+{
+ FileTag ftag; /* hash table key (must be first!) */
+ SyncRequestOwner owner; /* owner for request resolution */
+ CycleCtr cycle_ctr; /* sync_cycle_ctr of oldest request */
+ bool canceled; /* canceled is true if we canceled "recently" */
+} PendingFsyncEntry;
+
+typedef struct
+{
+ FileTag ftag; /* tag for relation file to delete */
+ SyncRequestOwner owner; /* owner for request resolution */
+ CycleCtr cycle_ctr; /* checkpoint_cycle_ctr when request was made */
+} PendingUnlinkEntry;
+
+static HTAB *pendingOps = NULL;
+static List *pendingUnlinks = NIL;
+static MemoryContext pendingOpsCxt; /* context for the above */
+
+static CycleCtr sync_cycle_ctr = 0;
+static CycleCtr checkpoint_cycle_ctr = 0;
+
+/* Intervals for calling AbsorbFsyncRequests */
+#define FSYNCS_PER_ABSORB 10
+#define UNLINKS_PER_ABSORB 10
+
+/*
+ * This struct of function pointers defines the API between smgr.c and
+ * any individual storage manager module. Note that smgr subfunctions are
+ * generally expected to report problems via elog(ERROR). An exception is
+ * that smgr_unlink should use elog(WARNING), rather than erroring out,
+ * because we normally unlink relations during post-commit/abort cleanup,
+ * and so it's too late to raise an error. Also, various conditions that
+ * would normally be errors should be allowed during bootstrap and/or WAL
+ * recovery --- see comments in md.c for details.
+ */
+typedef struct f_sync
+{
+ char* (*sync_filepath) (FileTag ftag);
+ bool (*sync_tagmatches) (FileTag ftag,
+ FileTag predicate, SyncRequestType type);
+} f_sync;
+
+static const f_sync syncsw[] = {
+ /* magnetic disk */
+ {
+ .sync_filepath = mdfilepath,
+ .sync_tagmatches = mdtagmatches
+ }
+};
+
+static const int NSync = lengthof(syncsw);
+
+
+/*
+ * Initialize data structures for the file sync tracking.
+ */
+void
+InitSync(void)
+{
+ /*
+ * Create pending-operations hashtable if we need it. Currently, we need
+ * it if we are standalone (not under a postmaster) or if we are a startup
+ * or checkpointer auxiliary process.
+ */
+ if (!IsUnderPostmaster || AmStartupProcess() || AmCheckpointerProcess())
+ {
+ HASHCTL hash_ctl;
+
+ /*
+ * XXX: The checkpointer needs to add entries to the pending ops table
+ * when absorbing fsync requests. That is done within a critical
+ * section, which isn't usually allowed, but we make an exception. It
+ * means that there's a theoretical possibility that you run out of
+ * memory while absorbing fsync requests, which leads to a PANIC.
+ * Fortunately the hash table is small so that's unlikely to happen in
+ * practice.
+ */
+ pendingOpsCxt = AllocSetContextCreate(TopMemoryContext,
+ "Pending ops context",
+ ALLOCSET_DEFAULT_SIZES);
+ MemoryContextAllowInCriticalSection(pendingOpsCxt, true);
+
+ MemSet(&hash_ctl, 0, sizeof(hash_ctl));
+ hash_ctl.keysize = sizeof(FileTag);
+ hash_ctl.entrysize = sizeof(PendingFsyncEntry);
+ hash_ctl.hcxt = pendingOpsCxt;
+ pendingOps = hash_create("Pending Ops Table",
+ 100L,
+ &hash_ctl,
+ HASH_ELEM | HASH_BLOBS | HASH_CONTEXT);
+ pendingUnlinks = NIL;
+ }
+
+}
+
+/*
+ * PreCheckpoint() -- Do pre-checkpoint work
+ *
+ * To distinguish unlink requests that arrived before this checkpoint
+ * started from those that arrived during the checkpoint, we use a cycle
+ * counter similar to the one we use for fsync requests. That cycle
+ * counter is incremented here.
+ *
+ * This must be called *before* the checkpoint REDO point is determined.
+ * That ensures that we won't delete files too soon.
+ *
+ * Note that we can't do anything here that depends on the assumption
+ * that the checkpoint will be completed.
+ */
+void
+PreCheckpoint(void)
+{
+ /*
+ * Any unlink requests arriving after this point will be assigned the next
+ * cycle counter, and won't be unlinked until next checkpoint.
+ */
+ checkpoint_cycle_ctr++;
+}
+
+/*
+ * PostCheckpoint() -- Do post-checkpoint work
+ *
+ * Remove any lingering files that can now be safely removed.
+ */
+void
+PostCheckpoint(void)
+{
+ int absorb_counter;
+
+ absorb_counter = UNLINKS_PER_ABSORB;
+ while (pendingUnlinks != NIL)
+ {
+ PendingUnlinkEntry *entry = (PendingUnlinkEntry *) linitial(pendingUnlinks);
+ char *path;
+
+ /*
+ * New entries are appended to the end, so if the entry is new we've
+ * reached the end of old entries.
+ *
+ * Note: if just the right number of consecutive checkpoints fail, we
+ * could be fooled here by cycle_ctr wraparound. However, the only
+ * consequence is that we'd delay unlinking for one more checkpoint,
+ * which is perfectly tolerable.
+ */
+ if (entry->cycle_ctr == checkpoint_cycle_ctr)
+ break;
+
+ /* Unlink the file */
+ path = syncsw[entry->owner].sync_filepath(entry->ftag);
+ if (unlink(path) < 0)
+ {
+ /*
+ * There's a race condition, when the database is dropped at the
+ * same time that we process the pending unlink requests. If the
+ * DROP DATABASE deletes the file before we do, we will get ENOENT
+ * here. rmtree() also has to ignore ENOENT errors, to deal with
+ * the possibility that we delete the file first.
+ */
+ if (errno != ENOENT)
+ ereport(WARNING,
+ (errcode_for_file_access(),
+ errmsg("could not remove file \"%s\": %m", path)));
+ }
+ pfree(path);
+
+ /* And remove the list entry */
+ pendingUnlinks = list_delete_first(pendingUnlinks);
+ pfree(entry);
+
+ /*
+ * As in ProcessFsyncRequests, we don't want to stop absorbing fsync
+ * requests for along time when there are many deletions to be done. We
+ * can safelycall AbsorbFsyncRequests() at this point in the loop
+ * (note it might try to delete list entries).
+ */
+ if (--absorb_counter <= 0)
+ {
+ AbsorbSyncRequests();
+ absorb_counter = UNLINKS_PER_ABSORB;
+ }
+ }
+}
+
+/*
+
+ * ProcessSyncRequests() -- Process queued fsync requests.
+ */
+void
+ProcessSyncRequests(void)
+{
+ static bool sync_in_progress = false;
+
+ HASH_SEQ_STATUS hstat;
+ PendingFsyncEntry *entry;
+ int absorb_counter;
+
+ /* Statistics on sync times */
+ int processed = 0;
+ instr_time sync_start,
+ sync_end,
+ sync_diff;
+ uint64 elapsed;
+ uint64 longest = 0;
+ uint64 total_elapsed = 0;
+
+ /*
+ * This is only called during checkpoints, and checkpoints should only
+ * occur in processes that have created a pendingOps.
+ */
+ if (!pendingOps)
+ elog(ERROR, "cannot sync without a pendingOps table");
+
+ /*
+ * If we are in the checkpointer, the sync had better include all fsync
+ * requests that were queued by backends up to this point. The tightest
+ * race condition that could occur is that a buffer that must be written
+ * and fsync'd for the checkpoint could have been dumped by a backend just
+ * before it was visited by BufferSync(). We know the backend will have
+ * queued an fsync request before clearing the buffer's dirtybit, so we
+ * are safe as long as we do an Absorb after completing BufferSync().
+ */
+ AbsorbSyncRequests();
+
+ /*
+ * To avoid excess fsync'ing (in the worst case, maybe a never-terminating
+ * checkpoint), we want to ignore fsync requests that are entered into the
+ * hashtable after this point --- they should be processed next time,
+ * instead. We use sync_cycle_ctr to tell old entries apart from new
+ * ones: new ones will have cycle_ctr equal to the incremented value of
+ * sync_cycle_ctr.
+ *
+ * In normal circumstances, all entries present in the table at this point
+ * will have cycle_ctr exactly equal to the current (about to be old)
+ * value of sync_cycle_ctr. However, if we fail partway through the
+ * fsync'ing loop, then older values of cycle_ctr might remain when we
+ * come back here to try again. Repeated checkpoint failures would
+ * eventually wrap the counter around to the point where an old entry
+ * might appear new, causing us to skip it, possibly allowing a checkpoint
+ * to succeed that should not have. To forestall wraparound, any time the
+ * previous ProcessFsyncRequests() failed to complete, run through the
+ * table and forcibly set cycle_ctr = sync_cycle_ctr.
+ *
+ * Think not to merge this loop with the main loop, as the problem is
+ * exactly that that loop may fail before having visited all the entries.
+ * From a performance point of view it doesn't matter anyway, as this path
+ * will never be taken in a system that's functioning normally.
+ */
+ if (sync_in_progress)
+ {
+ /* prior try failed, so update any stale cycle_ctr values */
+ hash_seq_init(&hstat, pendingOps);
+ while ((entry = (PendingFsyncEntry *) hash_seq_search(&hstat)) != NULL)
+ {
+ entry->cycle_ctr = sync_cycle_ctr;
+ }
+ }
+
+ /* Advance counter so that new hashtable entries are distinguishable */
+ sync_cycle_ctr++;
+
+ /* Set flag to detect failure if we don't reach the end of the loop */
+ sync_in_progress = true;
+
+ /* Now scan the hashtable for fsync requests to process */
+ absorb_counter = FSYNCS_PER_ABSORB;
+ hash_seq_init(&hstat, pendingOps);
+ while ((entry = (PendingFsyncEntry *) hash_seq_search(&hstat)) != NULL)
+ {
+ int failures;
+
+ /*
+ * If fsync is off then we don't have to bother opening the
+ * file at all. (We delay checking until this point so that
+ * changing fsync on the fly behaves sensibly.)
+ */
+ if (!enableFsync)
+ continue;
+
+ /*
+ * If the entry is new then don't process it this time; it might
+ * contain multiple fsync-request bits, but they are all new. Note
+ * "continue" bypasses the hash-remove call at the bottom of the loop.
+ */
+ if (entry->cycle_ctr == sync_cycle_ctr)
+ continue;
+
+ /* Else assert we haven't missed it */
+ Assert((CycleCtr) (entry->cycle_ctr + 1) == sync_cycle_ctr);
+
+ /*
+ * If in checkpointer, we want to absorb pending requests
+ * every so often to prevent overflow of the fsync request
+ * queue. It is unspecified whether newly-added entries will
+ * be visited by hash_seq_search, but we don't care since we
+ * don't need to process them anyway.
+ */
+ if (--absorb_counter <= 0)
+ {
+ AbsorbSyncRequests();
+ absorb_counter = FSYNCS_PER_ABSORB;
+ }
+
+ /*
+ * The fsync table could contain requests to fsync segments
+ * that have been deleted (unlinked) by the time we get to
+ * them. Rather than just hoping an ENOENT (or EACCES on
+ * Windows) error can be ignored, what we do on error is
+ * absorb pending requests and then retry. Since mdunlink()
+ * queues a "cancel" message before actually unlinking, the
+ * fsync request is guaranteed to be marked canceled after the
+ * absorb if it really was this case. DROP DATABASE likewise
+ * has to tell us to forget fsync requests before it starts
+ * deletions.
+ */
+ for (failures = 0;; failures++) /* loop exits at "break" */
+ {
+ char *path;
+ File fd;
+
+ path = syncsw[entry->owner].sync_filepath(entry->ftag);
+ fd = PathNameOpenFile(path, O_RDWR | PG_BINARY);
+
+ INSTR_TIME_SET_CURRENT(sync_start);
+ if (fd >= 0 &&
+ FileSync(fd, WAIT_EVENT_DATA_FILE_SYNC) >= 0)
+ {
+ /* Success; update statistics about sync timing */
+ INSTR_TIME_SET_CURRENT(sync_end);
+ sync_diff = sync_end;
+ INSTR_TIME_SUBTRACT(sync_diff, sync_start);
+ elapsed = INSTR_TIME_GET_MICROSEC(sync_diff);
+ if (elapsed > longest)
+ longest = elapsed;
+ total_elapsed += elapsed;
+ processed++;
+
+ if (log_checkpoints)
+ elog(DEBUG1, "checkpoint sync: number=%d file=%s time=%.3f msec",
+ processed,
+ path,
+ (double) elapsed / 1000);
+
+ pfree(path);
+ break; /* out of retry loop */
+ }
+
+ /*
+ * It is possible that the relation has been dropped or
+ * truncated since the fsync request was entered.
+ * Therefore, allow ENOENT, but only if we didn't fail
+ * already on this file. This applies both for
+ * smgrgetseg() and for FileSync, since fd.c might have
+ * closed the file behind our back.
+ *
+ * XXX is there any point in allowing more than one retry?
+ * Don't see one at the moment, but easy to change the
+ * test here if so.
+ */
+ if (!FILE_POSSIBLY_DELETED(errno) || failures > 0)
+ ereport(data_sync_elevel(ERROR),
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\": %m",
+ path)));
+ else
+ ereport(DEBUG1,
+ (errcode_for_file_access(),
+ errmsg("could not fsync file \"%s\" but retrying: %m",
+ path)));
+
+ pfree(path);
+
+ /*
+ * Absorb incoming requests and check to see if a cancel
+ * arrived for this relation fork.
+ */
+ AbsorbSyncRequests();
+ absorb_counter = FSYNCS_PER_ABSORB; /* might as well... */
+
+ if (entry->canceled)
+ break;
+ } /* end retry loop */
+
+ /* We are done with this entry, remove it */
+ if (hash_search(pendingOps, &entry->ftag, HASH_REMOVE, NULL) == NULL)
+ elog(ERROR, "pendingOps corrupted");
+ } /* end loop over hashtable entries */
+
+ /* Return sync performance metrics for report at checkpoint end */
+ CheckpointStats.ckpt_sync_rels = processed;
+ CheckpointStats.ckpt_longest_sync = longest;
+ CheckpointStats.ckpt_agg_sync_time = total_elapsed;
+
+ /* Flag successful completion of ProcessSyncRequests */
+ sync_in_progress = false;
+}
+
+/*
+ * RememberSyncRequest() -- callback from checkpointer side of sync request
+ *
+ * We stuff fsync requests into the local hash table for execution
+ * during the checkpointer's next checkpoint. UNLINK requests go into a
+ * separate linked list, however, because they get processed separately.
+ *
+ * See sync.h for more information on the types of sync requests supported.
+ */
+void
+RememberSyncRequest(FileTag ftag, SyncRequestType type, SyncRequestOwner owner)
+{
+ Assert(pendingOps);
+
+ if (type == FORGET_REQUEST)
+ {
+ /* Remove previously entered request */
+ if (hash_search(pendingOps,
+ (void *) &ftag,
+ HASH_REMOVE, NULL) == NULL)
+ elog(DEBUG5, "pendingOps table couldn't find entry for forget request");
+ }
+ else if (type == FORGET_HIERARCHY_REQUEST)
+ {
+ /* Remove any pending requests for the entire database */
+ HASH_SEQ_STATUS hstat;
+ PendingFsyncEntry *entry;
+ ListCell *cell,
+ *prev,
+ *next;
+
+ /* Remove fsync requests */
+ hash_seq_init(&hstat, pendingOps);
+ while ((entry = (PendingFsyncEntry *) hash_seq_search(&hstat)) != NULL)
+ {
+ if (syncsw[entry->owner].sync_tagmatches(entry->ftag,
+ ftag /* predicate */, type))
+ {
+ entry->canceled = true;
+ }
+ }
+
+ /* Remove unlink requests */
+ prev = NULL;
+ for (cell = list_head(pendingUnlinks); cell; cell = next)
+ {
+ PendingUnlinkEntry *entry = (PendingUnlinkEntry *) lfirst(cell);
+
+ next = lnext(cell);
+ if (syncsw[entry->owner].sync_tagmatches(entry->ftag,
+ ftag /* predicate */, type))
+ {
+ pendingUnlinks = list_delete_cell(pendingUnlinks, cell, prev);
+ pfree(entry);
+ }
+ else
+ prev = cell;
+ }
+ }
+ else if (type == UNLINK_REQUEST)
+ {
+ /* Unlink request: put it in the linked list */
+ MemoryContext oldcxt = MemoryContextSwitchTo(pendingOpsCxt);
+ PendingUnlinkEntry *entry;
+
+ entry = palloc(sizeof(PendingUnlinkEntry));
+ entry->ftag = ftag;
+ entry->owner = owner;
+ entry->cycle_ctr = checkpoint_cycle_ctr;
+
+ pendingUnlinks = lappend(pendingUnlinks, entry);
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+ else
+ {
+ /* Normal case: enter a request to fsync this segment */
+ MemoryContext oldcxt = MemoryContextSwitchTo(pendingOpsCxt);
+ PendingFsyncEntry *entry;
+ bool found;
+
+ Assert(type == SYNC_REQUEST);
+
+ entry = (PendingFsyncEntry *) hash_search(pendingOps,
+ &ftag,
+ HASH_ENTER,
+ &found);
+ /* if new entry, initialize it */
+ if (!found)
+ {
+ entry->owner = owner;
+ entry->cycle_ctr = sync_cycle_ctr;
+ entry->canceled = false;
+ }
+
+ /*
+ * NB: it's intentional that we don't change cycle_ctr if the entry
+ * already exists. The cycle_ctr must represent the oldest fsync
+ * request that could be in the entry.
+ */
+
+ MemoryContextSwitchTo(oldcxt);
+ }
+}
+
+/*
+ * Are sync requests managed locally by the backend?
+ */
+bool
+IsSyncManagedLocally(void)
+{
+ return pendingOps != NULL;
+}
+
+/*
+ * In archive recovery, we rely on checkpointer to do fsyncs, but we will have
+ * already created the pendingOps during initialization of the startup
+ * process. Calling this function drops the local pendingOps so that
+ * subsequent requests will be forwarded to checkpointer.
+ */
+void
+EnableSyncRequestForwarding(void)
+{
+ /* Perform any pending fsyncs we may have queued up, then drop table */
+ if (pendingOps)
+ {
+ ProcessSyncRequests();
+ hash_destroy(pendingOps);
+ }
+ pendingOps = NULL;
+
+ /*
+ * We should not have any pending unlink requests, since mdunlink doesn't
+ * queue unlink requests when isRedo.
+ */
+ Assert(pendingUnlinks == NIL);
+}
diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c
index a5ee209f91..0326e6c6ed 100644
--- a/src/backend/utils/init/postinit.c
+++ b/src/backend/utils/init/postinit.c
@@ -50,6 +50,7 @@
#include "storage/proc.h"
#include "storage/sinvaladt.h"
#include "storage/smgr.h"
+#include "storage/sync.h"
#include "tcop/tcopprot.h"
#include "utils/acl.h"
#include "utils/fmgroids.h"
@@ -554,6 +555,7 @@ BaseInit(void)
/* Do local initialization of file, storage and buffer managers */
InitFileAccess();
+ InitSync();
smgrinit();
InitBufferPoolAccess();
}
diff --git a/src/include/postmaster/bgwriter.h b/src/include/postmaster/bgwriter.h
index 53b8f5fe3c..2d3672f84a 100644
--- a/src/include/postmaster/bgwriter.h
+++ b/src/include/postmaster/bgwriter.h
@@ -17,6 +17,8 @@
#include "storage/block.h"
#include "storage/relfilenode.h"
+#include "storage/smgr.h"
+#include "storage/sync.h"
/* GUC options */
@@ -31,9 +33,9 @@ extern void CheckpointerMain(void) pg_attribute_noreturn();
extern void RequestCheckpoint(int flags);
extern void CheckpointWriteDelay(int flags, double progress);
-extern bool ForwardFsyncRequest(RelFileNode rnode, ForkNumber forknum,
- BlockNumber segno);
-extern void AbsorbFsyncRequests(void);
+extern bool ForwardSyncRequest(FileTag ftag, SyncRequestType type,
+ SyncRequestOwner owner);
+extern void AbsorbSyncRequests(void);
extern Size CheckpointerShmemSize(void);
extern void CheckpointerShmemInit(void);
diff --git a/src/include/storage/fd.h b/src/include/storage/fd.h
index 74c34757fb..40f46b871d 100644
--- a/src/include/storage/fd.h
+++ b/src/include/storage/fd.h
@@ -54,6 +54,18 @@ extern PGDLLIMPORT bool data_sync_retry;
*/
extern int max_safe_fds;
+/*
+ * On Windows, we have to interpret EACCES as possibly meaning the same as
+ * ENOENT, because if a file is unlinked-but-not-yet-gone on that platform,
+ * that's what you get. Ugh. This code is designed so that we don't
+ * actually believe these cases are okay without further evidence (namely,
+ * a pending fsync request getting canceled ... see mdsync).
+ */
+#ifndef WIN32
+#define FILE_POSSIBLY_DELETED(err) ((err) == ENOENT)
+#else
+#define FILE_POSSIBLY_DELETED(err) ((err) == ENOENT || (err) == EACCES)
+#endif
/*
* prototypes for functions in fd.c
diff --git a/src/include/storage/segment.h b/src/include/storage/segment.h
new file mode 100644
index 0000000000..c7af945168
--- /dev/null
+++ b/src/include/storage/segment.h
@@ -0,0 +1,28 @@
+/*-------------------------------------------------------------------------
+ *
+ * segment.h
+ * POSTGRES disk segment definitions.
+ *
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/segment.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef SEGMENT_H
+#define SEGMENT_H
+
+
+/*
+ * Segment Number:
+ *
+ * Each relation and its forks are divided into segments. This
+ * definition formalizes the definition of the segment number.
+ */
+typedef uint32 SegmentNumber;
+
+#define InvalidSegmentNumber ((SegmentNumber) 0xFFFFFFFF)
+
+#endif /* SEGMENT_H */
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index 820d08ed4e..528a831140 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -17,7 +17,7 @@
#include "fmgr.h"
#include "storage/block.h"
#include "storage/relfilenode.h"
-
+#include "storage/segment.h"
/*
* smgr.c maintains a table of SMgrRelation objects, which are essentially
@@ -106,12 +106,8 @@ extern BlockNumber smgrnblocks(SMgrRelation reln, ForkNumber forknum);
extern void smgrtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber nblocks);
extern void smgrimmedsync(SMgrRelation reln, ForkNumber forknum);
-extern void smgrpreckpt(void);
-extern void smgrsync(void);
-extern void smgrpostckpt(void);
extern void AtEOXact_SMgr(void);
-
/* internals: move me elsewhere -- ay 7/94 */
/* in md.c */
@@ -134,15 +130,10 @@ extern BlockNumber mdnblocks(SMgrRelation reln, ForkNumber forknum);
extern void mdtruncate(SMgrRelation reln, ForkNumber forknum,
BlockNumber nblocks);
extern void mdimmedsync(SMgrRelation reln, ForkNumber forknum);
-extern void mdpreckpt(void);
-extern void mdsync(void);
-extern void mdpostckpt(void);
-extern void SetForwardFsyncRequests(void);
-extern void RememberFsyncRequest(RelFileNode rnode, ForkNumber forknum,
- BlockNumber segno);
-extern void ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum);
-extern void ForgetDatabaseFsyncRequests(Oid dbid);
+extern void ForgetRelationSyncRequests(RelFileNode rnode, ForkNumber forknum,
+ SegmentNumber segno);
+extern void ForgetDatabaseSyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileNode *delrels, int ndelrels, bool isRedo);
#endif /* SMGR_H */
diff --git a/src/include/storage/sync.h b/src/include/storage/sync.h
new file mode 100644
index 0000000000..b5a918a487
--- /dev/null
+++ b/src/include/storage/sync.h
@@ -0,0 +1,108 @@
+/*-------------------------------------------------------------------------
+ *
+ * sync.h
+ * File synchronization management code.
+ *
+ * Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+ * Portions Copyright (c) 1994, Regents of the University of California
+ *
+ * src/include/storage/sync.h
+ *
+ *-------------------------------------------------------------------------
+ */
+#ifndef SYNC_H
+#define SYNC_H
+
+#include "storage/block.h"
+#include "storage/relfilenode.h"
+#include "storage/segment.h"
+
+/*
+ * Caller specified type of sync request.
+ *
+ * SYNC_REQUESTs are issued to sync a particular file whose path is determined
+ * by calling back the owner. A FORGET_REQUEST instructs the sync mechanism to
+ * cancel a previously submitted sync request.
+ *
+ * FORGET_HIERARCHY_REQUEST is a special type of forget request that involves
+ * scanning all pending sync requests and cancelling any entry that matches. The
+ * entries are resolved by calling back the owner as the key is opaque to the
+ * sync mechanism. Handling these types of requests are a tad slow because we
+ * have to search all the requests linearly, but usage of this such as
+ * dropping databases, is a pretty heavyweight operation anyhow, so we'll live
+ * with it.
+ *
+ * UNLINK_REQUEST is a request to delete the file after the next checkpoint.
+ * The path is determined by calling back the owner.
+ */
+typedef enum syncrequesttype
+{
+ SYNC_REQUEST,
+ FORGET_REQUEST,
+ FORGET_HIERARCHY_REQUEST,
+ UNLINK_REQUEST
+} syncrequesttype;
+
+/*
+ * Identifies the owner for the sync callbacks.
+ *
+ * These enums map back to entries in the callback function table. For
+ * consistency, explicitly set the value to 0. See sync.c for more information.
+ */
+typedef enum syncrequestowner
+{
+ SYNC_MD = 0 /* md smgr */
+} syncrequestowner;
+
+/*
+ * Store the request type and owner identifier in uint8 to reduce the overall
+ * memory footprint of the SyncRequest structure which is used in the
+ * checkpointer queue.
+ */
+typedef uint8 SyncRequestType;
+typedef uint8 SyncRequestOwner;
+
+/*
+ * Augmenting a relfilenode with the fork and segment number provides all
+ * the information to locate the particular segment of interest for a relation.
+ */
+typedef struct
+{
+ RelFileNode rnode;
+ ForkNumber forknum;
+ SegmentNumber segno;
+} FileTag;
+
+typedef struct
+{
+ SyncRequestType type; /* type of sync request */
+ SyncRequestOwner owner; /* owner for request resolution */
+
+ /*
+ * Currently, sync requests can be satisfied by information available in
+ * the FileIdentifier. In the future, this can be combined with a
+ * physical file descriptor or the full path to a file and put inside
+ * an union.
+ *
+ * This value is opaque to sync mechanism and is used to pass to callback
+ * handlers to retrieve path of the file to sync or to resolve forget
+ * requests.
+ */
+ FileTag ftag;
+} SyncRequest;
+
+/* sync forward declarations */
+extern void InitSync(void);
+extern void PreCheckpoint(void);
+extern void PostCheckpoint(void);
+extern void ProcessSyncRequests(void);
+extern void RememberSyncRequest(FileTag ftag, SyncRequestType type,
+ SyncRequestOwner owner);
+extern bool IsSyncManagedLocally(void);
+extern void EnableSyncRequestForwarding(void);
+
+/* md callback forward declarations */
+extern char* mdfilepath(FileTag ftag);
+extern bool mdtagmatches(FileTag ftag, FileTag predicate, SyncRequestType type);
+
+#endif /* SYNC_H */
--
2.16.5
--huq684BweRXVnRxX--
^ permalink raw reply [nested|flat] 4+ messages in thread
* [PATCH v7 1/7] Row pattern recognition patch for raw parser.
@ 2023-09-22 04:53 Tatsuo Ishii <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Tatsuo Ishii @ 2023-09-22 04:53 UTC (permalink / raw)
---
src/backend/parser/gram.y | 222 ++++++++++++++++++++++++++++++--
src/include/nodes/parsenodes.h | 56 ++++++++
src/include/parser/kwlist.h | 8 ++
src/include/parser/parse_node.h | 1 +
4 files changed, 273 insertions(+), 14 deletions(-)
diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y
index 7d2032885e..74c2069050 100644
--- a/src/backend/parser/gram.y
+++ b/src/backend/parser/gram.y
@@ -251,6 +251,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
DefElem *defelt;
SortBy *sortby;
WindowDef *windef;
+ RPCommonSyntax *rpcom;
+ RPSubsetItem *rpsubset;
JoinExpr *jexpr;
IndexElem *ielem;
StatsElem *selem;
@@ -278,6 +280,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
MergeWhenClause *mergewhen;
struct KeyActions *keyactions;
struct KeyAction *keyaction;
+ RPSkipTo skipto;
}
%type <node> stmt toplevel_stmt schema_stmt routine_body_stmt
@@ -453,8 +456,12 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
TriggerTransitions TriggerReferencing
vacuum_relation_list opt_vacuum_relation_list
drop_option_list pub_obj_list
-
-%type <node> opt_routine_body
+ row_pattern_measure_list row_pattern_definition_list
+ opt_row_pattern_subset_clause
+ row_pattern_subset_list row_pattern_subset_rhs
+ row_pattern
+%type <rpsubset> row_pattern_subset_item
+%type <node> opt_routine_body row_pattern_term
%type <groupclause> group_clause
%type <list> group_by_list
%type <node> group_by_item empty_grouping_set rollup_clause cube_clause
@@ -551,6 +558,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <range> relation_expr_opt_alias
%type <node> tablesample_clause opt_repeatable_clause
%type <target> target_el set_target insert_column_item
+ row_pattern_measure_item row_pattern_definition
+%type <skipto> first_or_last
%type <str> generic_option_name
%type <node> generic_option_arg
@@ -633,6 +642,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
%type <list> window_clause window_definition_list opt_partition_clause
%type <windef> window_definition over_clause window_specification
opt_frame_clause frame_extent frame_bound
+%type <rpcom> opt_row_pattern_common_syntax opt_row_pattern_skip_to
+%type <boolean> opt_row_pattern_initial_or_seek
+%type <list> opt_row_pattern_measures
%type <ival> opt_window_exclusion_clause
%type <str> opt_existing_window_name
%type <boolean> opt_if_not_exists
@@ -659,7 +671,6 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
json_object_constructor_null_clause_opt
json_array_constructor_null_clause_opt
-
/*
* Non-keyword token types. These are hard-wired into the "flex" lexer.
* They must be listed first so that their numeric codes do not depend on
@@ -702,7 +713,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE
DATA_P DATABASE DAY_P DEALLOCATE DEC DECIMAL_P DECLARE DEFAULT DEFAULTS
- DEFERRABLE DEFERRED DEFINER DELETE_P DELIMITER DELIMITERS DEPENDS DEPTH DESC
+ DEFERRABLE DEFERRED DEFINE DEFINER DELETE_P DELIMITER DELIMITERS DEPENDS DEPTH DESC
DETACH DICTIONARY DISABLE_P DISCARD DISTINCT DO DOCUMENT_P DOMAIN_P
DOUBLE_P DROP
@@ -718,7 +729,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
HANDLER HAVING HEADER_P HOLD HOUR_P
IDENTITY_P IF_P ILIKE IMMEDIATE IMMUTABLE IMPLICIT_P IMPORT_P IN_P INCLUDE
- INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIALLY INLINE_P
+ INCLUDING INCREMENT INDENT INDEX INDEXES INHERIT INHERITS INITIAL INITIALLY INLINE_P
INNER_P INOUT INPUT_P INSENSITIVE INSERT INSTEAD INT_P INTEGER
INTERSECT INTERVAL INTO INVOKER IS ISNULL ISOLATION
@@ -731,7 +742,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
LEADING LEAKPROOF LEAST LEFT LEVEL LIKE LIMIT LISTEN LOAD LOCAL
LOCALTIME LOCALTIMESTAMP LOCATION LOCK_P LOCKED LOGGED
- MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MERGE METHOD
+ MAPPING MATCH MATCHED MATERIALIZED MAXVALUE MEASURES MERGE METHOD
MINUTE_P MINVALUE MODE MONTH_P MOVE
NAME_P NAMES NATIONAL NATURAL NCHAR NEW NEXT NFC NFD NFKC NFKD NO NONE
@@ -743,8 +754,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
ORDER ORDINALITY OTHERS OUT_P OUTER_P
OVER OVERLAPS OVERLAY OVERRIDING OWNED OWNER
- PARALLEL PARAMETER PARSER PARTIAL PARTITION PASSING PASSWORD
- PLACING PLANS POLICY
+ PARALLEL PARAMETER PARSER PARTIAL PARTITION PASSING PASSWORD PAST
+ PATTERN_P PERMUTE PLACING PLANS POLICY
POSITION PRECEDING PRECISION PRESERVE PREPARE PREPARED PRIMARY
PRIOR PRIVILEGES PROCEDURAL PROCEDURE PROCEDURES PROGRAM PUBLICATION
@@ -755,12 +766,13 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
RESET RESTART RESTRICT RETURN RETURNING RETURNS REVOKE RIGHT ROLE ROLLBACK ROLLUP
ROUTINE ROUTINES ROW ROWS RULE
- SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SELECT
+ SAVEPOINT SCALAR SCHEMA SCHEMAS SCROLL SEARCH SECOND_P SECURITY SEEK SELECT
SEQUENCE SEQUENCES
+
SERIALIZABLE SERVER SESSION SESSION_USER SET SETS SETOF SHARE SHOW
SIMILAR SIMPLE SKIP SMALLINT SNAPSHOT SOME SQL_P STABLE STANDALONE_P
START STATEMENT STATISTICS STDIN STDOUT STORAGE STORED STRICT_P STRIP_P
- SUBSCRIPTION SUBSTRING SUPPORT SYMMETRIC SYSID SYSTEM_P SYSTEM_USER
+ SUBSCRIPTION SUBSET SUBSTRING SUPPORT SYMMETRIC SYSID SYSTEM_P SYSTEM_USER
TABLE TABLES TABLESAMPLE TABLESPACE TEMP TEMPLATE TEMPORARY TEXT_P THEN
TIES TIME TIMESTAMP TO TRAILING TRANSACTION TRANSFORM
@@ -853,6 +865,7 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query);
*/
%nonassoc UNBOUNDED /* ideally would have same precedence as IDENT */
%nonassoc IDENT PARTITION RANGE ROWS GROUPS PRECEDING FOLLOWING CUBE ROLLUP
+%nonassoc MEASURES AFTER INITIAL SEEK PATTERN_P
%left Op OPERATOR /* multi-character ops and user-defined operators */
%left '+' '-'
%left '*' '/' '%'
@@ -15857,7 +15870,8 @@ over_clause: OVER window_specification
;
window_specification: '(' opt_existing_window_name opt_partition_clause
- opt_sort_clause opt_frame_clause ')'
+ opt_sort_clause opt_row_pattern_measures opt_frame_clause
+ opt_row_pattern_common_syntax ')'
{
WindowDef *n = makeNode(WindowDef);
@@ -15865,10 +15879,12 @@ window_specification: '(' opt_existing_window_name opt_partition_clause
n->refname = $2;
n->partitionClause = $3;
n->orderClause = $4;
+ n->rowPatternMeasures = $5;
/* copy relevant fields of opt_frame_clause */
- n->frameOptions = $5->frameOptions;
- n->startOffset = $5->startOffset;
- n->endOffset = $5->endOffset;
+ n->frameOptions = $6->frameOptions;
+ n->startOffset = $6->startOffset;
+ n->endOffset = $6->endOffset;
+ n->rpCommonSyntax = $7;
n->location = @1;
$$ = n;
}
@@ -15892,6 +15908,31 @@ opt_partition_clause: PARTITION BY expr_list { $$ = $3; }
| /*EMPTY*/ { $$ = NIL; }
;
+/*
+ * ROW PATTERN_P MEASURES
+ */
+opt_row_pattern_measures: MEASURES row_pattern_measure_list { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_measure_list:
+ row_pattern_measure_item
+ { $$ = list_make1($1); }
+ | row_pattern_measure_list ',' row_pattern_measure_item
+ { $$ = lappend($1, $3); }
+ ;
+
+row_pattern_measure_item:
+ a_expr AS ColLabel
+ {
+ $$ = makeNode(ResTarget);
+ $$->name = $3;
+ $$->indirection = NIL;
+ $$->val = (Node *) $1;
+ $$->location = @1;
+ }
+ ;
+
/*
* For frame clauses, we return a WindowDef, but only some fields are used:
* frameOptions, startOffset, and endOffset.
@@ -16051,6 +16092,143 @@ opt_window_exclusion_clause:
| /*EMPTY*/ { $$ = 0; }
;
+opt_row_pattern_common_syntax:
+opt_row_pattern_skip_to opt_row_pattern_initial_or_seek
+ PATTERN_P '(' row_pattern ')'
+ opt_row_pattern_subset_clause
+ DEFINE row_pattern_definition_list
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = $1->rpSkipTo;
+ n->rpSkipVariable = $1->rpSkipVariable;
+ n->initial = $2;
+ n->rpPatterns = $5;
+ n->rpSubsetClause = $7;
+ n->rpDefs = $9;
+ $$ = n;
+ }
+ | /*EMPTY*/ { $$ = NULL; }
+ ;
+
+opt_row_pattern_skip_to:
+ AFTER MATCH SKIP TO NEXT ROW
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_NEXT_ROW;
+ n->rpSkipVariable = NULL;
+ $$ = n;
+ }
+ | AFTER MATCH SKIP PAST LAST_P ROW
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_PAST_LAST_ROW;
+ n->rpSkipVariable = NULL;
+ $$ = n;
+ }
+ | AFTER MATCH SKIP TO first_or_last ColId
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = $5;
+ n->rpSkipVariable = $6;
+ $$ = n;
+ }
+/*
+ | AFTER MATCH SKIP TO LAST_P ColId %prec LAST_P
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_LAST_VARIABLE;
+ n->rpSkipVariable = $6;
+ $$ = n;
+ }
+ | AFTER MATCH SKIP TO ColId
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ n->rpSkipTo = ST_VARIABLE;
+ n->rpSkipVariable = $5;
+ $$ = n;
+ }
+*/
+ | /*EMPTY*/
+ {
+ RPCommonSyntax *n = makeNode(RPCommonSyntax);
+ /* temporary set default to ST_NEXT_ROW */
+ n->rpSkipTo = ST_PAST_LAST_ROW;
+ n->rpSkipVariable = NULL;
+ $$ = n;
+ }
+ ;
+
+first_or_last:
+ FIRST_P { $$ = ST_FIRST_VARIABLE; }
+ | LAST_P { $$ = ST_LAST_VARIABLE; }
+ ;
+
+opt_row_pattern_initial_or_seek:
+ INITIAL { $$ = true; }
+ | SEEK
+ {
+ ereport(ERROR,
+ (errcode(ERRCODE_SYNTAX_ERROR),
+ errmsg("SEEK is not supported"),
+ errhint("Use INITIAL."),
+ parser_errposition(@1)));
+ }
+ | /*EMPTY*/ { $$ = true; }
+ ;
+
+row_pattern:
+ row_pattern_term { $$ = list_make1($1); }
+ | row_pattern row_pattern_term { $$ = lappend($1, $2); }
+ ;
+
+row_pattern_term:
+ ColId { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "", (Node *)makeString($1), NULL, @1); }
+ | ColId '*' { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "*", (Node *)makeString($1), NULL, @1); }
+ | ColId '+' { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "+", (Node *)makeString($1), NULL, @1); }
+ | ColId '?' { $$ = (Node *) makeSimpleA_Expr(AEXPR_OP, "?", (Node *)makeString($1), NULL, @1); }
+ ;
+
+opt_row_pattern_subset_clause:
+ SUBSET row_pattern_subset_list { $$ = $2; }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_subset_list:
+ row_pattern_subset_item { $$ = list_make1($1); }
+ | row_pattern_subset_list ',' row_pattern_subset_item { $$ = lappend($1, $3); }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_subset_item: ColId '=' '(' row_pattern_subset_rhs ')'
+ {
+ RPSubsetItem *n = makeNode(RPSubsetItem);
+ n->name = $1;
+ n->rhsVariable = $4;
+ $$ = n;
+ }
+ ;
+
+row_pattern_subset_rhs:
+ ColId { $$ = list_make1(makeStringConst($1, @1)); }
+ | row_pattern_subset_rhs ',' ColId { $$ = lappend($1, makeStringConst($3, @1)); }
+ | /*EMPTY*/ { $$ = NIL; }
+ ;
+
+row_pattern_definition_list:
+ row_pattern_definition { $$ = list_make1($1); }
+ | row_pattern_definition_list ',' row_pattern_definition { $$ = lappend($1, $3); }
+ ;
+
+row_pattern_definition:
+ ColId AS a_expr
+ {
+ $$ = makeNode(ResTarget);
+ $$->name = $1;
+ $$->indirection = NIL;
+ $$->val = (Node *) $3;
+ $$->location = @1;
+ }
+ ;
/*
* Supporting nonterminals for expressions.
@@ -17146,6 +17324,7 @@ unreserved_keyword:
| INDEXES
| INHERIT
| INHERITS
+ | INITIAL
| INLINE_P
| INPUT_P
| INSENSITIVE
@@ -17173,6 +17352,7 @@ unreserved_keyword:
| MATCHED
| MATERIALIZED
| MAXVALUE
+ | MEASURES
| MERGE
| METHOD
| MINUTE_P
@@ -17215,6 +17395,9 @@ unreserved_keyword:
| PARTITION
| PASSING
| PASSWORD
+ | PAST
+ | PATTERN_P
+ | PERMUTE
| PLANS
| POLICY
| PRECEDING
@@ -17265,6 +17448,7 @@ unreserved_keyword:
| SEARCH
| SECOND_P
| SECURITY
+ | SEEK
| SEQUENCE
| SEQUENCES
| SERIALIZABLE
@@ -17290,6 +17474,7 @@ unreserved_keyword:
| STRICT_P
| STRIP_P
| SUBSCRIPTION
+ | SUBSET
| SUPPORT
| SYSID
| SYSTEM_P
@@ -17477,6 +17662,7 @@ reserved_keyword:
| CURRENT_USER
| DEFAULT
| DEFERRABLE
+ | DEFINE
| DESC
| DISTINCT
| DO
@@ -17639,6 +17825,7 @@ bare_label_keyword:
| DEFAULTS
| DEFERRABLE
| DEFERRED
+ | DEFINE
| DEFINER
| DELETE_P
| DELIMITER
@@ -17714,6 +17901,7 @@ bare_label_keyword:
| INDEXES
| INHERIT
| INHERITS
+ | INITIAL
| INITIALLY
| INLINE_P
| INNER_P
@@ -17763,6 +17951,7 @@ bare_label_keyword:
| MATCHED
| MATERIALIZED
| MAXVALUE
+ | MEASURES
| MERGE
| METHOD
| MINVALUE
@@ -17816,6 +18005,9 @@ bare_label_keyword:
| PARTITION
| PASSING
| PASSWORD
+ | PAST
+ | PATTERN_P
+ | PERMUTE
| PLACING
| PLANS
| POLICY
@@ -17872,6 +18064,7 @@ bare_label_keyword:
| SCROLL
| SEARCH
| SECURITY
+ | SEEK
| SELECT
| SEQUENCE
| SEQUENCES
@@ -17903,6 +18096,7 @@ bare_label_keyword:
| STRICT_P
| STRIP_P
| SUBSCRIPTION
+ | SUBSET
| SUBSTRING
| SUPPORT
| SYMMETRIC
diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h
index fef4c714b8..657651df1d 100644
--- a/src/include/nodes/parsenodes.h
+++ b/src/include/nodes/parsenodes.h
@@ -547,6 +547,44 @@ typedef struct SortBy
int location; /* operator location, or -1 if none/unknown */
} SortBy;
+/*
+ * AFTER MATCH row pattern skip to types in row pattern common syntax
+ */
+typedef enum RPSkipTo
+{
+ ST_NONE, /* AFTER MATCH omitted */
+ ST_NEXT_ROW, /* SKIP TO NEXT ROW */
+ ST_PAST_LAST_ROW, /* SKIP TO PAST LAST ROW */
+ ST_FIRST_VARIABLE, /* SKIP TO FIRST variable name */
+ ST_LAST_VARIABLE, /* SKIP TO LAST variable name */
+ ST_VARIABLE /* SKIP TO variable name */
+} RPSkipTo;
+
+/*
+ * Row Pattern SUBSET clause item
+ */
+typedef struct RPSubsetItem
+{
+ NodeTag type;
+ char *name; /* Row Pattern SUBSET clause variable name */
+ List *rhsVariable; /* Row Pattern SUBSET rhs variables (list of char *string) */
+} RPSubsetItem;
+
+/*
+ * RowPatternCommonSyntax - raw representation of row pattern common syntax
+ *
+ */
+typedef struct RPCommonSyntax
+{
+ NodeTag type;
+ RPSkipTo rpSkipTo; /* Row Pattern AFTER MATCH SKIP type */
+ char *rpSkipVariable; /* Row Pattern Skip To variable name, if any */
+ bool initial; /* true if <row pattern initial or seek> is initial */
+ List *rpPatterns; /* PATTERN variables (list of A_Expr) */
+ List *rpSubsetClause; /* row pattern subset clause (list of RPSubsetItem), if any */
+ List *rpDefs; /* row pattern definitions clause (list of ResTarget) */
+} RPCommonSyntax;
+
/*
* WindowDef - raw representation of WINDOW and OVER clauses
*
@@ -562,6 +600,8 @@ typedef struct WindowDef
char *refname; /* referenced window name, if any */
List *partitionClause; /* PARTITION BY expression list */
List *orderClause; /* ORDER BY (list of SortBy) */
+ List *rowPatternMeasures; /* row pattern measures (list of ResTarget) */
+ RPCommonSyntax *rpCommonSyntax; /* row pattern common syntax */
int frameOptions; /* frame_clause options, see below */
Node *startOffset; /* expression for starting bound, if any */
Node *endOffset; /* expression for ending bound, if any */
@@ -1483,6 +1523,11 @@ typedef struct GroupingSet
* the orderClause might or might not be copied (see copiedOrder); the framing
* options are never copied, per spec.
*
+ * "defineClause" is Row Pattern Recognition DEFINE clause (list of
+ * TargetEntry). TargetEntry.resname represents row pattern definition
+ * variable name. "patternVariable" and "patternRegexp" represents PATTERN
+ * clause.
+ *
* The information relevant for the query jumbling is the partition clause
* type and its bounds.
*/
@@ -1514,6 +1559,17 @@ typedef struct WindowClause
Index winref; /* ID referenced by window functions */
/* did we copy orderClause from refname? */
bool copiedOrder pg_node_attr(query_jumble_ignore);
+ /* Row Pattern AFTER MACH SKIP clause */
+ RPSkipTo rpSkipTo; /* Row Pattern Skip To type */
+ bool initial; /* true if <row pattern initial or seek> is initial */
+ /* Row Pattern DEFINE clause (list of TargetEntry) */
+ List *defineClause;
+ /* Row Pattern DEFINE variable initial names (list of String) */
+ List *defineInitial;
+ /* Row Pattern PATTERN variable name (list of String) */
+ List *patternVariable;
+ /* Row Pattern PATTERN regular expression quantifier ('+' or ''. list of String) */
+ List *patternRegexp;
} WindowClause;
/*
diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h
index 5984dcfa4b..2804333b53 100644
--- a/src/include/parser/kwlist.h
+++ b/src/include/parser/kwlist.h
@@ -128,6 +128,7 @@ PG_KEYWORD("default", DEFAULT, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("defaults", DEFAULTS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("deferrable", DEFERRABLE, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("deferred", DEFERRED, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("define", DEFINE, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("definer", DEFINER, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("delete", DELETE_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("delimiter", DELIMITER, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -212,6 +213,7 @@ PG_KEYWORD("index", INDEX, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("indexes", INDEXES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherit", INHERIT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inherits", INHERITS, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("initial", INITIAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("initially", INITIALLY, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inline", INLINE_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("inner", INNER_P, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL)
@@ -265,6 +267,7 @@ PG_KEYWORD("match", MATCH, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("matched", MATCHED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("materialized", MATERIALIZED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("maxvalue", MAXVALUE, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("measures", MEASURES, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("merge", MERGE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("method", METHOD, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("minute", MINUTE_P, UNRESERVED_KEYWORD, AS_LABEL)
@@ -326,6 +329,9 @@ PG_KEYWORD("partial", PARTIAL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("partition", PARTITION, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("passing", PASSING, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("password", PASSWORD, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("past", PAST, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("pattern", PATTERN_P, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("permute", PERMUTE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("placing", PLACING, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("plans", PLANS, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("policy", POLICY, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -385,6 +391,7 @@ PG_KEYWORD("scroll", SCROLL, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("search", SEARCH, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("second", SECOND_P, UNRESERVED_KEYWORD, AS_LABEL)
PG_KEYWORD("security", SECURITY, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("seek", SEEK, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("select", SELECT, RESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("sequence", SEQUENCE, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("sequences", SEQUENCES, UNRESERVED_KEYWORD, BARE_LABEL)
@@ -416,6 +423,7 @@ PG_KEYWORD("stored", STORED, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("strict", STRICT_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("strip", STRIP_P, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("subscription", SUBSCRIPTION, UNRESERVED_KEYWORD, BARE_LABEL)
+PG_KEYWORD("subset", SUBSET, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("substring", SUBSTRING, COL_NAME_KEYWORD, BARE_LABEL)
PG_KEYWORD("support", SUPPORT, UNRESERVED_KEYWORD, BARE_LABEL)
PG_KEYWORD("symmetric", SYMMETRIC, RESERVED_KEYWORD, BARE_LABEL)
diff --git a/src/include/parser/parse_node.h b/src/include/parser/parse_node.h
index f589112d5e..6640090910 100644
--- a/src/include/parser/parse_node.h
+++ b/src/include/parser/parse_node.h
@@ -51,6 +51,7 @@ typedef enum ParseExprKind
EXPR_KIND_WINDOW_FRAME_RANGE, /* window frame clause with RANGE */
EXPR_KIND_WINDOW_FRAME_ROWS, /* window frame clause with ROWS */
EXPR_KIND_WINDOW_FRAME_GROUPS, /* window frame clause with GROUPS */
+ EXPR_KIND_RPR_DEFINE, /* DEFINE */
EXPR_KIND_SELECT_TARGET, /* SELECT target list item */
EXPR_KIND_INSERT_TARGET, /* INSERT target list item */
EXPR_KIND_UPDATE_SOURCE, /* UPDATE assignment source item */
--
2.25.1
----Next_Part(Fri_Sep_22_14_16_40_2023_530)--
Content-Type: Text/X-Patch; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="v7-0002-Row-pattern-recognition-patch-parse-analysis.patch"
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Pasword expiration warning
@ 2026-02-04 20:56 Jacob Champion <[email protected]>
2026-02-04 21:00 ` Re: Pasword expiration warning Nathan Bossart <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Jacob Champion @ 2026-02-04 20:56 UTC (permalink / raw)
To: Nathan Bossart <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; Peter Eisentraut <[email protected]>; Gilles Darold <[email protected]>; Japin Li <[email protected]>; Yuefei Shi <[email protected]>; songjinzhou <[email protected]>; pgsql-hackers; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; liu xiaohui <[email protected]>; Steven Niu <[email protected]>; Euler Taveira <[email protected]>
On Wed, Feb 4, 2026 at 12:22 PM Nathan Bossart <[email protected]> wrote:
> Fair enough. The main problem we need to figure out for that approach is
> how to handle plurals without "(s)".
Euler pointed out upthread that the _() call should be replaced with
ngettext(). Or is there a reason you need the more complicated
errmsg_plural() machinery?
--Jacob
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: Pasword expiration warning
2026-02-04 20:56 Re: Pasword expiration warning Jacob Champion <[email protected]>
@ 2026-02-04 21:00 ` Nathan Bossart <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Nathan Bossart @ 2026-02-04 21:00 UTC (permalink / raw)
To: Jacob Champion <[email protected]>; +Cc: Zsolt Parragi <[email protected]>; Peter Eisentraut <[email protected]>; Gilles Darold <[email protected]>; Japin Li <[email protected]>; Yuefei Shi <[email protected]>; songjinzhou <[email protected]>; pgsql-hackers; Andrew Dunstan <[email protected]>; Tom Lane <[email protected]>; liu xiaohui <[email protected]>; Steven Niu <[email protected]>; Euler Taveira <[email protected]>
On Wed, Feb 04, 2026 at 12:56:18PM -0800, Jacob Champion wrote:
> On Wed, Feb 4, 2026 at 12:22 PM Nathan Bossart <[email protected]> wrote:
>> Fair enough. The main problem we need to figure out for that approach is
>> how to handle plurals without "(s)".
>
> Euler pointed out upthread that the _() call should be replaced with
> ngettext(). Or is there a reason you need the more complicated
> errmsg_plural() machinery?
Ah, that's what I was looking for, thanks. I missed that earlier message.
--
nathan
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-02-04 21:00 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-02-27 18:58 [PATCH] Refactor the fsync mechanism to support future SMGR implementations. Shawn Debnath <[email protected]>
2023-09-22 04:53 [PATCH v7 1/7] Row pattern recognition patch for raw parser. Tatsuo Ishii <[email protected]>
2026-02-04 20:56 Re: Pasword expiration warning Jacob Champion <[email protected]>
2026-02-04 21:00 ` Re: Pasword expiration warning Nathan Bossart <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox