public inbox for [email protected]
help / color / mirror / Atom feedFrom: Haozhou Wang <[email protected]>
To: [email protected]
Cc: [email protected]
Cc: [email protected]
Subject: Re: Control your disk usage in PG: Introduction to Disk Quota Extension
Date: Thu, 22 Nov 2018 19:25:37 +0800
Message-ID: <CAL_NLpKf6HzgGzWriuSW2U3CEykEXA9Z5vyEqBW0Hk5btr6Liw@mail.gmail.com> (raw)
In-Reply-To: <CA+Tgmoa9m74nhU8LpPfsiW-WS2MvWkipCEn4VW40kFrMMkbhKw@mail.gmail.com>
References: <CAB0yre=RP_ho6Bq4cV23ELKxRcfhV2Yqrb1zHp0RfUPEWCnBRw@mail.gmail.com>
<[email protected]>
<CAB0yrenHV2iw4hQEnHGkTTpXZN8jMUJNtsCV4=BmWckjnn16Mg@mail.gmail.com>
<CAL_NLp+jLqn9P+hzy1vYSqoUuCsJ4AtTF4QYH5=950rB9S3dSw@mail.gmail.com>
<CA+Tgmoa9m74nhU8LpPfsiW-WS2MvWkipCEn4VW40kFrMMkbhKw@mail.gmail.com>
Thank you very much for your review.
We refactored our patch with new names and comments.
For ReadBufferExtended hook, yes, Readbuffer with P_NEW will then call
smgrextend.
But in smgrextend, we cannot get the oid of a relation, and it will take
some time to get the oid via smgrrelation.
We would like to add a hook just before the smgrextend to get the oid and
avoid use RelidByRelfilenode().
New patch is attached in the attachment.
Thank a lot!
Regards,
Haozhou
On Wed, Nov 21, 2018 at 10:48 PM Robert Haas <[email protected]> wrote:
> On Tue, Nov 20, 2018 at 2:20 AM Haozhou Wang <[email protected]> wrote:
> > We prepared a patch that includes the hook points. And such hook points
> are needed for disk quota extension.
> > There are two hooks.
> > One is SmgrStat_hook. It's used to perform ad-hoc logic in storage when
> doing smgr create/extend/truncate in general. As for disk quota extension,
> this hook is used to detect active tables(new created tables, tables
> extending new blocks, or tables being truncated)
> > The other is BufferExtendCheckPerms_hook. It's used to perform ad-hoc
> logic when buffer extend a new block. Since ReadBufferExtended is a hot
> function, we call this hook only when blockNum == P_NEW. As for disk quota
> extension, this hook is used to do query enforcement during the query is
> loading data.
> >
> > Any comments are appreciated.
>
> +1 for adding some hooks to support this kind of thing, but I think
> the names you've chosen are not very good. The hook name should
> describe the place from which it is called, not the purpose for which
> one imagines that it will be used, because somebody else might imagine
> another use. Both BufferExtendCheckPerms_hook_type and
> SmgrStat_hook_type are imagining that they know what the hook does -
> CheckPerms in the first case and Stat in the second case.
>
> For this particular purpose, I don't immediately see why you need a
> hook in both places. If ReadBuffer is called with P_NEW, aren't we
> guaranteed to end up in smgrextend()?
>
> --
> Robert Haas
> EnterpriseDB: http://www.enterprisedb.com
> The Enterprise PostgreSQL Company
>
Attachments:
[application/octet-stream] disk_quota_hooks_v2.patch (6.4K, ../CAL_NLpKf6HzgGzWriuSW2U3CEykEXA9Z5vyEqBW0Hk5btr6Liw@mail.gmail.com/3-disk_quota_hooks_v2.patch)
download | inline diff:
From 3d275c78b304b308d288bd227f6dcab45dc5f595 Mon Sep 17 00:00:00 2001
From: Hubert Zhang <[email protected]>
Date: Tue, 6 Nov 2018 06:51:22 +0000
Subject: [PATCH] Add hooks for diskquota extension.
Add ReadBufferExtend_hook() and smgr*_hook()
hook points to extend logic of storage
management.
Co-authored-by: Haozhou Wang <[email protected]>
Co-authored-by: Hubert Zhang <[email protected]>
Co-authored-by: Hao Wu <[email protected]>
---
src/backend/storage/buffer/bufmgr.c | 14 ++++++++++++++
src/backend/storage/smgr/smgr.c | 33 +++++++++++++++++++++++++++++++++
src/include/storage/bufmgr.h | 10 ++++++++++
src/include/storage/smgr.h | 18 ++++++++++++++++++
4 files changed, 75 insertions(+)
diff --git a/src/backend/storage/buffer/bufmgr.c b/src/backend/storage/buffer/bufmgr.c
index 01eabe5706..5499495506 100644
--- a/src/backend/storage/buffer/bufmgr.c
+++ b/src/backend/storage/buffer/bufmgr.c
@@ -104,6 +104,13 @@ typedef struct CkptTsStatus
int index;
} CkptTsStatus;
+/*
+ * Hook for plugins to check permissions when doing a buffer extend.
+ * One example is to check whether there is additional disk quota for
+ * the table to be inserted.
+ */
+ReadBufferExtended_hook_type ReadBufferExtended_hook = NULL;
+
/* GUC variables */
bool zero_damaged_pages = false;
int bgwriter_lru_maxpages = 100;
@@ -661,6 +668,13 @@ ReadBufferExtended(Relation reln, ForkNumber forkNum, BlockNumber blockNum,
* miss.
*/
pgstat_count_buffer_read(reln);
+
+ /* hook function for doing a buffer extend */
+ if (blockNum == P_NEW && ReadBufferExtended_hook)
+ {
+ (*ReadBufferExtended_hook)(reln, forkNum, blockNum, mode, strategy);
+ }
+
buf = ReadBuffer_common(reln->rd_smgr, reln->rd_rel->relpersistence,
forkNum, blockNum, mode, strategy, &hit);
if (hit)
diff --git a/src/backend/storage/smgr/smgr.c b/src/backend/storage/smgr/smgr.c
index 189342ef86..fa36a18e15 100644
--- a/src/backend/storage/smgr/smgr.c
+++ b/src/backend/storage/smgr/smgr.c
@@ -90,6 +90,16 @@ static const f_smgr smgrsw[] = {
static const int NSmgr = lengthof(smgrsw);
+/*
+ * Hook for plugins to extend smgr functions.
+ * for example, collect statistics from smgr functions
+ * via recording the active relfilenode information.
+ */
+smgrcreate_hook_type smgrcreate_hook = NULL;
+smgrextend_hook_type smgrextend_hook = NULL;
+smgrtruncate_hook_type smgrtruncate_hook = NULL;
+smgrdounlinkall_hook_type smgrdounlinkall_hook = NULL;
+
/*
* Each backend has a hashtable that stores all extant SMgrRelation objects.
@@ -397,6 +407,11 @@ smgrcreate(SMgrRelation reln, ForkNumber forknum, bool isRedo)
if (isRedo && reln->md_num_open_segs[forknum] > 0)
return;
+ if (smgrcreate_hook)
+ {
+ (*smgrcreate_hook)(reln, forknum, isRedo);
+ }
+
/*
* We may be using the target table space for the first time in this
* database, so create a per-database subdirectory if needed.
@@ -411,6 +426,7 @@ smgrcreate(SMgrRelation reln, ForkNumber forknum, bool isRedo)
isRedo);
smgrsw[reln->smgr_which].smgr_create(reln, forknum, isRedo);
+
}
/*
@@ -492,6 +508,11 @@ smgrdounlinkall(SMgrRelation *rels, int nrels, bool isRedo)
if (nrels == 0)
return;
+ if (smgrdounlinkall_hook)
+ {
+ (*smgrdounlinkall_hook)(rels, nrels, isRedo);
+ }
+
/*
* create an array which contains all relations to be dropped, and close
* each relation's forks at the smgr level while at it
@@ -615,8 +636,14 @@ void
smgrextend(SMgrRelation reln, ForkNumber forknum, BlockNumber blocknum,
char *buffer, bool skipFsync)
{
+ if (smgrextend_hook)
+ {
+ (*smgrextend_hook)(reln, forknum, blocknum, buffer, skipFsync);
+ }
+
smgrsw[reln->smgr_which].smgr_extend(reln, forknum, blocknum,
buffer, skipFsync);
+
}
/*
@@ -698,6 +725,11 @@ smgrnblocks(SMgrRelation reln, ForkNumber forknum)
void
smgrtruncate(SMgrRelation reln, ForkNumber forknum, BlockNumber nblocks)
{
+ if (smgrtruncate_hook)
+ {
+ (*smgrtruncate_hook)(reln, forknum, nblocks);
+ }
+
/*
* Get rid of any buffers for the about-to-be-deleted blocks. bufmgr will
* just drop them without bothering to write the contents.
@@ -720,6 +752,7 @@ smgrtruncate(SMgrRelation reln, ForkNumber forknum, BlockNumber nblocks)
* Do the truncation.
*/
smgrsw[reln->smgr_which].smgr_truncate(reln, forknum, nblocks);
+
}
/*
diff --git a/src/include/storage/bufmgr.h b/src/include/storage/bufmgr.h
index 3cce3906a0..f1dcc77bf7 100644
--- a/src/include/storage/bufmgr.h
+++ b/src/include/storage/bufmgr.h
@@ -159,6 +159,16 @@ extern PGDLLIMPORT int32 *LocalRefCount;
*/
#define BufferGetPage(buffer) ((Page)BufferGetBlock(buffer))
+/*
+ * Hook for plugins to add external logic when doing a buffer extend.
+ * One example is to check whether there is additional disk quota for
+ * the table to be inserted.
+ */
+typedef bool (*ReadBufferExtended_hook_type) (Relation reln,
+ ForkNumber forkNum, BlockNumber blockNum,
+ ReadBufferMode mode, BufferAccessStrategy strategy);
+extern PGDLLIMPORT ReadBufferExtended_hook_type ReadBufferExtended_hook;
+
/*
* prototypes for functions in bufmgr.c
*/
diff --git a/src/include/storage/smgr.h b/src/include/storage/smgr.h
index c843bbc969..d070b3d573 100644
--- a/src/include/storage/smgr.h
+++ b/src/include/storage/smgr.h
@@ -144,5 +144,23 @@ extern void RememberFsyncRequest(RelFileNode rnode, ForkNumber forknum,
extern void ForgetRelationFsyncRequests(RelFileNode rnode, ForkNumber forknum);
extern void ForgetDatabaseFsyncRequests(Oid dbid);
extern void DropRelationFiles(RelFileNode *delrels, int ndelrels, bool isRedo);
+/*
+ * Hook for plugins to extend smgr functions.
+ * for example, collect statistics from smgr functions
+ * via recording the active relfilenode information.
+ */
+typedef void (*smgrcreate_hook_type)(SMgrRelation reln, ForkNumber forknum,
+ bool isRedo);
+extern PGDLLIMPORT smgrcreate_hook_type smgrcreate_hook;
+typedef void (*smgrextend_hook_type)(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber blocknum,
+ char *buffer, bool skipFsync);
+extern PGDLLIMPORT smgrextend_hook_type smgrextend_hook;
+typedef void (*smgrtruncate_hook_type)(SMgrRelation reln, ForkNumber forknum,
+ BlockNumber nblocks);
+extern PGDLLIMPORT smgrtruncate_hook_type smgrtruncate_hook;
+typedef void (*smgrdounlinkall_hook_type)(SMgrRelation *rels, int nrels,
+ bool isRedo);
+extern PGDLLIMPORT smgrdounlinkall_hook_type smgrdounlinkall_hook;
#endif /* SMGR_H */
--
2.16.2
view thread (24+ messages) latest in thread
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected]
Subject: Re: Control your disk usage in PG: Introduction to Disk Quota Extension
In-Reply-To: <CAL_NLpKf6HzgGzWriuSW2U3CEykEXA9Z5vyEqBW0Hk5btr6Liw@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox