public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v24 03/10] Add default_toast_compression GUC 3+ messages / 3 participants [nested] [flat]
* [PATCH v24 03/10] Add default_toast_compression GUC @ 2021-01-30 03:37 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Justin Pryzby @ 2021-01-30 03:37 UTC (permalink / raw) --- src/backend/access/common/tupdesc.c | 2 +- src/backend/bootstrap/bootstrap.c | 3 +- src/backend/commands/amcmds.c | 143 +++++++++++++++++++++++++++- src/backend/commands/tablecmds.c | 4 +- src/backend/utils/init/postinit.c | 4 + src/backend/utils/misc/guc.c | 12 +++ src/include/access/amapi.h | 2 + src/include/access/compressamapi.h | 12 ++- 8 files changed, 176 insertions(+), 6 deletions(-) diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index ca26fab487..7afaea000b 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -668,7 +668,7 @@ TupleDescInitEntry(TupleDesc desc, att->attcollation = typeForm->typcollation; if (IsStorageCompressible(typeForm->typstorage)) - att->attcompression = DefaultCompressionOid; + att->attcompression = GetDefaultToastCompression(); else att->attcompression = InvalidOid; diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 9b451eaa71..ec3376cf8a 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -732,8 +732,9 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) - attrtypes[attnum]->attcompression = DefaultCompressionOid; + attrtypes[attnum]->attcompression = GetDefaultToastCompression(); else attrtypes[attnum]->attcompression = InvalidOid; diff --git a/src/backend/commands/amcmds.c b/src/backend/commands/amcmds.c index 3ad4a61739..1682afd2a4 100644 --- a/src/backend/commands/amcmds.c +++ b/src/backend/commands/amcmds.c @@ -13,8 +13,11 @@ */ #include "postgres.h" +#include "access/amapi.h" +#include "access/compressamapi.h" #include "access/htup_details.h" #include "access/table.h" +#include "access/xact.h" #include "catalog/catalog.h" #include "catalog/dependency.h" #include "catalog/indexing.h" @@ -27,13 +30,20 @@ #include "parser/parse_func.h" #include "utils/builtins.h" #include "utils/lsyscache.h" +#include "utils/guc.h" +#include "utils/inval.h" #include "utils/rel.h" #include "utils/syscache.h" - static Oid lookup_am_handler_func(List *handler_name, char amtype); static const char *get_am_type_string(char amtype); +/* Compile-time default */ +char *default_toast_compression = DEFAULT_TOAST_COMPRESSION; // maybe needs pg_dump support ? +/* Invalid means need to lookup the text value in the catalog */ +static Oid default_toast_compression_oid = InvalidOid; + +static void AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue); /* * CreateAccessMethod @@ -277,3 +287,134 @@ lookup_am_handler_func(List *handler_name, char amtype) return handlerOid; } + +/* check_hook: validate new default_toast_compression */ +bool +check_default_toast_compression(char **newval, void **extra, GucSource source) +{ + if (**newval == '\0') + { + GUC_check_errdetail("%s cannot be empty.", + "default_toast_compression"); + return false; + } + + if (strlen(*newval) >= NAMEDATALEN) + { + GUC_check_errdetail("%s is too long (maximum %d characters).", + "default_toast_compression", NAMEDATALEN - 1); + return false; + } + + /* + * If we aren't inside a transaction, or not connected to a database, we + * cannot do the catalog access necessary to verify the method. Must + * accept the value on faith. + */ + if (IsTransactionState() && MyDatabaseId != InvalidOid) + { + if (!OidIsValid(get_compression_am_oid(*newval, true))) + { + /* + * When source == PGC_S_TEST, don't throw a hard error for a + * nonexistent table access method, only a NOTICE. See comments in + * guc.h. + */ + if (source == PGC_S_TEST) + { + ereport(NOTICE, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("compression access method \"%s\" does not exist", + *newval))); + } + else + { + GUC_check_errdetail("Compression access method \"%s\" does not exist.", + *newval); + return false; + } + } + } + + return true; +} + +/* + * assign_default_toast_compression: GUC assign_hook for default_toast_compression + */ +void +assign_default_toast_compression(const char *newval, void *extra) +{ + /* + * Invalidate setting, forcing it to be looked up as needed. + * This avoids trying to do database access during GUC initialization, + * or outside a transaction. + */ + + default_toast_compression = NULL; +fprintf(stderr, "set to null\n"); +} + + +/* + * GetDefaultToastCompression -- get the OID of the current toast compression + * + * This exists to hide and optimize the use of the default_toast_compression + * GUC variable. + */ +Oid +GetDefaultToastCompression(void) +{ + /* Avoid catalog access during bootstrap, and for default compression */ + if (strcmp(default_toast_compression, DEFAULT_TOAST_COMPRESSION) == 0) + return PGLZ_COMPRESSION_AM_OID; + + /* Cannot call get_compression_am_oid this early */ + // if (IsBootstrapProcessingMode()) + // return PGLZ_COMPRESSION_AM_OID; + Assert(!IsBootstrapProcessingMode()); + + /* + * If cached value isn't valid, look up the current default value, caching + * the result + */ + if (!OidIsValid(default_toast_compression_oid)) + default_toast_compression_oid = + get_am_type_oid(default_toast_compression, AMTYPE_COMPRESSION, + false); + + return default_toast_compression_oid; +} + +/* + * InitializeAccessMethods: initialize module during InitPostgres. + * + * This is called after we are up enough to be able to do catalog lookups. + */ +void +InitializeAccessMethods(void) +{ + if (IsBootstrapProcessingMode()) + return; + + /* + * In normal mode, arrange for a callback on any syscache invalidation + * of pg_am rows. + */ + CacheRegisterSyscacheCallback(AMOID, + AccessMethodCallback, + (Datum) 0); + /* Force cached default access method to be recomputed on next use */ + // default_toast_compression_oid = InvalidOid; +} + +/* + * AccessMethodCallback + * Syscache inval callback function + */ +static void +AccessMethodCallback(Datum arg, int cacheid, uint32 hashvalue) +{ + /* Force look up of compression oid on next use */ + default_toast_compression_oid = false; +} diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index dd81d5bf4e..72ba017814 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -11925,7 +11925,7 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, if (!IsStorageCompressible(tform->typstorage)) attTup->attcompression = InvalidOid; else if (!OidIsValid(attTup->attcompression)) - attTup->attcompression = DefaultCompressionOid; + attTup->attcompression = GetDefaultToastCompression(); } else attTup->attcompression = InvalidOid; @@ -17762,7 +17762,7 @@ GetAttributeCompression(Form_pg_attribute att, char *compression) /* fallback to default compression if it's not specified */ if (compression == NULL) - return DefaultCompressionOid; + return GetDefaultToastCompression(); amoid = get_compression_am_oid(compression, false); diff --git a/src/backend/utils/init/postinit.c b/src/backend/utils/init/postinit.c index e5965bc517..6a02bbd377 100644 --- a/src/backend/utils/init/postinit.c +++ b/src/backend/utils/init/postinit.c @@ -25,6 +25,7 @@ #include "access/session.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/amapi.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -1060,6 +1061,9 @@ InitPostgres(const char *in_dbname, Oid dboid, const char *username, /* set default namespace search path */ InitializeSearchPath(); + /* set callback for changes to pg_am */ + InitializeAccessMethods(); + /* initialize client encoding */ InitializeClientEncoding(); diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index eafdb1118e..3a2b33fcd1 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -30,6 +30,7 @@ #include <unistd.h> #include "access/commit_ts.h" +#include "access/compressamapi.h" #include "access/gin.h" #include "access/rmgr.h" #include "access/tableam.h" @@ -3915,6 +3916,17 @@ static struct config_string ConfigureNamesString[] = check_default_table_access_method, NULL, NULL }, + { + {"default_toast_compression", PGC_USERSET, CLIENT_CONN_STATEMENT, + gettext_noop("Sets the default compression for new columns."), + NULL, + GUC_IS_NAME + }, + &default_toast_compression, + DEFAULT_TOAST_COMPRESSION, + check_default_toast_compression, assign_default_toast_compression, NULL, NULL + }, + { {"default_tablespace", PGC_USERSET, CLIENT_CONN_STATEMENT, gettext_noop("Sets the default tablespace to create tables and indexes in."), diff --git a/src/include/access/amapi.h b/src/include/access/amapi.h index d357ebb559..1513cafcf4 100644 --- a/src/include/access/amapi.h +++ b/src/include/access/amapi.h @@ -287,4 +287,6 @@ typedef struct IndexAmRoutine extern IndexAmRoutine *GetIndexAmRoutine(Oid amhandler); extern IndexAmRoutine *GetIndexAmRoutineByAmId(Oid amoid, bool noerror); +void InitializeAccessMethods(void); + #endif /* AMAPI_H */ diff --git a/src/include/access/compressamapi.h b/src/include/access/compressamapi.h index 5a8e23d926..d75a8e9df2 100644 --- a/src/include/access/compressamapi.h +++ b/src/include/access/compressamapi.h @@ -17,6 +17,7 @@ #include "catalog/pg_am_d.h" #include "nodes/nodes.h" +#include "utils/guc.h" /* * Built-in compression method-id. The toast compression header will store @@ -29,8 +30,17 @@ typedef enum CompressionId LZ4_COMPRESSION_ID = 1 } CompressionId; -/* Use default compression method if it is not specified. */ +/* Default compression method if not specified. */ +#define DEFAULT_TOAST_COMPRESSION "pglz" #define DefaultCompressionOid PGLZ_COMPRESSION_AM_OID + +/* GUC */ +extern char *default_toast_compression; +extern void assign_default_toast_compression(const char *newval, void *extra); +extern bool check_default_toast_compression(char **newval, void **extra, GucSource source); + +extern Oid GetDefaultToastCompression(void); + #define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ (storage) != TYPSTORAGE_EXTERNAL) /* compression handler routines */ -- 2.17.0 --m51xatjYGsM+13rf Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v24-0004-default-to-with-lz4.patch" ^ permalink raw reply [nested|flat] 3+ messages in thread
* Build problem with square brackets in build path @ 2023-04-28 11:09 Nikolay Shaplov <[email protected]> 2023-04-28 18:06 ` Re: Build problem with square brackets in build path Tom Lane <[email protected]> 0 siblings, 1 reply; 3+ messages in thread From: Nikolay Shaplov @ 2023-04-28 11:09 UTC (permalink / raw) To: pgsql-hackers I am not sure it is really a bug, but nevertheless If you do mkdir [source] git clone git://git.postgresql.org/git/postgresql.git [source] mkdir build; cd build ../\[source\]/configure make you will get make[1]: *** No rule to make target 'generated-headers'. Stop. If there are no "[]" in the path to the source, everything is OK. It would be OK for me, if it still does not work. But I would appreciate at least proper error message there (or at configure step), this would save time, for figuring out, why it suddenly stopped working. -- Nikolay Shaplov aka Nataraj Fuzzing Engineer at Postgres Professional Matrix IM: @dhyan:nataraj.su Attachments: [application/pgp-signature] signature.asc (488B, ../../4793552.atqmVPU5g4@thinkpad-pgpro/2-signature.asc) download ^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Build problem with square brackets in build path 2023-04-28 11:09 Build problem with square brackets in build path Nikolay Shaplov <[email protected]> @ 2023-04-28 18:06 ` Tom Lane <[email protected]> 0 siblings, 0 replies; 3+ messages in thread From: Tom Lane @ 2023-04-28 18:06 UTC (permalink / raw) To: Nikolay Shaplov <[email protected]>; +Cc: pgsql-hackers Nikolay Shaplov <[email protected]> writes: > If you do > mkdir [source] > git clone git://git.postgresql.org/git/postgresql.git [source] > mkdir build; cd build > ../\[source\]/configure > make > you will get > make[1]: *** No rule to make target 'generated-headers'. Stop. > If there are no "[]" in the path to the source, everything is OK. It's generally quite unwise to use shell meta-characters in file or directory names. I give you one example: $ ls ../[source] COPYRIGHT README.git contrib/ GNUmakefile.in aclocal.m4 doc/ HISTORY config/ meson.build Makefile configure* meson_options.txt README configure.ac src/ $ ls ../[source]/*.ac ls: ../[source]/*.ac: No such file or directory This is expected behavior (I leave it as an exercise for the student to figure out why). While it might be possible to make the Postgres build scripts proof against funny characters in the build paths, the effort required would be far out of proportion to the value. Not least because manual operations in such a file tree would misbehave often enough to convince you to change, even if the scripts were all water-tight. regards, tom lane ^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2023-04-28 18:06 UTC | newest] Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-01-30 03:37 [PATCH v24 03/10] Add default_toast_compression GUC Justin Pryzby <[email protected]> 2023-04-28 11:09 Build problem with square brackets in build path Nikolay Shaplov <[email protected]> 2023-04-28 18:06 ` Re: Build problem with square brackets in build path Tom Lane <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox