public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 09/10] Add zstd compression levels
4+ messages / 3 participants
[nested] [flat]
* [PATCH 09/10] Add zstd compression levels
@ 2021-03-14 22:12 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Justin Pryzby @ 2021-03-14 22:12 UTC (permalink / raw)
---
src/backend/access/transam/xlog.c | 6 +++++-
src/backend/access/transam/xloginsert.c | 15 +++++++++++++--
src/backend/access/transam/xlogreader.c | 12 ++++++++++++
src/backend/utils/misc/guc.c | 2 +-
src/include/access/xlog_internal.h | 4 ++++
5 files changed, 35 insertions(+), 4 deletions(-)
diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c
index 92023de9f5..b14c7c5929 100644
--- a/src/backend/access/transam/xlog.c
+++ b/src/backend/access/transam/xlog.c
@@ -99,7 +99,7 @@ bool EnableHotStandby = false;
bool fullPageWrites = true;
bool wal_log_hints = false;
bool wal_compression = false;
-int wal_compression_method = WAL_COMPRESSION_ZSTD;
+int wal_compression_method = WAL_COMPRESSION_ZSTD_FAST_10;
char *wal_consistency_checking_string = NULL;
bool *wal_consistency_checking = NULL;
bool wal_init_zero = true;
@@ -192,6 +192,10 @@ const struct config_enum_entry wal_compression_options[] = {
#endif
#ifdef USE_ZSTD
{"zstd", WAL_COMPRESSION_ZSTD, false},
+ {"zstd-1", WAL_COMPRESSION_ZSTD_1, false},
+ {"zstd-fast-10", WAL_COMPRESSION_ZSTD_FAST_10, false},
+ {"zstd-fast-20", WAL_COMPRESSION_ZSTD_FAST_20, false},
+ {"zstd-fast-40", WAL_COMPRESSION_ZSTD_FAST_40, false},
#endif
{NULL, 0, false}
};
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 4591e476c6..4f11f96373 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -915,11 +915,22 @@ XLogCompressBackupBlock(char *page, uint16 hole_offset, uint16 hole_length,
#ifdef USE_ZSTD
case WAL_COMPRESSION_ZSTD:
- len = ZSTD_compress(dest, COMPRESS_BUFSIZE, source, orig_len,
- ZSTD_CLEVEL_DEFAULT);
+ case WAL_COMPRESSION_ZSTD_1:
+ case WAL_COMPRESSION_ZSTD_FAST_10:
+ case WAL_COMPRESSION_ZSTD_FAST_20:
+ case WAL_COMPRESSION_ZSTD_FAST_40:
+ {
+ int level = compression == WAL_COMPRESSION_ZSTD_1 ? 1 :
+ compression == WAL_COMPRESSION_ZSTD_FAST_10 ? -10 :
+ compression == WAL_COMPRESSION_ZSTD_FAST_20 ? -20 :
+ compression == WAL_COMPRESSION_ZSTD_FAST_40 ? -40 :
+ ZSTD_CLEVEL_DEFAULT;
+
+ len = ZSTD_compress(dest, COMPRESS_BUFSIZE, source, orig_len, level);
if (ZSTD_isError(len))
len = -1;
break;
+ }
#endif
default:
diff --git a/src/backend/access/transam/xlogreader.c b/src/backend/access/transam/xlogreader.c
index 0f9d522087..0de61d3073 100644
--- a/src/backend/access/transam/xlogreader.c
+++ b/src/backend/access/transam/xlogreader.c
@@ -1554,6 +1554,10 @@ struct walcompression walmethods[] = {
{"zlib", WAL_COMPRESSION_ZLIB},
{"lz4", WAL_COMPRESSION_LZ4},
{"zstd", WAL_COMPRESSION_ZSTD},
+ {"zstd-1", WAL_COMPRESSION_ZSTD},
+ {"zstd-fast-10",WAL_COMPRESSION_ZSTD},
+ {"zstd-fast-20",WAL_COMPRESSION_ZSTD},
+ {"zstd-fast-40",WAL_COMPRESSION_ZSTD},
};
/*
@@ -1628,6 +1632,14 @@ RestoreBlockImage(XLogReaderState *record, uint8 block_id, char *page)
#ifdef USE_ZSTD
case WAL_COMPRESSION_ZSTD:
+ /*
+ * There aren't actually written into the header - decompression is the
+ * same.
+ * case WAL_COMPRESSION_ZSTD_1:
+ * case WAL_COMPRESSION_ZSTD_FAST_10:
+ * case WAL_COMPRESSION_ZSTD_FAST_20:
+ * case WAL_COMPRESSION_ZSTD_FAST_40:
+ */
decomp_result = ZSTD_decompress(tmp.data, BLCKSZ-bkpb->hole_length,
ptr, bkpb->bimg_len);
// XXX: ZSTD_getErrorName
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index 8031e027aa..667fc4c0c1 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -4728,7 +4728,7 @@ static struct config_enum ConfigureNamesEnum[] =
NULL
},
&wal_compression_method,
- WAL_COMPRESSION_ZSTD, wal_compression_options,
+ WAL_COMPRESSION_ZSTD_FAST_10, wal_compression_options,
NULL, NULL, NULL
},
diff --git a/src/include/access/xlog_internal.h b/src/include/access/xlog_internal.h
index fa8146645d..a435b1a654 100644
--- a/src/include/access/xlog_internal.h
+++ b/src/include/access/xlog_internal.h
@@ -338,6 +338,10 @@ typedef enum WalCompression
WAL_COMPRESSION_ZLIB,
WAL_COMPRESSION_LZ4,
WAL_COMPRESSION_ZSTD,
+ WAL_COMPRESSION_ZSTD_1,
+ WAL_COMPRESSION_ZSTD_FAST_10, /* level = -10 */
+ WAL_COMPRESSION_ZSTD_FAST_20, /* level = -20 */
+ WAL_COMPRESSION_ZSTD_FAST_40, /* level = -40 */
} WalCompression;
extern const char *wal_compression_name(WalCompression compression);
--
2.17.0
--jozmn01XJZjDjM3N--
^ permalink raw reply [nested|flat] 4+ messages in thread
* FmgrInfo allocation patterns (and PL handling as staged programming)
@ 2025-04-06 17:18 Chapman Flack <[email protected]>
2025-04-06 17:33 ` Re: FmgrInfo allocation patterns (and PL handling as staged programming) Tom Lane <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Chapman Flack @ 2025-04-06 17:18 UTC (permalink / raw)
To: pgsql-hackers
Hi hackers,
The way the core code allocates FmgrInfo structures has a pleasing property
(at least in the parts of the code I have read and the cases I've tested)
that the docs don't seem to emphasize.
To wit, given a query like
SELECT hello(n), hello(x)
FROM (VALUES
(1::int4, 1.0::float4),
(2, 2.0 ),
(3, 3.0 )) AS t(n,x);
the core code allocates one FmgrInfo for each of the two uses.
If the PL handler has a useful way to specialize the polymorphic-typed
function to int4 in the one case and float4 in the other, each can be
stashed in its respective flinfo->fn_extra and the specialized versions
used as expected for the duration of the query.
Likewise, the trigger manager prepares for a query by allocating
one FmgrInfo for each distinct trigger involved, so if there is a single
trigger function referenced by multiple triggers, a version specialized
to each trigger can be stashed on its respective flinfo and used for
all firings of that trigger the query may generate.
This lends itself to a nice staged-programming view of a PL handler's job:
prepare: RegProcedure -> Template (cache by regproc)
specialize: (Template, call site) -> Routine (cache on fn_extra)
apply: (Routine, fcinfo) -> result
(where I consider some members both of flinfo, such as fn_expr, and of
fcinfo, such as nargs, context, resultinfo, to be logically properties
of a notional "call site").
I wonder, though, if there might be code in the wild, or even in corners
of the core I haven't looked in, where FmgrInfo structs aren't being used
that way, and could get reused for successive calls of one routine but
with, say, different nargs or argument types. That would seem odd but
I don't see that the documentation ever came right out and said not to.
If that seems like a non-imaginary risk, I wonder if FmgrInfo could sprout
something like a safe-to-specialize bit, initialized to false for old code
that doesn't know about it, but set true in places where FmgrInfo structs
are definitely being managed as described above?
I suspect that would result in the vast majority of FmgrInfo structs
ever really encountered having the bit set.
Regards,
-Chap
A work-in-progress PL dispatcher based on the above can be seen at:
https://tada.github.io/pljava/preview1.7/pljava-api/apidocs/org.postgresql.pljava/org/postgresql/plj...
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: FmgrInfo allocation patterns (and PL handling as staged programming)
2025-04-06 17:18 FmgrInfo allocation patterns (and PL handling as staged programming) Chapman Flack <[email protected]>
@ 2025-04-06 17:33 ` Tom Lane <[email protected]>
2025-04-06 19:39 ` Re: FmgrInfo allocation patterns (and PL handling as staged programming) Chapman Flack <[email protected]>
0 siblings, 1 reply; 4+ messages in thread
From: Tom Lane @ 2025-04-06 17:33 UTC (permalink / raw)
To: Chapman Flack <[email protected]>; +Cc: pgsql-hackers
Chapman Flack <[email protected]> writes:
> To wit, given a query like
> SELECT hello(n), hello(x)
> FROM (VALUES
> (1::int4, 1.0::float4),
> (2, 2.0 ),
> (3, 3.0 )) AS t(n,x);
> the core code allocates one FmgrInfo for each of the two uses.
Yeah, there's no attempt to merge FmgrInfos across call sites
within a query. It's typical to use fn_extra to point to dynamic
state for a call, so that any such merging could break things.
> I wonder, though, if there might be code in the wild, or even in corners
> of the core I haven't looked in, where FmgrInfo structs aren't being used
> that way, and could get reused for successive calls of one routine but
> with, say, different nargs or argument types. That would seem odd but
> I don't see that the documentation ever came right out and said not to.
The only case I'm aware of that might require some thought is that the
relcache caches FmgrInfo structs for the opclass support functions for
each column of an index. That seems like it's close enough to being
just as specialized as a query callsite, but maybe not?
A downside of relying entirely on fn_extra is that you don't get to
amortize the specialization work across queries, even though it's
probably pretty repetitive. You might be interested in this recent
commit:
https://git.postgresql.org/gitweb/?p=postgresql.git;a=commitdiff;h=0dca5d68d7bebf2c1036fd84875533afe...
which formalizes some caching rules that plpgsql has used for
a long time, and extends the rules to support SQL-language
functions (which need to specialize on output rowtype as well
as what plpgsql has traditionally considered). Maybe you'd be
interested in using funccache.
regards, tom lane
^ permalink raw reply [nested|flat] 4+ messages in thread
* Re: FmgrInfo allocation patterns (and PL handling as staged programming)
2025-04-06 17:18 FmgrInfo allocation patterns (and PL handling as staged programming) Chapman Flack <[email protected]>
2025-04-06 17:33 ` Re: FmgrInfo allocation patterns (and PL handling as staged programming) Tom Lane <[email protected]>
@ 2025-04-06 19:39 ` Chapman Flack <[email protected]>
0 siblings, 0 replies; 4+ messages in thread
From: Chapman Flack @ 2025-04-06 19:39 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-hackers
On 04/06/25 13:33, Tom Lane wrote:
> Maybe you'd be interested in using funccache.
O funccache, where were you a year or two ago?
Can a spread-out variadic "any" arg list ever vary
in length or type on the fly at a single call site?
I notice that funccache only hashes the first nargs
argument types.
Regards,
-Chap
^ permalink raw reply [nested|flat] 4+ messages in thread
end of thread, other threads:[~2025-04-06 19:39 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-03-14 22:12 [PATCH 09/10] Add zstd compression levels Justin Pryzby <[email protected]>
2025-04-06 17:18 FmgrInfo allocation patterns (and PL handling as staged programming) Chapman Flack <[email protected]>
2025-04-06 17:33 ` Re: FmgrInfo allocation patterns (and PL handling as staged programming) Tom Lane <[email protected]>
2025-04-06 19:39 ` Re: FmgrInfo allocation patterns (and PL handling as staged programming) Chapman Flack <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox