agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v6 01/10] doc: btree deduplication 16+ messages / 1 participants [nested] [flat]
* [PATCH v6 01/10] doc: btree deduplication @ 2020-03-31 00:43 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2020-03-31 00:43 UTC (permalink / raw) commit 0d861bbb702f8aa05c2a4e3f1650e7e8df8c8c27 Author: Peter Geoghegan <[email protected]> --- doc/src/sgml/btree.sgml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/doc/src/sgml/btree.sgml b/doc/src/sgml/btree.sgml index d03ee4d6fa..69c1ee0e97 100644 --- a/doc/src/sgml/btree.sgml +++ b/doc/src/sgml/btree.sgml @@ -642,7 +642,7 @@ options(<replaceable>relopts</replaceable> <type>local_relopts *</type>) returns </para> <para> Deduplication works by periodically merging groups of duplicate - tuples together, forming a single posting list tuple for each + tuples together, forming a single <firstterm>posting list</firstterm> tuple for each group. The column key value(s) only appear once in this representation. This is followed by a sorted array of <acronym>TID</acronym>s that point to rows in the table. This -- 2.17.0 --C+ts3FVlLX8+P6JN Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v6-0002-doc-pg_stat_progress_basebackup.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 2/4] f! @ 2021-11-18 03:49 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2021-11-18 03:49 UTC (permalink / raw) --- doc/src/sgml/func.sgml | 27 +++++++++---------- src/backend/postmaster/autovacuum.c | 2 +- src/backend/storage/ipc/procsignal.c | 4 +-- src/backend/storage/ipc/signalfuncs.c | 33 +++++++++++------------ src/backend/tcop/postgres.c | 2 +- src/backend/utils/error/elog.c | 9 +++---- src/include/catalog/pg_proc.dat | 2 +- src/test/regress/expected/backtrace.out | 2 +- src/test/regress/expected/backtrace_1.out | 2 +- src/test/regress/sql/backtrace.sql | 2 +- 10 files changed, 41 insertions(+), 44 deletions(-) diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index a88583edfd..f2a1b701e6 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -25360,14 +25360,14 @@ SELECT collation for ('foo' COLLATE "de_DE"); level <literal>LOG</literal>. It will appear in the server log based on the log configuration set (See <xref linkend="runtime-config-logging"/> for more information), but will not be sent to the client regardless of - <xref linkend="guc-client-min-messages"/>. A backtrace will identify - where exactly the backend process is currently executing. This may be - useful to developers to diagnose stuck processes and other problems. + <xref linkend="guc-client-min-messages"/>. A backtrace identifies + what function the backend process is currently executing, and how it was called. + This may be useful to developers to diagnose stuck processes and other problems. This feature is not supported for the postmaster, logger, checkpointer, - walwriter, background writer or statistics collector process. This - feature will be available if PostgreSQL was built with the ability to - capture backtrace. If not available, the function will emit a warning - and return false. + walwriter, background writer or statistics collector processes. This + feature is available if PostgreSQL was built with the ability to + capture backtraces. If the feature is not supported, the function will + emit a warning and return false. </para></entry> </row> @@ -25494,9 +25494,8 @@ postgres=# select pg_log_backtrace(pg_backend_pid()); t (1 row) </programlisting> -The backtrace will be logged to the log file if logging is enabled, if logging -is disabled backtrace will be logged to the console where the postmaster was -started. For example: +The backtrace will be logged as specified by the logging configuration. +For example: <screen> 2021-01-27 11:33:50.247 IST [111735] LOG: current backtrace: postgres: postgresdba postgres [local] SELECT(set_backtrace+0x38) [0xae06c5] @@ -25518,9 +25517,9 @@ started. For example: /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2107bbd505] postgres: postgresdba postgres [local] SELECT() [0x4842a9] </screen> - You can get the file name and line number by using gdb/addr2line in - linux platforms, as a prerequisite users must ensure gdb/addr2line is - already installed: + You can get the file name and line number from the logged details by using gdb/addr2line in + linux platforms (users must ensure gdb/addr2line is + already installed). <programlisting> 1) "info line *address" from gdb on postgres executable. For example: gdb ./postgres @@ -25535,7 +25534,7 @@ addr2line -e ./postgres 0x71c25d function will emit a warning and return false, for example: <screen> WARNING: backtrace generation is not supported by this installation -HINT: You need to rebuild PostgreSQL using library containing backtrace_symbols. +HINT: You need to rebuild PostgreSQL using a library containing backtrace_symbols. pg_log_backtrace ------------------ f diff --git a/src/backend/postmaster/autovacuum.c b/src/backend/postmaster/autovacuum.c index 90111135ff..ca74937603 100644 --- a/src/backend/postmaster/autovacuum.c +++ b/src/backend/postmaster/autovacuum.c @@ -840,7 +840,7 @@ HandleAutoVacLauncherInterrupts(void) if (LogMemoryContextPending) ProcessLogMemoryContextInterrupt(); - /* Process logging backtrace */ + /* Process a request to log our backtrace */ if (LogBacktracePending) ProcessLogBacktraceInterrupt(); diff --git a/src/backend/storage/ipc/procsignal.c b/src/backend/storage/ipc/procsignal.c index 3dac4a8f4b..e769a7f785 100644 --- a/src/backend/storage/ipc/procsignal.c +++ b/src/backend/storage/ipc/procsignal.c @@ -613,8 +613,8 @@ ProcessBarrierPlaceholder(void) } /* - * HandleLogBacktraceInterrupt - Handle receipt of an interrupt indicating - * logging of backtrace. + * HandleLogBacktraceInterrupt - Handle receipt of an interrupt requesting to + * log a backtrace. * * All the actual work is deferred to ProcessLogBacktraceInterrupt(), * because we cannot safely emit a log message inside the signal handler. diff --git a/src/backend/storage/ipc/signalfuncs.c b/src/backend/storage/ipc/signalfuncs.c index d042feefee..0315968aa2 100644 --- a/src/backend/storage/ipc/signalfuncs.c +++ b/src/backend/storage/ipc/signalfuncs.c @@ -303,23 +303,29 @@ pg_rotate_logfile_v2(PG_FUNCTION_ARGS) /* * pg_log_backtrace - log backtrace of backend process. * - * By default, only superusers can log backtrace. Additional roles can be + * By default, only superusers can log a backtrace. Additional roles can be * permitted with GRANT. */ Datum pg_log_backtrace(PG_FUNCTION_ARGS) { -#ifdef HAVE_BACKTRACE_SYMBOLS int pid = PG_GETARG_INT32(0); PGPROC *proc; +#ifndef HAVE_BACKTRACE_SYMBOLS + ereport(WARNING, + errmsg("backtrace generation is not supported by this installation"), + errhint("You need to rebuild PostgreSQL using a library containing backtrace_symbols.")); + PG_RETURN_BOOL(false); +#endif + /* * BackendPidGetProc returns NULL if the pid isn't valid; but by the time * we reach kill(), a process for which we get a valid proc here might have - * terminated on its own. There's no way to acquire a lock on an arbitrary - * process to prevent that. But since this mechanism is usually used to - * debug a backend running and consuming lots of CPU cycles, that it might - * end on its own first and its backtrace are not logged is not a problem. + * terminated on its own. There's no way to prevent a process from ending + * to avoid that. But since this mechanism is usually used to debug a + * backend consuming lots of CPU cycles, it's not considered a problem if + * it ends on its own without first logging its backtrace. */ proc = BackendPidGetProc(pid); if (proc == NULL) @@ -330,20 +336,13 @@ pg_log_backtrace(PG_FUNCTION_ARGS) } /* - * Send SIGUSR1 to postgres backend whose pid matches pid by - * setting PROCSIG_LOG_BACKTRACE, the backend process will log - * the backtrace once the signal is received. + * Send SIGUSR1 to postgres backend with the given pid, setting + * PROCSIG_LOG_BACKTRACE, requesting the backend to log its backtrace. */ if (!SendProcSignal(pid, PROCSIG_LOG_BACKTRACE, proc->backendId)) PG_RETURN_BOOL(true); - else - ereport(WARNING, - (errmsg("could not send signal to process %d: %m", pid))); /* return false below */ -#else - ereport(WARNING, - errmsg("backtrace generation is not supported by this installation"), - errhint("You need to rebuild PostgreSQL using library containing backtrace_symbols.")); -#endif + ereport(WARNING, + (errmsg("could not send signal to process %d: %m", pid))); PG_RETURN_BOOL(false); } diff --git a/src/backend/tcop/postgres.c b/src/backend/tcop/postgres.c index 3a2b670d23..cb3d218449 100644 --- a/src/backend/tcop/postgres.c +++ b/src/backend/tcop/postgres.c @@ -3367,7 +3367,7 @@ ProcessInterrupts(void) if (LogMemoryContextPending) ProcessLogMemoryContextInterrupt(); - /* Process logging backtrace */ + /* Process a request to log our backtrace */ if (LogBacktracePending) ProcessLogBacktraceInterrupt(); } diff --git a/src/backend/utils/error/elog.c b/src/backend/utils/error/elog.c index 6166965a39..dce0e2b155 100644 --- a/src/backend/utils/error/elog.c +++ b/src/backend/utils/error/elog.c @@ -948,7 +948,7 @@ errbacktrace(void) * Compute backtrace data and add it to the supplied ErrorData. num_skip * specifies how many inner frames to skip. Use this to avoid showing the * internal backtrace support functions in the backtrace. This requires that - * this and related functions are not inlined. If edata pointer is valid + * this and related functions are not inlined. If the edata pointer is valid, * backtrace information will be set in edata. */ void @@ -983,10 +983,9 @@ set_backtrace(ErrorData *edata, int num_skip) else { /* - * LOG_SERVER_ONLY is used intentionally to make sure this information - * is not sent to client based on client_min_messages. We don't want - * to mess up a different session as pg_log_backtrace will be - * sending SIGNAL to a different backend. + * LOG_SERVER_ONLY is used to make sure the backtrace is never + * sent to client. We want to avoid messing up the other client + * session. */ elog(LOG_SERVER_ONLY, "current backtrace:%s", errtrace.data); pfree(errtrace.data); diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index 2537edca12..498ee4e869 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -11694,7 +11694,7 @@ prosrc => 'brin_minmax_multi_summary_send' }, # function to get the backtrace of server process -{ oid => '6105', descr => 'log backtrace of process', +{ oid => '6105', descr => 'log backtrace of server process', proname => 'pg_log_backtrace', provolatile => 'v', prorettype => 'bool', proargtypes => 'int4', prosrc => 'pg_log_backtrace' }, diff --git a/src/test/regress/expected/backtrace.out b/src/test/regress/expected/backtrace.out index 88ee383d4d..dfb044fd1d 100644 --- a/src/test/regress/expected/backtrace.out +++ b/src/test/regress/expected/backtrace.out @@ -1,7 +1,7 @@ -- -- pg_log_backtrace() -- --- Backtrace are logged and they are not returned to the function. +-- Backtraces are logged and not returned to the function. -- Furthermore, their contents can vary depending on the timing. However, -- we can at least verify that the code doesn't fail, and that the -- permissions are set properly. diff --git a/src/test/regress/expected/backtrace_1.out b/src/test/regress/expected/backtrace_1.out index 62beef8f51..f2c837965e 100644 --- a/src/test/regress/expected/backtrace_1.out +++ b/src/test/regress/expected/backtrace_1.out @@ -1,7 +1,7 @@ -- -- pg_log_backtrace() -- --- Backtrace are logged and they are not returned to the function. +-- Backtraces are logged and not returned to the function. -- Furthermore, their contents can vary depending on the timing. However, -- we can at least verify that the code doesn't fail, and that the -- permissions are set properly. diff --git a/src/test/regress/sql/backtrace.sql b/src/test/regress/sql/backtrace.sql index 2688bc7330..a8465b017f 100644 --- a/src/test/regress/sql/backtrace.sql +++ b/src/test/regress/sql/backtrace.sql @@ -1,7 +1,7 @@ -- -- pg_log_backtrace() -- --- Backtrace are logged and they are not returned to the function. +-- Backtraces are logged and not returned to the function. -- Furthermore, their contents can vary depending on the timing. However, -- we can at least verify that the code doesn't fail, and that the -- permissions are set properly. -- 2.17.0 --exKAuxcSaZxx4B14 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-pg_log_backtrace-support-for-logging-backtrace-of-au.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 4/4] f! @ 2021-11-18 04:03 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2021-11-18 04:03 UTC (permalink / raw) --- src/backend/postmaster/checkpointer.c | 2 +- src/backend/postmaster/interrupt.c | 2 +- src/backend/postmaster/pgarch.c | 2 +- src/backend/postmaster/startup.c | 2 +- src/backend/postmaster/walwriter.c | 2 +- src/backend/storage/ipc/signalfuncs.c | 2 +- 6 files changed, 6 insertions(+), 6 deletions(-) diff --git a/src/backend/postmaster/checkpointer.c b/src/backend/postmaster/checkpointer.c index 7a25557702..31fb8abae4 100644 --- a/src/backend/postmaster/checkpointer.c +++ b/src/backend/postmaster/checkpointer.c @@ -582,7 +582,7 @@ HandleCheckpointerInterrupts(void) proc_exit(0); /* done */ } - /* Process logging backtrace */ + /* Process a request to log our backtrace */ if (LogBacktracePending) ProcessLogBacktraceInterrupt(); } diff --git a/src/backend/postmaster/interrupt.c b/src/backend/postmaster/interrupt.c index 6298b48da3..2a75f89fc6 100644 --- a/src/backend/postmaster/interrupt.c +++ b/src/backend/postmaster/interrupt.c @@ -44,7 +44,7 @@ HandleMainLoopInterrupts(void) if (ShutdownRequestPending) proc_exit(0); - /* Process logging backtrace */ + /* Process a request to log our backtrace */ if (LogBacktracePending) ProcessLogBacktraceInterrupt(); } diff --git a/src/backend/postmaster/pgarch.c b/src/backend/postmaster/pgarch.c index 938c08b0eb..b24a8f0264 100644 --- a/src/backend/postmaster/pgarch.c +++ b/src/backend/postmaster/pgarch.c @@ -857,7 +857,7 @@ HandlePgArchInterrupts(void) ProcessConfigFile(PGC_SIGHUP); } - /* Process logging backtrace */ + /* Process a request to log our backtrace */ if (LogBacktracePending) ProcessLogBacktraceInterrupt(); } diff --git a/src/backend/postmaster/startup.c b/src/backend/postmaster/startup.c index 41d40358bf..f920b73a4b 100644 --- a/src/backend/postmaster/startup.c +++ b/src/backend/postmaster/startup.c @@ -201,7 +201,7 @@ HandleStartupProcInterrupts(void) if (ProcSignalBarrierPending) ProcessProcSignalBarrier(); - /* Process logging backtrace */ + /* Process a request to log our backtrace */ if (LogBacktracePending) ProcessLogBacktraceInterrupt(); } diff --git a/src/backend/postmaster/walwriter.c b/src/backend/postmaster/walwriter.c index 8bc404d17b..4ac1383e07 100644 --- a/src/backend/postmaster/walwriter.c +++ b/src/backend/postmaster/walwriter.c @@ -307,7 +307,7 @@ HandleWalWriterInterrupts(void) proc_exit(0); } - /* Process logging backtrace */ + /* Process a request to log our backtrace */ if (LogBacktracePending) ProcessLogBacktraceInterrupt(); } diff --git a/src/backend/storage/ipc/signalfuncs.c b/src/backend/storage/ipc/signalfuncs.c index 562ff104bb..6bf1de27e2 100644 --- a/src/backend/storage/ipc/signalfuncs.c +++ b/src/backend/storage/ipc/signalfuncs.c @@ -344,7 +344,7 @@ pg_log_backtrace(PG_FUNCTION_ARGS) } /* - * Only regular backends will have valid backend id, auxiliary processes + * Only regular backends will have a valid backend id; auxiliary processes * don't. */ if (!aux_proc) -- 2.17.0 --exKAuxcSaZxx4B14-- ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 3/4] f! @ 2022-04-01 12:00 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2022-04-01 12:00 UTC (permalink / raw) --- src/backend/utils/sort/tuplesort.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c index 5cc1328febf..a0ab61e8f9a 100644 --- a/src/backend/utils/sort/tuplesort.c +++ b/src/backend/utils/sort/tuplesort.c @@ -1271,7 +1271,7 @@ tuplesort_begin_datum(Oid datumType, Oid sortOperator, Oid sortCollation, false, /* no unique check */ 1, workMem, - randomAccess, + sortopt & TUPLESORT_RANDOMACCESS, PARALLEL_SORT(state)); state->comparetup = comparetup_datum; -- 2.17.1 --RHnfto7yGd+QdmRl Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0004-Use-Generation-memory-contexts-to-store-tuples-in-so.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH] f @ 2022-07-04 21:11 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2022-07-04 21:11 UTC (permalink / raw) --- doc/src/sgml/brin.sgml | 28 ++++++++++++++-------------- doc/src/sgml/ref/create_index.sgml | 2 +- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/doc/src/sgml/brin.sgml b/doc/src/sgml/brin.sgml index 0a715d41c71..90897a4af07 100644 --- a/doc/src/sgml/brin.sgml +++ b/doc/src/sgml/brin.sgml @@ -76,17 +76,17 @@ </para> <para> - There are several triggers for initial summarization of a page range - to occur. If the table is vacuumed, either because - <xref linkend="sql-vacuum" /> has been manually invoked or because - autovacuum causes it, + There are several ways to trigger the initial summarization of a page range. + If the table is vacuumed, either manually or by + <link linkend="autovacuum">autovacuum</link>, all existing unsummarized page ranges are summarized. - Also, if the index has the - <xref linkend="index-reloption-autosummarize"/> parameter set to on, - then any run of autovacuum in the database will summarize all - unsummarized page ranges that have been completely filled recently, - regardless of whether the table is processed by autovacuum for other - reasons; see below. + Also, if the index's + <xref linkend="index-reloption-autosummarize"/> parameter is enabled, + whenever autovacuum runs in that database, summarization will + occur for all + unsummarized page ranges that have been filled, + regardless of whether the table itself is processed by autovacuum; see below. + Lastly, the following functions can be used: <simplelist> @@ -102,10 +102,10 @@ </para> <para> - When autosummarization is enabled, each time a page range is filled a - request is sent to <literal>autovacuum</literal> for it to execute a targeted - summarization for that range, to be fulfilled at the end of the next - autovacuum worker run on the same database. If the request queue is full, the + When autosummarization is enabled, each time a page range is filled, a + request is sent to <literal>autovacuum</literal> to execute a targeted + summarization for that range, to be fulfilled the next time an autovacuum + worker finishes running in that database. If the request queue is full, the request is not recorded and a message is sent to the server log: <screen> LOG: request for BRIN range summarization for index "brin_wi_idx" page 128 was not recorded diff --git a/doc/src/sgml/ref/create_index.sgml b/doc/src/sgml/ref/create_index.sgml index d3db03278d6..a5bac9f7373 100644 --- a/doc/src/sgml/ref/create_index.sgml +++ b/doc/src/sgml/ref/create_index.sgml @@ -578,7 +578,7 @@ CREATE [ UNIQUE ] INDEX [ CONCURRENTLY ] [ [ IF NOT EXISTS ] <replaceable class= </term> <listitem> <para> - Defines whether a summarization run is invoked for the previous page + Defines whether a summarization run is queued for the previous page range whenever an insertion is detected on the next one. See <xref linkend="brin-operation"/> for more details. The default is <literal>off</literal>. -- 2.17.1 --RpqchZ26BWispMcB-- ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 2/2] f! @ 2022-07-28 15:57 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2022-07-28 15:57 UTC (permalink / raw) ci-os-only: mingw --- .cirrus.yml | 23 ++++++++++++++++------- config/prep_buildtree | 18 ++++++------------ 2 files changed, 22 insertions(+), 19 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 1a06cdcaadb..93774485bef 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -348,8 +348,6 @@ WINDOWS_ENVIRONMENT_BASE: &WINDOWS_ENVIRONMENT_BASE # Avoids port conflicts between concurrent tap test runs PG_TEST_USE_UNIX_SOCKETS: 1 - only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' - sysinfo_script: | chcp systeminfo @@ -360,6 +358,8 @@ task: << : *WINDOWS_ENVIRONMENT_BASE name: Windows - Server 2019, VS 2019 + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + env: # Our windows infrastructure doesn't have test concurrency above the level # of a single vcregress test target. Due to that, it's useful to run prove @@ -465,6 +465,8 @@ task: task: << : *WINDOWS_ENVIRONMENT_BASE name: Windows - Server 2019, MinGW64 + only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*' + windows_container: image: $CONTAINER_REPO/windows_ci_mingw64:latest cpu: $CPUS @@ -482,21 +484,28 @@ task: - C:\msys64\usr\bin\dash.exe -lc "where perl" - C:\msys64\usr\bin\dash.exe -lc "perl --version" + setup_additional_packages_script: | + REM C:\msys64\usr\bin\pacman.exe -S --noconfirm busybox + configure_script: - C:\msys64\usr\bin\dash.exe -lc "mkdir %BUILD_DIR% && - cd %BUILD_DIR% && + cd %BUILD_DIR% && for i in 1 2; do %CIRRUS_WORKING_DIR%/configure + --cache-file=${CCACHE_DIR}/configure.cache --enable-cassert + --enable-debug --enable-tap-tests --with-icu --with-libxml --with-libxslt --with-lz4 - --enable-debug CC='ccache gcc' CXX='ccache g++' CFLAGS='-Og -ggdb -pipe' - CXXFLAGS='-Og -ggdb'" + CXXFLAGS='-Og -ggdb' && break; + rm -v ${CCACHE_DIR}/configure.cache; + done + " build_script: C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s world-bin -j${CPUS}" @@ -504,8 +513,8 @@ task: upload_caches: ccache tests_script: - - set "NoDefaultCurrentDirectoryInExePath=0" - - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install" + - set "NoDefaultCurrentDirectoryInExePath=0" + - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install" on_failure: *on_failure diff --git a/config/prep_buildtree b/config/prep_buildtree index a0eabd3dee2..4de30ac28a5 100644 --- a/config/prep_buildtree +++ b/config/prep_buildtree @@ -26,20 +26,14 @@ buildtree=`cd ${2:-'.'} && pwd` # If we did, it would interfere with installation of prebuilt docs from # the source tree, if a VPATH build is done from a distribution tarball. # See bug #5595. -for item in `find "$sourcetree" -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) | grep -v "$sourcetree/doc/src/sgml/\+"`; do - subdir=`expr "$item" : "$sourcetree\(.*\)"` - if test ! -d "$buildtree/$subdir"; then - mkdir -p "$buildtree/$subdir" || exit 1 - fi -done +( cd "$sourcetree" && find . -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) |grep -v "doc/src/sgml/\+" ) | + ( cd "$buildtree" && xargs mkdir -p ) for item in `find "$sourcetree" -name Makefile -print -o -name GNUmakefile -print | grep -v "$sourcetree/doc/src/sgml/images/"`; do - filename=`expr "$item" : "$sourcetree\(.*\)"` - if test ! -f "${item}.in"; then - if cmp "$item" "$buildtree/$filename" >/dev/null 2>&1; then : ; else - ln -fs "$item" "$buildtree/$filename" || exit 1 - fi - fi + filename=${item#$sourcetree} + [ -e "$buildtree/$filename" ] && continue + cmp "$item" "$buildtree/$filename" >/dev/null 2>&1 || + ln -fs "$item" "$buildtree/$filename" done exit 0 -- 2.17.1 --Ca0e2zgpnh8/XhnM-- ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 2/2] f! @ 2022-07-28 15:57 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2022-07-28 15:57 UTC (permalink / raw) ci-os-only: mingw --- .cirrus.yml | 31 +++++++++++++++++++++++-------- config/prep_buildtree | 20 ++++++++------------ src/tools/ci/cores_backtrace.sh | 17 +++++++++++++++-- 3 files changed, 46 insertions(+), 22 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 472661c0936..b51a625d210 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -34,6 +34,7 @@ on_failure: &on_failure - "**/*.log" - "**/*.diffs" - "**/regress_log_*" + - "**/*.stackdump" type: text/plain task: @@ -349,8 +350,6 @@ WINDOWS_ENVIRONMENT_BASE: &WINDOWS_ENVIRONMENT_BASE # Avoids port conflicts between concurrent tap test runs PG_TEST_USE_UNIX_SOCKETS: 1 - only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' - sysinfo_script: | chcp systeminfo @@ -361,6 +360,8 @@ task: << : *WINDOWS_ENVIRONMENT_BASE name: Windows - Server 2019, VS 2019 + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + env: # Our windows infrastructure doesn't have test concurrency above the level # of a single vcregress test target. Due to that, it's useful to run prove @@ -484,6 +485,8 @@ task: task: << : *WINDOWS_ENVIRONMENT_BASE name: Windows - Server 2019, MinGW64 + #XXX only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*' + windows_container: image: $CONTAINER_REPO/windows_ci_mingw64:latest cpu: $CPUS @@ -495,6 +498,10 @@ task: ccache_cache: folder: ${CCACHE_DIR} + setup_additional_packages_script: | + REM choco install -y --no-progress ... + REM C:\msys64\usr\bin\pacman.exe -S --noconfirm busybox + mingw_info_script: - C:\msys64\usr\bin\dash.exe -lc "where gcc" - C:\msys64\usr\bin\dash.exe -lc "gcc --version" @@ -502,20 +509,25 @@ task: - C:\msys64\usr\bin\dash.exe -lc "perl --version" configure_script: + # Try to configure with the cache file, and retry without if it fails, in case the flags changed. - C:\msys64\usr\bin\dash.exe -lc "mkdir %BUILD_DIR% && - cd %BUILD_DIR% && + cd %BUILD_DIR% && for i in 1 2; do %CIRRUS_WORKING_DIR%/configure + --cache-file=${CCACHE_DIR}/configure.cache --enable-cassert + --enable-debug --enable-tap-tests --with-icu --with-libxml --with-libxslt --with-lz4 - --enable-debug CC='ccache gcc' CXX='ccache g++' CFLAGS='-Og -ggdb -pipe' - CXXFLAGS='-Og -ggdb'" + CXXFLAGS='-Og -ggdb' && break; + rm -v ${CCACHE_DIR}/configure.cache; + done + " build_script: C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s world-bin -j${CPUS}" @@ -523,10 +535,13 @@ task: upload_caches: ccache tests_script: - - set "NoDefaultCurrentDirectoryInExePath=0" - - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install" + - set "NoDefaultCurrentDirectoryInExePath=0" + - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install" - on_failure: *on_failure + on_failure: + <<: *on_failure + cores_script: + - C:\tools\cygwin\bin\dash.exe --login -c "cd '%cd%' && src/tools/ci/cores_backtrace.sh msys ." task: name: CompilerWarnings diff --git a/config/prep_buildtree b/config/prep_buildtree index a0eabd3dee2..97bf65967f1 100644 --- a/config/prep_buildtree +++ b/config/prep_buildtree @@ -26,20 +26,16 @@ buildtree=`cd ${2:-'.'} && pwd` # If we did, it would interfere with installation of prebuilt docs from # the source tree, if a VPATH build is done from a distribution tarball. # See bug #5595. -for item in `find "$sourcetree" -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) | grep -v "$sourcetree/doc/src/sgml/\+"`; do - subdir=`expr "$item" : "$sourcetree\(.*\)"` - if test ! -d "$buildtree/$subdir"; then - mkdir -p "$buildtree/$subdir" || exit 1 - fi -done +( cd "$sourcetree" && find . -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) |grep -v "doc/src/sgml/\+" ) | + ( cd "$buildtree" && xargs mkdir -p ) for item in `find "$sourcetree" -name Makefile -print -o -name GNUmakefile -print | grep -v "$sourcetree/doc/src/sgml/images/"`; do - filename=`expr "$item" : "$sourcetree\(.*\)"` - if test ! -f "${item}.in"; then - if cmp "$item" "$buildtree/$filename" >/dev/null 2>&1; then : ; else - ln -fs "$item" "$buildtree/$filename" || exit 1 - fi - fi + [ -e "$item.in" ] && echo "skipping: $item" && + continue + + filename=${item#$sourcetree} + cmp "$item" "$buildtree/$filename" >/dev/null 2>&1 || + ln -fs "$item" "$buildtree/$filename" done exit 0 diff --git a/src/tools/ci/cores_backtrace.sh b/src/tools/ci/cores_backtrace.sh index 28d3cecfc67..93e90f284e6 100755 --- a/src/tools/ci/cores_backtrace.sh +++ b/src/tools/ci/cores_backtrace.sh @@ -10,11 +10,24 @@ directory=$2 case $os in freebsd|linux|macos) - ;; + ;; + + msys) + # XXX Evidently I don't know how to write two arguments here without pathname expansion later, other than eval. + #findargs='-name "*.stackdump"' + for corefile in $(find "$directory" -type f -name "*.stackdump") ; do + binary=`basename "$corefile" .stackdump` + echo;echo; + echo "dumping ${corefile} for ${binary}" + awk '/^0/{print $2}' $corefile |addr2line -f -i -e ./src/backend/postgres.exe + done + exit 0 + ;; + *) echo "unsupported operating system ${os}" exit 1 - ;; + ;; esac first=1 -- 2.17.1 --lildS9pRFgpM/xzO-- ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 2/2] f! @ 2022-07-28 15:57 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2022-07-28 15:57 UTC (permalink / raw) ci-os-only: mingw --- .cirrus.yml | 14 ++++++++++---- config/prep_buildtree | 20 ++++++++------------ 2 files changed, 18 insertions(+), 16 deletions(-) diff --git a/.cirrus.yml b/.cirrus.yml index 60f9ae2b65c..f7cd6de93d1 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -486,7 +486,8 @@ task: << : *WINDOWS_ENVIRONMENT_BASE name: Windows - Server 2019, MinGW64 - only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.* || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*' + #XXX only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*' + only_if: $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-include:[^\n]*mingw.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*mingw.*' windows_container: image: $CONTAINER_REPO/windows_ci_mingw64:latest @@ -517,6 +518,7 @@ task: --enable-cassert --enable-debug --enable-tap-tests + --host=x86_64-w64-mingw64 --with-icu --with-libxml --with-libxslt @@ -530,18 +532,22 @@ task: " build_script: - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s world-bin -j${CPUS}" + - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} world-bin" + - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} -C src/test" + - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} -C src/interfaces/ecpg/test" + - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s -j${CPUS} -C src/interfaces/libpq/test" upload_caches: ccache - tests_script: + test_world_script: - set "NoDefaultCurrentDirectoryInExePath=0" - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% && make -s ${CHECK} ${CHECKFLAGS} -j${CPUS} TMPDIR=%BUILD_DIR%/tmp_install" on_failure: <<: *on_failure cores_script: - - C:\tools\cygwin\bin\dash.exe --login -c "cd '%cd%' && src/tools/ci/cores_backtrace.sh msys ." + - C:\msys64\usr\bin\dash.exe -lc "cd %BUILD_DIR% src/tools/ci/cores_backtrace.sh msys %BUILD_DIR%/tmp_install" + task: name: CompilerWarnings diff --git a/config/prep_buildtree b/config/prep_buildtree index a0eabd3dee2..97bf65967f1 100644 --- a/config/prep_buildtree +++ b/config/prep_buildtree @@ -26,20 +26,16 @@ buildtree=`cd ${2:-'.'} && pwd` # If we did, it would interfere with installation of prebuilt docs from # the source tree, if a VPATH build is done from a distribution tarball. # See bug #5595. -for item in `find "$sourcetree" -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) | grep -v "$sourcetree/doc/src/sgml/\+"`; do - subdir=`expr "$item" : "$sourcetree\(.*\)"` - if test ! -d "$buildtree/$subdir"; then - mkdir -p "$buildtree/$subdir" || exit 1 - fi -done +( cd "$sourcetree" && find . -type d \( \( -name CVS -prune \) -o \( -name .git -prune \) -o -print \) |grep -v "doc/src/sgml/\+" ) | + ( cd "$buildtree" && xargs mkdir -p ) for item in `find "$sourcetree" -name Makefile -print -o -name GNUmakefile -print | grep -v "$sourcetree/doc/src/sgml/images/"`; do - filename=`expr "$item" : "$sourcetree\(.*\)"` - if test ! -f "${item}.in"; then - if cmp "$item" "$buildtree/$filename" >/dev/null 2>&1; then : ; else - ln -fs "$item" "$buildtree/$filename" || exit 1 - fi - fi + [ -e "$item.in" ] && echo "skipping: $item" && + continue + + filename=${item#$sourcetree} + cmp "$item" "$buildtree/$filename" >/dev/null 2>&1 || + ln -fs "$item" "$buildtree/$filename" done exit 0 -- 2.17.1 --TcuvTDpCAASXpu1W-- ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 2/2] f @ 2022-09-09 12:52 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2022-09-09 12:52 UTC (permalink / raw) --- src/bin/pg_upgrade/check.c | 11 ++++----- src/bin/pg_upgrade/pg_upgrade.h | 2 +- src/bin/pg_upgrade/version.c | 41 ++++++++++++++------------------- 3 files changed, 22 insertions(+), 32 deletions(-) diff --git a/src/bin/pg_upgrade/check.c b/src/bin/pg_upgrade/check.c index f1bc1e68868..dc722da910c 100644 --- a/src/bin/pg_upgrade/check.c +++ b/src/bin/pg_upgrade/check.c @@ -1328,15 +1328,12 @@ check_for_reg_data_type_usage(ClusterInfo *cluster) static void check_for_jsonb_9_4_usage(ClusterInfo *cluster) { - char output_path[MAXPGPATH]; + const char output_basename[] = "tables_using_jsonb.txt"; prep_status("Checking for incompatible \"jsonb\" data type"); - snprintf(output_path, sizeof(output_path), "%s/%s", - log_opts.basedir, - "tables_using_jsonb.txt"); - - if (check_for_data_type_usage(cluster, "pg_catalog.jsonb", output_path)) + if (check_for_data_type_usage(cluster, "pg_catalog.jsonb", + output_basename)) { pg_log(PG_REPORT, "fatal"); pg_fatal("Your installation contains the \"jsonb\" data type in user tables.\n" @@ -1344,7 +1341,7 @@ check_for_jsonb_9_4_usage(ClusterInfo *cluster) "cluster cannot currently be upgraded. You can\n" "drop the problem columns and restart the upgrade.\n" "A list of the problem columns is in the file:\n" - " %s", output_path); + " %s/%s", log_opts.basedir, output_basename); } else check_ok(); diff --git a/src/bin/pg_upgrade/pg_upgrade.h b/src/bin/pg_upgrade/pg_upgrade.h index 60c3c8dd68e..0a1bdfca5b2 100644 --- a/src/bin/pg_upgrade/pg_upgrade.h +++ b/src/bin/pg_upgrade/pg_upgrade.h @@ -443,7 +443,7 @@ unsigned int str2uint(const char *str); bool check_for_data_types_usage(ClusterInfo *cluster, const char *base_query, - const char *output_path); + const char *output_basename); bool check_for_data_type_usage(ClusterInfo *cluster, const char *type_name, const char *output_path); diff --git a/src/bin/pg_upgrade/version.c b/src/bin/pg_upgrade/version.c index dc19fc6ec84..059bd9b1e1c 100644 --- a/src/bin/pg_upgrade/version.c +++ b/src/bin/pg_upgrade/version.c @@ -30,12 +30,16 @@ bool check_for_data_types_usage(ClusterInfo *cluster, const char *base_query, - const char *output_path) + const char *output_basename) { bool found = false; FILE *script = NULL; int dbnum; + char output_path[MAXPGPATH]; + snprintf(output_path, sizeof(output_path), "%s/%s", log_opts.basedir, + output_basename); + for (dbnum = 0; dbnum < cluster->dbarr.ndbs; dbnum++) { DbInfo *active_db = &cluster->dbarr.dbs[dbnum]; @@ -152,7 +156,7 @@ check_for_data_types_usage(ClusterInfo *cluster, bool check_for_data_type_usage(ClusterInfo *cluster, const char *type_name, - const char *output_path) + const char *output_basename) { bool found; char *base_query; @@ -160,7 +164,7 @@ check_for_data_type_usage(ClusterInfo *cluster, base_query = psprintf("SELECT '%s'::pg_catalog.regtype AS oid", type_name); - found = check_for_data_types_usage(cluster, base_query, output_path); + found = check_for_data_types_usage(cluster, base_query, output_basename); free(base_query); @@ -179,15 +183,11 @@ check_for_data_type_usage(ClusterInfo *cluster, void old_9_3_check_for_line_data_type_usage(ClusterInfo *cluster) { - char output_path[MAXPGPATH]; + const char output_basename[] = "tables_using_line.txt"; prep_status("Checking for incompatible \"line\" data type"); - snprintf(output_path, sizeof(output_path), "%s/%s", - log_opts.basedir, - "tables_using_line.txt"); - - if (check_for_data_type_usage(cluster, "pg_catalog.line", output_path)) + if (check_for_data_type_usage(cluster, "pg_catalog.line", output_basename)) { pg_log(PG_REPORT, "fatal"); pg_fatal("Your installation contains the \"line\" data type in user tables.\n" @@ -196,7 +196,7 @@ old_9_3_check_for_line_data_type_usage(ClusterInfo *cluster) "cluster cannot currently be upgraded. You can\n" "drop the problem columns and restart the upgrade.\n" "A list of the problem columns is in the file:\n" - " %s", output_path); + " %s/%s", log_opts.basedir, output_basename); } else check_ok(); @@ -219,15 +219,12 @@ old_9_3_check_for_line_data_type_usage(ClusterInfo *cluster) void old_9_6_check_for_unknown_data_type_usage(ClusterInfo *cluster) { - char output_path[MAXPGPATH]; + const char output_basename[] = "tables_using_unknown.txt"; prep_status("Checking for invalid \"unknown\" user columns"); - snprintf(output_path, sizeof(output_path), "%s/%s", - log_opts.basedir, - "tables_using_unknown.txt"); - - if (check_for_data_type_usage(cluster, "pg_catalog.unknown", output_path)) + if (check_for_data_type_usage(cluster, "pg_catalog.unknown", + output_basename)) { pg_log(PG_REPORT, "fatal"); pg_fatal("Your installation contains the \"unknown\" data type in user tables.\n" @@ -235,7 +232,7 @@ old_9_6_check_for_unknown_data_type_usage(ClusterInfo *cluster) "cluster cannot currently be upgraded. You can\n" "drop the problem columns and restart the upgrade.\n" "A list of the problem columns is in the file:\n" - " %s", output_path); + " %s/%s", log_opts.basedir, output_basename); } else check_ok(); @@ -364,16 +361,12 @@ old_9_6_invalidate_hash_indexes(ClusterInfo *cluster, bool check_mode) void old_11_check_for_sql_identifier_data_type_usage(ClusterInfo *cluster) { - char output_path[MAXPGPATH]; + const char output_basename[] = "tables_using_sql_identifier.txt"; prep_status("Checking for invalid \"sql_identifier\" user columns"); - snprintf(output_path, sizeof(output_path), "%s/%s", - log_opts.basedir, - "tables_using_sql_identifier.txt"); - if (check_for_data_type_usage(cluster, "information_schema.sql_identifier", - output_path)) + output_basename)) { pg_log(PG_REPORT, "fatal"); pg_fatal("Your installation contains the \"sql_identifier\" data type in user tables.\n" @@ -381,7 +374,7 @@ old_11_check_for_sql_identifier_data_type_usage(ClusterInfo *cluster) "cluster cannot currently be upgraded. You can\n" "drop the problem columns and restart the upgrade.\n" "A list of the problem columns is in the file:\n" - " %s", output_path); + " %s/%s", log_opts.basedir, output_basename); } else check_ok(); -- 2.17.1 --RyB1I56Pcq42iZjE-- ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH] f! @ 2023-01-07 00:28 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2023-01-07 00:28 UTC (permalink / raw) --- doc/src/sgml/datatype.sgml | 4 ++-- doc/src/sgml/ddl.sgml | 4 ++-- doc/src/sgml/libpq.sgml | 2 +- doc/src/sgml/protocol.sgml | 2 +- src/bin/pg_dump/pg_dump.c | 4 ++-- src/interfaces/libpq/fe-exec.c | 12 ++++++------ 6 files changed, 14 insertions(+), 14 deletions(-) diff --git a/doc/src/sgml/datatype.sgml b/doc/src/sgml/datatype.sgml index 56b2e1d0d1e..243e6861506 100644 --- a/doc/src/sgml/datatype.sgml +++ b/doc/src/sgml/datatype.sgml @@ -5369,7 +5369,7 @@ WHERE ... <type>pg_encrypted_rnd</type> (for randomized encryption) or <type>pg_encrypted_det</type> (for deterministic encryption); see <xref linkend="datatype-encrypted-table"/>. Most of the database system treats - this as normal types. For example, the type <type>pg_encrypted_det</type> has + these as normal types. For example, the type <type>pg_encrypted_det</type> has an equals operator that allows lookup of encrypted values. It is, however, not allowed to create a table using one of these types directly as a column type. @@ -5383,7 +5383,7 @@ WHERE ... Clients that don't support transparent column encryption or have disabled it will see the encrypted values in this format. Clients that support transparent data encryption will not see these types in result sets, as - the protocol layer will translate them back to declared underlying type in + the protocol layer will translate them back to the declared underlying type in the table definition. </para> diff --git a/doc/src/sgml/ddl.sgml b/doc/src/sgml/ddl.sgml index b8364a91f9a..a7624d6a60c 100644 --- a/doc/src/sgml/ddl.sgml +++ b/doc/src/sgml/ddl.sgml @@ -1263,7 +1263,7 @@ CREATE TABLE customers ( randomized encryption, which is the default. Randomized encryption uses a random initialization vector for each encryption, so that even if the plaintext of two rows is equal, the encrypted values will be different. - This prevents someone with direct access to the database server to make + This prevents someone with direct access to the database server from making computations such as distinct counts on the encrypted values. Deterministic encryption uses a fixed initialization vector. This reduces security, but it allows equality searches on encrypted values. The @@ -1540,7 +1540,7 @@ export PGCMKLOOKUP <para> In general, column encryption is never a replacement for additional - security and encryption techniques such as transmission encryption + security and encryption techniques such as transport encryption (SSL/TLS), storage encryption, strong access control, and password security. Column encryption only targets specific use cases and should be used in conjunction with additional security measures. diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml index a2d413bafd5..9b7f76db7a9 100644 --- a/doc/src/sgml/libpq.sgml +++ b/doc/src/sgml/libpq.sgml @@ -3012,7 +3012,7 @@ PGresult *PQexecParams(PGconn *conn, If encryption is forced for a parameter but the parameter does not correspond to an encrypted column on the server, then the call will fail and the parameter will not be sent. This can be used - for additional security against a comprimised server. (The + for additional security against a compromised server. (The drawback is that application code then needs to be kept up to date with knowledge about which columns are encrypted rather than letting the server specify this.) diff --git a/doc/src/sgml/protocol.sgml b/doc/src/sgml/protocol.sgml index 1a9b8abd7f2..caa6e3174ee 100644 --- a/doc/src/sgml/protocol.sgml +++ b/doc/src/sgml/protocol.sgml @@ -5807,7 +5807,7 @@ psql "dbname=postgres replication=database" -c "IDENTIFY_SYSTEM;" <listitem> <para> If the field is encrypted, this specifies the identifier of the - encrypt algorithm, else zero. + encryption algorithm, else zero. </para> </listitem> </varlistentry> diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index f82c2496fd5..99f2583c34a 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -690,7 +690,7 @@ main(int argc, char **argv) * --rows-per-insert were specified. */ if (dopt.cparams.column_encryption && dopt.dump_inserts == 0) - pg_fatal("option --decrypt_encrypted_columns requires option --inserts, --rows-per-insert, or --column-inserts"); + pg_fatal("option --decrypt-encrypted-columns requires option --inserts, --rows-per-insert, or --column-inserts"); if (dopt.do_nothing && dopt.dump_inserts == 0) pg_fatal("option --on-conflict-do-nothing requires option --inserts, --rows-per-insert, or --column-inserts"); @@ -13546,7 +13546,7 @@ dumpColumnEncryptionKey(Archive *fout, const CekInfo *cekinfo) appendPQExpBuffer(query, ")"); if (i < cekinfo->numdata - 1) - appendPQExpBuffer(query, ", "); + appendPQExpBuffer(query, ", "); } appendPQExpBufferStr(query, ";\n"); diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c index 4e835d6c681..7c7e2bac128 100644 --- a/src/interfaces/libpq/fe-exec.c +++ b/src/interfaces/libpq/fe-exec.c @@ -1699,13 +1699,13 @@ pqRowProcessor(PGconn *conn, const char **errmsgp) } else { - val = (char *) pqResultAlloc(res, clen + 1, isbinary); - if (val == NULL) - goto fail; + val = (char *) pqResultAlloc(res, clen + 1, isbinary); + if (val == NULL) + goto fail; - /* copy and zero-terminate the data (even if it's binary) */ - memcpy(val, columns[i].value, clen); - val[clen] = '\0'; + /* copy and zero-terminate the data (even if it's binary) */ + memcpy(val, columns[i].value, clen); + val[clen] = '\0'; } tup[i].len = clen; -- 2.25.1 --aVD9QWMuhilNxW9f-- ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 5/8] f! @ 2023-01-14 16:31 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2023-01-14 16:31 UTC (permalink / raw) --- src/bin/pg_dump/compress_gzip.c | 13 +---------- src/bin/pg_dump/compress_io.c | 40 ++++++++++----------------------- 2 files changed, 13 insertions(+), 40 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 37c841c5a9b..a021d414624 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -220,8 +220,6 @@ InitCompressorGzip(CompressorState *cs, const pg_compress_specification *compres cs->writeData = WriteDataToArchiveGzip; cs->end = EndCompressorGzip; - cs->compression_spec = *compression_spec; - gzipcs = (GzipCompressorState *) pg_malloc0(sizeof(GzipCompressorState)); cs->private_data = gzipcs; @@ -291,17 +289,10 @@ static int Gzip_close(CompressFileHandle *CFH) { gzFile gzfp = (gzFile) CFH->private_data; - int save_errno; - int ret; CFH->private_data = NULL; - ret = gzclose(gzfp); - - save_errno = errno; - errno = save_errno; - - return ret; + return gzclose(gzfp); } static int @@ -386,8 +377,6 @@ InitCompressGzip(CompressFileHandle *CFH, const pg_compress_specification *compr CFH->eof_func = Gzip_eof; CFH->get_error_func = Gzip_get_error; - CFH->compression_spec = *compression_spec; - CFH->private_data = NULL; } #else /* HAVE_LIBZ */ diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index c4ac4042794..7a1fd318c12 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -39,14 +39,13 @@ * * The compressed stream API is a wrapper around the C standard fopen() and * libz's gzopen() APIs and custom LZ4 calls which provide similar - * functionality. It allows you to use the same functions for compressed and - * uncompressed streams. cfopen_read() first tries to open the file with given - * name, and if it fails, it tries to open the same file with the .gz suffix, - * failing that it tries to open the same file with the .lz4 suffix. - * cfopen_write() opens a file for writing, an extra argument specifies the - * method to use should the file be compressed, and adds the appropriate - * suffix, .gz or .lz4, to the filename if so. This allows you to easily handle - * both compressed and uncompressed files. + * libz's gzopen() APIs. It allows you to use the same functions for + * compressed and uncompressed streams. cfopen_read() first tries to open + * the file with given name, and if it fails, it tries to open the same + * file with the .gz suffix. cfopen_write() opens a file for writing, an + * extra argument specifies if the file should be compressed, and adds the + * .gz suffix to the filename if so. This allows you to easily handle both + * compressed and uncompressed files. * * IDENTIFICATION * src/bin/pg_dump/compress_io.c @@ -110,8 +109,6 @@ InitCompressorNone(CompressorState *cs, cs->readData = ReadDataFromArchiveNone; cs->writeData = WriteDataToArchiveNone; cs->end = EndCompressorNone; - - cs->compression_spec = *compression_spec; } /* Public interface routines */ @@ -126,7 +123,8 @@ AllocateCompressor(const pg_compress_specification *compression_spec, cs = (CompressorState *) pg_malloc0(sizeof(CompressorState)); cs->readF = readF; cs->writeF = writeF; - cs->compression_spec = *compression_spec; + cs->compression_spec = *compression_spec; // XXX: should do this here rather than every compressor ? + // does it even need to be passed at all ? switch (compression_spec->algorithm) { @@ -340,10 +338,9 @@ InitCompressFileHandle(const pg_compress_specification *compression_spec) * Open a file for reading. 'path' is the file to open, and 'mode' should * be either "r" or "rb". * - * If the file at 'path' does not exist, we append the "{.gz,.lz4}" suffix (i - * 'path' doesn't already have it) and try again. So if you pass "foo" as - * 'path', this will open either "foo" or "foo.gz" or "foo.lz4", trying in that - * order. + * If the file at 'path' does not exist, we append the ".gz" suffix (if 'path' + * doesn't already have it) and try again. So if you pass "foo" as 'path', + * this will open either "foo" or "foo.gz". * * On failure, return NULL with an error code in errno. */ @@ -355,8 +352,6 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) char *fname; pg_compress_specification compression_spec = {0}; - // compression_spec.algorithm = PG_COMPRESSION_NONE; - Assert(strcmp(mode, "r") == 0 || strcmp(mode, "rb") == 0); fname = strdup(path); @@ -381,17 +376,6 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) if (exists) compression_spec.algorithm = PG_COMPRESSION_GZIP; } -#endif -#ifdef USE_LZ4 - if (!exists) - { - free_keep_errno(fname); - fname = psprintf("%s.lz4", path); - exists = (stat(fname, &st) == 0); - - if (exists) - compression_spec.algorithm = PG_COMPRESSION_LZ4; - } #endif } -- 2.25.1 --aBaYPhOdNx+t7mr3 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0006-Add-LZ4-compression-in-pg_-dump-restore.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 4/7] f! @ 2023-01-14 16:31 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2023-01-14 16:31 UTC (permalink / raw) --- src/bin/pg_dump/compress_gzip.c | 9 +-------- src/bin/pg_dump/pg_backup_archiver.c | 1 - 2 files changed, 1 insertion(+), 9 deletions(-) diff --git a/src/bin/pg_dump/compress_gzip.c b/src/bin/pg_dump/compress_gzip.c index 37c841c5a9b..b00be32f2e9 100644 --- a/src/bin/pg_dump/compress_gzip.c +++ b/src/bin/pg_dump/compress_gzip.c @@ -291,17 +291,10 @@ static int Gzip_close(CompressFileHandle *CFH) { gzFile gzfp = (gzFile) CFH->private_data; - int save_errno; - int ret; CFH->private_data = NULL; - ret = gzclose(gzfp); - - save_errno = errno; - errno = save_errno; - - return ret; + return gzclose(gzfp); } static int diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 06f0b46cbfc..7f06beff61c 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -385,7 +385,6 @@ RestoreArchive(Archive *AHX) */ supports_compression = true; if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE && - AH->compression_spec.algorithm == PG_COMPRESSION_GZIP && AH->PrintTocDataPtr != NULL) { for (te = AH->toc->next; te != AH->toc; te = te->next) -- 2.25.1 --e7jIye1Ygp5H0AIi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-Add-LZ4-compression-in-pg_-dump-restore.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 6/7] f! @ 2023-01-14 16:45 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2023-01-14 16:45 UTC (permalink / raw) --- src/bin/pg_dump/compress_lz4.c | 34 ++++++++++++++++---------------- src/bin/pg_dump/compress_lz4.h | 4 ++-- src/bin/pg_dump/t/002_pg_dump.pl | 2 +- 3 files changed, 20 insertions(+), 20 deletions(-) diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index c97e16187a0..0e259a6251a 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -117,13 +117,13 @@ EndCompressorLZ4(ArchiveHandle *AH, CompressorState *cs) /* Public routines that support LZ4 compressed data I/O */ void -InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec) +InitCompressorLZ4(CompressorState *cs, const pg_compress_specification *compression_spec) { cs->readData = ReadDataFromArchiveLZ4; cs->writeData = WriteDataToArchiveLZ4; cs->end = EndCompressorLZ4; - cs->compression_spec = compression_spec; + cs->compression_spec = *compression_spec; /* Will be lazy init'd */ cs->private_data = pg_malloc0(sizeof(LZ4CompressorState)); @@ -189,7 +189,7 @@ LZ4File_get_error(CompressFileHandle *CFH) /* * Prepare an already alloc'ed LZ4File struct for subsequent calls. * - * It creates the nessary contexts for the operations. When compressing, + * It creates the necessary contexts for the operations. When compressing, * it additionally writes the LZ4 header in the output stream. */ static int @@ -228,7 +228,7 @@ LZ4File_init(LZ4File * fs, int size, bool compressing) if (fwrite(fs->buffer, 1, status, fs->fp) != status) { - errno = errno ? : ENOSPC; + errno = errno ? errno : ENOSPC; return 1; } } @@ -255,7 +255,7 @@ LZ4File_init(LZ4File * fs, int size, bool compressing) /* * Read already decompressed content from the overflow buffer into 'ptr' up to * 'size' bytes, if available. If the eol_flag is set, then stop at the first - * occurance of the new line char prior to 'size' bytes. + * occurrence of the new line char prior to 'size' bytes. * * Any unread content in the overflow buffer, is moved to the beginning. */ @@ -309,10 +309,10 @@ LZ4File_read_internal(LZ4File * fs, void *ptr, int ptrsize, bool eol_flag) void *readbuf; /* Lazy init */ - if (!fs->inited && LZ4File_init(fs, size, false /* decompressing */ )) + if (LZ4File_init(fs, size, false /* decompressing */ )) return -1; - /* Verfiy that there is enough space in the outbuf */ + /* Verify that there is enough space in the outbuf */ if (size > fs->buflen) { fs->buflen = size; @@ -363,10 +363,10 @@ LZ4File_read_internal(LZ4File * fs, void *ptr, int ptrsize, bool eol_flag) if (outlen > 0 && dsize < size && eol_found == false) { char *p; - size_t lib = (eol_flag == 0) ? size - dsize : size - 1 - dsize; + size_t lib = eol_flag ? size - 1 - dsize : size - dsize ; size_t len = outlen < lib ? outlen : lib; - if (eol_flag == true && + if (eol_flag && (p = memchr(fs->buffer, '\n', outlen)) && (size_t) (p - fs->buffer + 1) <= len) { @@ -377,7 +377,7 @@ LZ4File_read_internal(LZ4File * fs, void *ptr, int ptrsize, bool eol_flag) memcpy((char *) ptr + dsize, fs->buffer, len); dsize += len; - /* move what did not fit, if any, at the begining of the buf */ + /* move what did not fit, if any, at the beginning of the buf */ if (len < outlen) memmove(fs->buffer, fs->buffer + len, outlen - len); outlen -= len; @@ -414,7 +414,7 @@ LZ4File_write(const void *ptr, size_t size, CompressFileHandle *CFH) size_t status; int remaining = size; - if (!fs->inited && LZ4File_init(fs, size, true)) + if (LZ4File_init(fs, size, true)) return -1; while (remaining > 0) @@ -433,7 +433,7 @@ LZ4File_write(const void *ptr, size_t size, CompressFileHandle *CFH) if (fwrite(fs->buffer, 1, status, fs->fp) != status) { - errno = errno ? : ENOSPC; + errno = errno ? errno : ENOSPC; return 1; } } @@ -520,7 +520,7 @@ LZ4File_close(CompressFileHandle *CFH) LZ4F_getErrorName(status)); else if ((ret = fwrite(fs->buffer, 1, status, fs->fp)) != status) { - errno = errno ? : ENOSPC; + errno = errno ? errno : ENOSPC; WRITE_ERROR_EXIT; } @@ -582,7 +582,7 @@ LZ4File_open_write(const char *path, const char *mode, CompressFileHandle *CFH) } void -InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification *compression_spec) { LZ4File *lz4fp; @@ -596,7 +596,7 @@ InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compres CFH->close_func = LZ4File_close; CFH->get_error_func = LZ4File_get_error; - CFH->compression_spec = compression_spec; + CFH->compression_spec = *compression_spec; lz4fp = pg_malloc0(sizeof(*lz4fp)); if (CFH->compression_spec.level >= 0) lz4fp->prefs.compressionLevel = CFH->compression_spec.level; @@ -605,13 +605,13 @@ InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compres } #else /* USE_LZ4 */ void -InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec) +InitCompressorLZ4(CompressorState *cs, const pg_compress_specification *compression_spec) { pg_fatal("this build does not support compression with %s", "LZ4"); } void -InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification *compression_spec) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 74595db1b98..69a3d9c171f 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -16,7 +16,7 @@ #include "compress_io.h" -extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); -extern void InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec); +extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification *compression_spec); +extern void InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification *compression_spec); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index f497ec60407..263995a2b7a 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -183,7 +183,7 @@ my %pgdump_runs = ( "$tempdir/compression_lz4_dir/blobs.toc.lz4", ], }, - # Verify that data files where compressed + # Verify that data files were compressed glob_patterns => [ "$tempdir/compression_lz4_dir/toc.dat", "$tempdir/compression_lz4_dir/*.dat.lz4", -- 2.25.1 --e7jIye1Ygp5H0AIi Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0007-TMP-pg_dump-use-lz4-by-default-for-CI-only.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 7/8] f! @ 2023-01-14 16:45 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2023-01-14 16:45 UTC (permalink / raw) --- src/bin/pg_dump/compress_io.c | 42 ++++++++++++++------------- src/bin/pg_dump/compress_io.h | 3 +- src/bin/pg_dump/compress_lz4.c | 34 +++++++++++----------- src/bin/pg_dump/compress_lz4.h | 4 +-- src/bin/pg_dump/pg_backup_directory.c | 10 +++++-- src/bin/pg_dump/t/002_pg_dump.pl | 2 +- 6 files changed, 51 insertions(+), 44 deletions(-) diff --git a/src/bin/pg_dump/compress_io.c b/src/bin/pg_dump/compress_io.c index 95b18843080..01bf3df0db0 100644 --- a/src/bin/pg_dump/compress_io.c +++ b/src/bin/pg_dump/compress_io.c @@ -39,13 +39,14 @@ * * The compressed stream API is a wrapper around the C standard fopen() and * libz's gzopen() APIs and custom LZ4 calls which provide similar - * libz's gzopen() APIs. It allows you to use the same functions for - * compressed and uncompressed streams. cfopen_read() first tries to open - * the file with given name, and if it fails, it tries to open the same - * file with the .gz suffix. cfopen_write() opens a file for writing, an - * extra argument specifies if the file should be compressed, and adds the - * .gz suffix to the filename if so. This allows you to easily handle both - * compressed and uncompressed files. + * functionality. It allows you to use the same functions for compressed and + * uncompressed streams. cfopen_read() first tries to open the file with given + * name, and if it fails, it tries to open the same file with the .gz suffix, + * failing that it tries to open the same file with the .lz4 suffix. + * cfopen_write() opens a file for writing, an extra argument specifies the + * method to use should the file be compressed, and adds the appropriate + * suffix, .gz or .lz4, to the filename if so. This allows you to easily handle + * both compressed and uncompressed files. * * IDENTIFICATION * src/bin/pg_dump/compress_io.c @@ -335,6 +336,14 @@ InitCompressFileHandle(const pg_compress_specification *compression_spec) return CFH; } +static bool +check_compressed_file(const char *path, char **fname, char *ext) +{ + free_keep_errno(*fname); + *fname = psprintf("%s.%s", path, ext); + return (access(*fname, F_OK) == 0); +} + /* * Open a file for reading. 'path' is the file to open, and 'mode' should * be either "r" or "rb". @@ -348,7 +357,7 @@ InitCompressFileHandle(const pg_compress_specification *compression_spec) * */ CompressFileHandle * -InitDiscoverCompressFileHandle(const char *path, const char *mode) +InitDiscoverCompressFileHandle(const char *path, const char *mode, pg_compress_algorithm alg) { CompressFileHandle *CFH = NULL; struct stat st; @@ -366,20 +375,13 @@ InitDiscoverCompressFileHandle(const char *path, const char *mode) bool exists; exists = (stat(path, &st) == 0); - /* avoid unused warning if it is not build with compression */ + /* avoid unused warning if it is not built with compression */ if (exists) compression_spec.algorithm = PG_COMPRESSION_NONE; -#ifdef HAVE_LIBZ - if (!exists) - { - free_keep_errno(fname); - fname = psprintf("%s.gz", path); - exists = (stat(fname, &st) == 0); - - if (exists) - compression_spec.algorithm = PG_COMPRESSION_GZIP; - } -#endif + else if (alg == PG_COMPRESSION_GZIP && check_compressed_file(path, &fname, "gz")) + compression_spec.algorithm = alg; + else if (alg == PG_COMPRESSION_LZ4 && check_compressed_file(path, &fname, "lz4")) + compression_spec.algorithm = alg; } CFH = InitCompressFileHandle(&compression_spec); diff --git a/src/bin/pg_dump/compress_io.h b/src/bin/pg_dump/compress_io.h index afe6b22efaf..2600182c469 100644 --- a/src/bin/pg_dump/compress_io.h +++ b/src/bin/pg_dump/compress_io.h @@ -93,7 +93,8 @@ extern CompressFileHandle *InitCompressFileHandle( const pg_compress_specification *compression_spec); extern CompressFileHandle *InitDiscoverCompressFileHandle(const char *path, - const char *mode); + const char *mode, + pg_compress_algorithm alg); extern int DestroyCompressFileHandle(CompressFileHandle *CFH); #endif diff --git a/src/bin/pg_dump/compress_lz4.c b/src/bin/pg_dump/compress_lz4.c index c97e16187a0..0e259a6251a 100644 --- a/src/bin/pg_dump/compress_lz4.c +++ b/src/bin/pg_dump/compress_lz4.c @@ -117,13 +117,13 @@ EndCompressorLZ4(ArchiveHandle *AH, CompressorState *cs) /* Public routines that support LZ4 compressed data I/O */ void -InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec) +InitCompressorLZ4(CompressorState *cs, const pg_compress_specification *compression_spec) { cs->readData = ReadDataFromArchiveLZ4; cs->writeData = WriteDataToArchiveLZ4; cs->end = EndCompressorLZ4; - cs->compression_spec = compression_spec; + cs->compression_spec = *compression_spec; /* Will be lazy init'd */ cs->private_data = pg_malloc0(sizeof(LZ4CompressorState)); @@ -189,7 +189,7 @@ LZ4File_get_error(CompressFileHandle *CFH) /* * Prepare an already alloc'ed LZ4File struct for subsequent calls. * - * It creates the nessary contexts for the operations. When compressing, + * It creates the necessary contexts for the operations. When compressing, * it additionally writes the LZ4 header in the output stream. */ static int @@ -228,7 +228,7 @@ LZ4File_init(LZ4File * fs, int size, bool compressing) if (fwrite(fs->buffer, 1, status, fs->fp) != status) { - errno = errno ? : ENOSPC; + errno = errno ? errno : ENOSPC; return 1; } } @@ -255,7 +255,7 @@ LZ4File_init(LZ4File * fs, int size, bool compressing) /* * Read already decompressed content from the overflow buffer into 'ptr' up to * 'size' bytes, if available. If the eol_flag is set, then stop at the first - * occurance of the new line char prior to 'size' bytes. + * occurrence of the new line char prior to 'size' bytes. * * Any unread content in the overflow buffer, is moved to the beginning. */ @@ -309,10 +309,10 @@ LZ4File_read_internal(LZ4File * fs, void *ptr, int ptrsize, bool eol_flag) void *readbuf; /* Lazy init */ - if (!fs->inited && LZ4File_init(fs, size, false /* decompressing */ )) + if (LZ4File_init(fs, size, false /* decompressing */ )) return -1; - /* Verfiy that there is enough space in the outbuf */ + /* Verify that there is enough space in the outbuf */ if (size > fs->buflen) { fs->buflen = size; @@ -363,10 +363,10 @@ LZ4File_read_internal(LZ4File * fs, void *ptr, int ptrsize, bool eol_flag) if (outlen > 0 && dsize < size && eol_found == false) { char *p; - size_t lib = (eol_flag == 0) ? size - dsize : size - 1 - dsize; + size_t lib = eol_flag ? size - 1 - dsize : size - dsize ; size_t len = outlen < lib ? outlen : lib; - if (eol_flag == true && + if (eol_flag && (p = memchr(fs->buffer, '\n', outlen)) && (size_t) (p - fs->buffer + 1) <= len) { @@ -377,7 +377,7 @@ LZ4File_read_internal(LZ4File * fs, void *ptr, int ptrsize, bool eol_flag) memcpy((char *) ptr + dsize, fs->buffer, len); dsize += len; - /* move what did not fit, if any, at the begining of the buf */ + /* move what did not fit, if any, at the beginning of the buf */ if (len < outlen) memmove(fs->buffer, fs->buffer + len, outlen - len); outlen -= len; @@ -414,7 +414,7 @@ LZ4File_write(const void *ptr, size_t size, CompressFileHandle *CFH) size_t status; int remaining = size; - if (!fs->inited && LZ4File_init(fs, size, true)) + if (LZ4File_init(fs, size, true)) return -1; while (remaining > 0) @@ -433,7 +433,7 @@ LZ4File_write(const void *ptr, size_t size, CompressFileHandle *CFH) if (fwrite(fs->buffer, 1, status, fs->fp) != status) { - errno = errno ? : ENOSPC; + errno = errno ? errno : ENOSPC; return 1; } } @@ -520,7 +520,7 @@ LZ4File_close(CompressFileHandle *CFH) LZ4F_getErrorName(status)); else if ((ret = fwrite(fs->buffer, 1, status, fs->fp)) != status) { - errno = errno ? : ENOSPC; + errno = errno ? errno : ENOSPC; WRITE_ERROR_EXIT; } @@ -582,7 +582,7 @@ LZ4File_open_write(const char *path, const char *mode, CompressFileHandle *CFH) } void -InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification *compression_spec) { LZ4File *lz4fp; @@ -596,7 +596,7 @@ InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compres CFH->close_func = LZ4File_close; CFH->get_error_func = LZ4File_get_error; - CFH->compression_spec = compression_spec; + CFH->compression_spec = *compression_spec; lz4fp = pg_malloc0(sizeof(*lz4fp)); if (CFH->compression_spec.level >= 0) lz4fp->prefs.compressionLevel = CFH->compression_spec.level; @@ -605,13 +605,13 @@ InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compres } #else /* USE_LZ4 */ void -InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec) +InitCompressorLZ4(CompressorState *cs, const pg_compress_specification *compression_spec) { pg_fatal("this build does not support compression with %s", "LZ4"); } void -InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec) +InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification *compression_spec) { pg_fatal("this build does not support compression with %s", "LZ4"); } diff --git a/src/bin/pg_dump/compress_lz4.h b/src/bin/pg_dump/compress_lz4.h index 74595db1b98..69a3d9c171f 100644 --- a/src/bin/pg_dump/compress_lz4.h +++ b/src/bin/pg_dump/compress_lz4.h @@ -16,7 +16,7 @@ #include "compress_io.h" -extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification compression_spec); -extern void InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification compression_spec); +extern void InitCompressorLZ4(CompressorState *cs, const pg_compress_specification *compression_spec); +extern void InitCompressLZ4(CompressFileHandle *CFH, const pg_compress_specification *compression_spec); #endif /* _COMPRESS_LZ4_H_ */ diff --git a/src/bin/pg_dump/pg_backup_directory.c b/src/bin/pg_dump/pg_backup_directory.c index 2d4baf58c22..4845fd9368c 100644 --- a/src/bin/pg_dump/pg_backup_directory.c +++ b/src/bin/pg_dump/pg_backup_directory.c @@ -201,7 +201,8 @@ InitArchiveFmt_Directory(ArchiveHandle *AH) setFilePath(AH, fname, "toc.dat"); - tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R); + tocFH = InitDiscoverCompressFileHandle(fname, PG_BINARY_R, + PG_COMPRESSION_NONE); if (tocFH == NULL) pg_fatal("could not open input file \"%s\": %m", fname); @@ -390,7 +391,8 @@ _PrintFileData(ArchiveHandle *AH, char *filename) if (!filename) return; - CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R); + CFH = InitDiscoverCompressFileHandle(filename, PG_BINARY_R, + AH->compression_spec.algorithm); if (!CFH) pg_fatal("could not open input file \"%s\": %m", filename); @@ -442,7 +444,8 @@ _LoadLOs(ArchiveHandle *AH) setFilePath(AH, tocfname, "blobs.toc"); - CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R); + CFH = ctx->LOsTocFH = InitDiscoverCompressFileHandle(tocfname, PG_BINARY_R, + AH->compression_spec.algorithm); if (ctx->LOsTocFH == NULL) pg_fatal("could not open large object TOC file \"%s\" for input: %m", @@ -783,6 +786,7 @@ _PrepParallelRestore(ArchiveHandle *AH) { /* It might be compressed */ strlcat(fname, ".gz", sizeof(fname)); + // XXX if (stat(fname, &st) == 0) te->dataLength = st.st_size; } diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index f497ec60407..263995a2b7a 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -183,7 +183,7 @@ my %pgdump_runs = ( "$tempdir/compression_lz4_dir/blobs.toc.lz4", ], }, - # Verify that data files where compressed + # Verify that data files were compressed glob_patterns => [ "$tempdir/compression_lz4_dir/toc.dat", "$tempdir/compression_lz4_dir/*.dat.lz4", -- 2.25.1 --aBaYPhOdNx+t7mr3 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0008-TMP-pg_dump-use-lz4-by-default-for-CI-only.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 3/8] f! @ 2023-01-15 22:27 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2023-01-15 22:27 UTC (permalink / raw) --- src/bin/pg_dump/pg_backup_archiver.c | 1 - 1 file changed, 1 deletion(-) diff --git a/src/bin/pg_dump/pg_backup_archiver.c b/src/bin/pg_dump/pg_backup_archiver.c index 5164b57f042..d4f1e09fce6 100644 --- a/src/bin/pg_dump/pg_backup_archiver.c +++ b/src/bin/pg_dump/pg_backup_archiver.c @@ -385,7 +385,6 @@ RestoreArchive(Archive *AHX) */ supports_compression = true; if (AH->compression_spec.algorithm != PG_COMPRESSION_NONE && - AH->compression_spec.algorithm == PG_COMPRESSION_GZIP && AH->PrintTocDataPtr != NULL) { for (te = AH->toc->next; te != AH->toc; te = te->next) -- 2.25.1 --aBaYPhOdNx+t7mr3 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0004-Introduce-Compressor-API-in-pg_dump.patch" ^ permalink raw reply [nested|flat] 16+ messages in thread
* [PATCH 2/2] f @ 2024-03-02 13:23 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 16+ messages in thread From: Justin Pryzby @ 2024-03-02 13:23 UTC (permalink / raw) --- src/backend/commands/tablecmds.c | 74 +++++++++++--------------------- 1 file changed, 25 insertions(+), 49 deletions(-) diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index cf595329b6c..3f26379b1bd 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -183,8 +183,8 @@ typedef struct AlteredTableInfo List *afterStmts; /* List of utility command parsetrees */ bool verify_new_notnull; /* T if we should recheck NOT NULL */ int rewrite; /* Reason for forced rewrite, if any */ - Oid newAccessMethod; /* new access method; 0 means no change */ - bool defaultAccessMethod; /* true if SET ACCESS METHOD DEFAULT */ + bool chgAccessMethod; /* T if SET ACCESS METHOD is used */ + Oid newAccessMethod; /* new access method; 0 means no change - if above is true */ Oid newTableSpace; /* new tablespace; 0 means no change */ bool chgPersistence; /* T if SET LOGGED/UNLOGGED is used */ char newrelpersistence; /* if above is true */ @@ -590,8 +590,7 @@ static ObjectAddress ATExecClusterOn(Relation rel, const char *indexName, LOCKMODE lockmode); static void ATExecDropCluster(Relation rel, LOCKMODE lockmode); static void ATPrepSetAccessMethod(AlteredTableInfo *tab, Relation rel, const char *amname); -static void ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethod, - bool defaultAccessMethod); +static void ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethod); static bool ATPrepChangePersistence(Relation rel, bool toLogged); static void ATPrepSetTableSpace(AlteredTableInfo *tab, Relation rel, const char *tablespacename, LOCKMODE lockmode); @@ -5053,7 +5052,7 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, ATSimplePermissions(cmd->subtype, rel, ATT_TABLE | ATT_MATVIEW); /* check if another access method change was already requested */ - if (OidIsValid(tab->newAccessMethod)) + if (tab->chgAccessMethod) ereport(ERROR, (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), errmsg("cannot have multiple SET ACCESS METHOD subcommands"))); @@ -5413,9 +5412,8 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, * Only do this for partitioned tables, for which this is just a * catalog change. Tables with storage are handled by Phase 3. */ - if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) - ATExecSetAccessMethodNoStorage(rel, tab->newAccessMethod, - tab->defaultAccessMethod); + if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE && tab->chgAccessMethod) + ATExecSetAccessMethodNoStorage(rel, tab->newAccessMethod); break; case AT_SetTableSpace: /* SET TABLESPACE */ @@ -5821,7 +5819,7 @@ ATRewriteTables(AlterTableStmt *parsetree, List **wqueue, LOCKMODE lockmode, * Select destination access method (same as original unless user * requested a change) */ - if (OidIsValid(tab->newAccessMethod)) + if (tab->chgAccessMethod) NewAccessMethod = tab->newAccessMethod; else NewAccessMethod = OldHeap->rd_rel->relam; @@ -6408,7 +6406,7 @@ ATGetQueueEntry(List **wqueue, Relation rel) tab->relkind = rel->rd_rel->relkind; tab->oldDesc = CreateTupleDescCopyConstr(RelationGetDescr(rel)); tab->newAccessMethod = InvalidOid; - tab->defaultAccessMethod = false; + tab->chgAccessMethod = false; tab->newTableSpace = InvalidOid; tab->newrelpersistence = RELPERSISTENCE_PERMANENT; tab->chgPersistence = false; @@ -15225,39 +15223,25 @@ ATPrepSetAccessMethod(AlteredTableInfo *tab, Relation rel, const char *amname) { Oid amoid; - /* Check that the table access method exists */ - amoid = get_table_am_oid(amname ? amname : default_table_access_method, - false); + /* + * Check that the table access method exists. + * Use the access method, if specified, otherwise (when not specified) 0 + * for partitioned tables or the configured default AM. + */ + if (amname != NULL) + amoid = get_table_am_oid(amname, false); + else if (rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + amoid = 0; + else + amoid = get_table_am_oid(default_table_access_method, false); - if (RELKIND_HAS_STORAGE(rel->rd_rel->relkind) && - rel->rd_rel->relam == amoid) + if (rel->rd_rel->relam == amoid) return; - /* Partitioned tables are handled here */ - if (!RELKIND_HAS_STORAGE(rel->rd_rel->relkind)) - { - if (amname == NULL) - { - /* DEFAULT implied, leave if relam is already InvalidOid */ - if (!OidIsValid(rel->rd_rel->relam)) - return; - } - else - { - /* - * Access method set by query, leave if only this matches with its - * relam. - */ - if (OidIsValid(rel->rd_rel->relam) && - amoid == rel->rd_rel->relam) - return; - } - } - /* Save info for Phase 3 to do the real work */ tab->rewrite |= AT_REWRITE_ACCESS_METHOD; tab->newAccessMethod = amoid; - tab->defaultAccessMethod = (amname == NULL); + tab->chgAccessMethod = true; } /* @@ -15587,12 +15571,10 @@ ATExecSetTableSpace(Oid tableOid, Oid newTableSpace, LOCKMODE lockmode) * storage that have an interest in preserving AM. * * Since these have no storage, setting the access method is a catalog only - * operation. defaultAccessMethod tracks if a default value is wanted for - * the access method. + * operation. */ static void -ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethodId, - bool defaultAccessMethod) +ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethodId) { Relation pg_class; Oid oldAccessMethodId; @@ -15600,12 +15582,6 @@ ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethodId, Form_pg_class rd_rel; Oid reloid = RelationGetRelid(rel); - if (!OidIsValid(newAccessMethodId)) - { - /* if the access method is unchanged, leave */ - return; - } - /* * Shouldn't be called on relations having storage; these are processed in * phase 3. @@ -15622,7 +15598,7 @@ ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethodId, /* Update the pg_class row. */ oldAccessMethodId = rd_rel->relam; - rd_rel->relam = defaultAccessMethod ? InvalidOid : newAccessMethodId; + rd_rel->relam = newAccessMethodId; /* Leave if no update required */ if (rd_rel->relam == oldAccessMethodId) @@ -15640,7 +15616,7 @@ ATExecSetAccessMethodNoStorage(Relation rel, Oid newAccessMethodId, * that this has to compare the previous value stored in pg_class with the * new one. */ - if (!OidIsValid(oldAccessMethodId) && OidIsValid(rd_rel->relam)) + if (!OidIsValid(oldAccessMethodId) && OidIsValid(newAccessMethodId)) { ObjectAddress relobj, referenced; -- 2.42.0 --Ssywp6WJwi1+mehD-- ^ permalink raw reply [nested|flat] 16+ messages in thread
end of thread, other threads:[~2024-03-02 13:23 UTC | newest] Thread overview: 16+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-03-31 00:43 [PATCH v6 01/10] doc: btree deduplication Justin Pryzby <[email protected]> 2021-11-18 03:49 [PATCH 2/4] f! Justin Pryzby <[email protected]> 2021-11-18 04:03 [PATCH 4/4] f! Justin Pryzby <[email protected]> 2022-04-01 12:00 [PATCH 3/4] f! Justin Pryzby <[email protected]> 2022-07-04 21:11 [PATCH] f Justin Pryzby <[email protected]> 2022-07-28 15:57 [PATCH 2/2] f! Justin Pryzby <[email protected]> 2022-07-28 15:57 [PATCH 2/2] f! Justin Pryzby <[email protected]> 2022-07-28 15:57 [PATCH 2/2] f! Justin Pryzby <[email protected]> 2022-09-09 12:52 [PATCH 2/2] f Justin Pryzby <[email protected]> 2023-01-07 00:28 [PATCH] f! Justin Pryzby <[email protected]> 2023-01-14 16:31 [PATCH 5/8] f! Justin Pryzby <[email protected]> 2023-01-14 16:31 [PATCH 4/7] f! Justin Pryzby <[email protected]> 2023-01-14 16:45 [PATCH 7/8] f! Justin Pryzby <[email protected]> 2023-01-14 16:45 [PATCH 6/7] f! Justin Pryzby <[email protected]> 2023-01-15 22:27 [PATCH 3/8] f! Justin Pryzby <[email protected]> 2024-03-02 13:23 [PATCH 2/2] f Justin Pryzby <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox