agora inbox for [email protected]help / color / mirror / Atom feed
[PATCH v13 6/6] snapshot scalability: cache snapshots using a xact completion counter. 21+ messages / 2 participants [nested] [flat]
* [PATCH v13 6/6] snapshot scalability: cache snapshots using a xact completion counter. @ 2020-07-15 22:35 Andres Freund <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Andres Freund @ 2020-07-15 22:35 UTC (permalink / raw) Previous commits made it faster/more scalable to compute snapshots. But not building a snapshot is still faster. Now that GetSnapshotData() does not maintain RecentGlobal* anymore, that is actually not too hard: This commit introduces xactCompletionCount, which tracks the number of top-level transactions with xids (i.e. which may have modified the database) that completed in some form since the start of the server. We can avoid rebuilding the snapshot's contents whenever the current xactCompletionCount is the same as it was when the snapshot was originally built. Currently this check happens while holding ProcArrayLock. While it's likely possible to perform the check before acquiring ProcArrayLock, it's too complicated for now. Author: Andres Freund Reviewed-By: Robert Haas, Thomas Munro, David Rowley Discussion: https://postgr.es/m/[email protected] --- src/include/access/transam.h | 9 ++ src/include/utils/snapshot.h | 7 ++ src/backend/replication/logical/snapbuild.c | 1 + src/backend/storage/ipc/procarray.c | 125 ++++++++++++++++---- src/backend/utils/time/snapmgr.c | 4 + 5 files changed, 126 insertions(+), 20 deletions(-) diff --git a/src/include/access/transam.h b/src/include/access/transam.h index b32044153b0..2f1f144db4d 100644 --- a/src/include/access/transam.h +++ b/src/include/access/transam.h @@ -231,6 +231,15 @@ typedef struct VariableCacheData FullTransactionId latestCompletedXid; /* newest full XID that has * committed or aborted */ + /* + * Number of top-level transactions with xids (i.e. which may have + * modified the database) that completed in some form since the start of + * the server. This currently is solely used to check whether + * GetSnapshotData() needs to recompute the contents of the snapshot, or + * not. There are likely other users of this. Always above 1. + */ + uint64 xactCompletionCount; + /* * These fields are protected by XactTruncationLock */ diff --git a/src/include/utils/snapshot.h b/src/include/utils/snapshot.h index 35b1f05bea6..dea072e5edf 100644 --- a/src/include/utils/snapshot.h +++ b/src/include/utils/snapshot.h @@ -207,6 +207,13 @@ typedef struct SnapshotData TimestampTz whenTaken; /* timestamp when snapshot was taken */ XLogRecPtr lsn; /* position in the WAL stream when taken */ + + /* + * The transaction completion count at the time GetSnapshotData() built + * this snapshot. Allows to avoid re-computing static snapshots when no + * transactions completed since the last GetSnapshotData(). + */ + uint64 snapXactCompletionCount; } SnapshotData; #endif /* SNAPSHOT_H */ diff --git a/src/backend/replication/logical/snapbuild.c b/src/backend/replication/logical/snapbuild.c index e9701ea7221..9d5d68f3fa7 100644 --- a/src/backend/replication/logical/snapbuild.c +++ b/src/backend/replication/logical/snapbuild.c @@ -524,6 +524,7 @@ SnapBuildBuildSnapshot(SnapBuild *builder) snapshot->curcid = FirstCommandId; snapshot->active_count = 0; snapshot->regd_count = 0; + snapshot->snapXactCompletionCount = 0; return snapshot; } diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c index 8e8049d9715..ac62343c1bb 100644 --- a/src/backend/storage/ipc/procarray.c +++ b/src/backend/storage/ipc/procarray.c @@ -407,6 +407,7 @@ CreateSharedProcArray(void) procArray->lastOverflowedXid = InvalidTransactionId; procArray->replication_slot_xmin = InvalidTransactionId; procArray->replication_slot_catalog_xmin = InvalidTransactionId; + ShmemVariableCache->xactCompletionCount = 1; } allProcs = ProcGlobal->allProcs; @@ -534,6 +535,9 @@ ProcArrayRemove(PGPROC *proc, TransactionId latestXid) /* Advance global latestCompletedXid while holding the lock */ MaintainLatestCompletedXid(latestXid); + /* Same with xactCompletionCount */ + ShmemVariableCache->xactCompletionCount++; + ProcGlobal->xids[proc->pgxactoff] = 0; ProcGlobal->subxidStates[proc->pgxactoff].overflowed = false; ProcGlobal->subxidStates[proc->pgxactoff].count = 0; @@ -668,6 +672,7 @@ ProcArrayEndTransactionInternal(PGPROC *proc, TransactionId latestXid) { size_t pgxactoff = proc->pgxactoff; + Assert(LWLockHeldByMe(ProcArrayLock)); Assert(TransactionIdIsValid(ProcGlobal->xids[pgxactoff])); Assert(ProcGlobal->xids[pgxactoff] == proc->xid); @@ -699,6 +704,9 @@ ProcArrayEndTransactionInternal(PGPROC *proc, TransactionId latestXid) /* Also advance global latestCompletedXid while holding the lock */ MaintainLatestCompletedXid(latestXid); + + /* Same with xactCompletionCount */ + ShmemVariableCache->xactCompletionCount++; } /* @@ -1913,6 +1921,93 @@ GetMaxSnapshotSubxidCount(void) return TOTAL_MAX_CACHED_SUBXIDS; } +/* + * Initialize old_snapshot_threshold specific parts of a newly build snapshot. + */ +static void +GetSnapshotDataInitOldSnapshot(Snapshot snapshot) +{ + if (!OldSnapshotThresholdActive()) + { + /* + * If not using "snapshot too old" feature, fill related fields with + * dummy values that don't require any locking. + */ + snapshot->lsn = InvalidXLogRecPtr; + snapshot->whenTaken = 0; + } + else + { + /* + * Capture the current time and WAL stream location in case this + * snapshot becomes old enough to need to fall back on the special + * "old snapshot" logic. + */ + snapshot->lsn = GetXLogInsertRecPtr(); + snapshot->whenTaken = GetSnapshotCurrentTimestamp(); + MaintainOldSnapshotTimeMapping(snapshot->whenTaken, snapshot->xmin); + } +} + +/* + * Helper function for GetSnapshotData() that check if the bulk of the + * visibility information in the snapshot is still valid. If so, it updates + * the fields that need to change and returns true. Otherwise it returns + * false. + * + * This very likely can be evolved to not need ProcArrayLock held (at very + * least in the case we already hold a snapshot), but that's for another day. + */ +static bool +GetSnapshotDataReuse(Snapshot snapshot) +{ + uint64 curXactCompletionCount; + + Assert(LWLockHeldByMe(ProcArrayLock)); + + if (unlikely(snapshot->snapXactCompletionCount == 0)) + return false; + + curXactCompletionCount = ShmemVariableCache->xactCompletionCount; + if (curXactCompletionCount != snapshot->snapXactCompletionCount) + return false; + + /* + * If the current xactCompletionCount is still the same as it was at the + * time the snapshot was built, we can be sure that rebuilding the + * contents of the snapshot the hard way would result in the same snapshot + * contents: + * + * As explained in transam/README, the set of xids considered running by + * GetSnapshotData() cannot change while ProcArrayLock is held. Snapshot + * contents only depend on transactions with xids and xactCompletionCount + * is incremented whenever a transaction with an xid finishes (while + * holding ProcArrayLock) exclusively). Thus the xactCompletionCount check + * ensures we would detect if the snapshot would have changed. + * + * As the snapshot contents are the same as it was before, it is is safe + * to re-enter the snapshot's xmin into the PGPROC array. None of the rows + * visible under the snapshot could already have been removed (that'd + * require the set of running transactions to change) and it fulfills the + * requirement that concurrent GetSnapshotData() calls yield the same + * xmin. + */ + if (!TransactionIdIsValid(MyProc->xmin)) + MyProc->xmin = TransactionXmin = snapshot->xmin; + + RecentXmin = snapshot->xmin; + Assert(TransactionIdPrecedesOrEquals(TransactionXmin, RecentXmin)); + + snapshot->curcid = GetCurrentCommandId(false); + snapshot->active_count = 0; + snapshot->regd_count = 0; + snapshot->copied = false; + + GetSnapshotDataInitOldSnapshot(snapshot); + + return true; +} + /* * GetSnapshotData -- returns information about running transactions. * @@ -1961,6 +2056,7 @@ GetSnapshotData(Snapshot snapshot) TransactionId oldestxid; int mypgxactoff; TransactionId myxid; + uint64 curXactCompletionCount; TransactionId replication_slot_xmin = InvalidTransactionId; TransactionId replication_slot_catalog_xmin = InvalidTransactionId; @@ -2005,12 +2101,19 @@ GetSnapshotData(Snapshot snapshot) */ LWLockAcquire(ProcArrayLock, LW_SHARED); + if (GetSnapshotDataReuse(snapshot)) + { + LWLockRelease(ProcArrayLock); + return snapshot; + } + latest_completed = ShmemVariableCache->latestCompletedXid; mypgxactoff = MyProc->pgxactoff; myxid = other_xids[mypgxactoff]; Assert(myxid == MyProc->xid); oldestxid = ShmemVariableCache->oldestXid; + curXactCompletionCount = ShmemVariableCache->xactCompletionCount; /* xmax is always latestCompletedXid + 1 */ xmax = XidFromFullTransactionId(latest_completed); @@ -2264,6 +2367,7 @@ GetSnapshotData(Snapshot snapshot) snapshot->xcnt = count; snapshot->subxcnt = subcount; snapshot->suboverflowed = suboverflowed; + snapshot->snapXactCompletionCount = curXactCompletionCount; snapshot->curcid = GetCurrentCommandId(false); @@ -2275,26 +2379,7 @@ GetSnapshotData(Snapshot snapshot) snapshot->regd_count = 0; snapshot->copied = false; - if (old_snapshot_threshold < 0) - { - /* - * If not using "snapshot too old" feature, fill related fields with - * dummy values that don't require any locking. - */ - snapshot->lsn = InvalidXLogRecPtr; - snapshot->whenTaken = 0; - } - else - { - /* - * Capture the current time and WAL stream location in case this - * snapshot becomes old enough to need to fall back on the special - * "old snapshot" logic. - */ - snapshot->lsn = GetXLogInsertRecPtr(); - snapshot->whenTaken = GetSnapshotCurrentTimestamp(); - MaintainOldSnapshotTimeMapping(snapshot->whenTaken, xmin); - } + GetSnapshotDataInitOldSnapshot(snapshot); return snapshot; } diff --git a/src/backend/utils/time/snapmgr.c b/src/backend/utils/time/snapmgr.c index 689a3b6a597..09ea03c2063 100644 --- a/src/backend/utils/time/snapmgr.c +++ b/src/backend/utils/time/snapmgr.c @@ -595,6 +595,8 @@ SetTransactionSnapshot(Snapshot sourcesnap, VirtualTransactionId *sourcevxid, CurrentSnapshot->takenDuringRecovery = sourcesnap->takenDuringRecovery; /* NB: curcid should NOT be copied, it's a local matter */ + CurrentSnapshot->snapXactCompletionCount = 0; + /* * Now we have to fix what GetSnapshotData did with MyProc->xmin and * TransactionXmin. There is a race condition: to make sure we are not @@ -670,6 +672,7 @@ CopySnapshot(Snapshot snapshot) newsnap->regd_count = 0; newsnap->active_count = 0; newsnap->copied = true; + newsnap->snapXactCompletionCount = 0; /* setup XID array */ if (snapshot->xcnt > 0) @@ -2207,6 +2210,7 @@ RestoreSnapshot(char *start_address) snapshot->curcid = serialized_snapshot.curcid; snapshot->whenTaken = serialized_snapshot.whenTaken; snapshot->lsn = serialized_snapshot.lsn; + snapshot->snapXactCompletionCount = 0; /* Copy XIDs, if present. */ if (serialized_snapshot.xcnt > 0) -- 2.25.0.114.g5b0ca878e0 --4il4azyv6rmablac-- ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 2/3] cirrus/windows: add compiler_warnings_script @ 2022-02-13 00:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-02-13 00:59 UTC (permalink / raw) ci-os-only: windows https://cirrus-ci.com/task/6316260295180288 --- .cirrus.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index eda8ac9596c..ea87c890cc3 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -397,7 +397,7 @@ task: - perl src/tools/msvc/mkvcbuild.pl build_script: - vcvarsall x64 - - msbuild %MSBFLAGS% pgsql.sln + - msbuild %MSBFLAGS% pgsql.sln >build.log tempinstall_script: # Installation on windows currently only completely works from src/tools/msvc - cd src/tools/msvc && perl install.pl %CIRRUS_WORKING_DIR%/tmp_install @@ -440,6 +440,11 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + always: + compiler_warnings_script: | + findstr "warning " build.log && exit /b 1 + exit /b 0 + on_failure: <<: *on_failure crashlog_artifacts: -- 2.17.1 --1XWsVB21DFCvn2e8 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-cirrus-upload-changed-html-docs-as-artifacts.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 2/8] cirrus/windows: add compiler_warnings_script @ 2022-02-13 00:59 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-02-13 00:59 UTC (permalink / raw) ci-os-only: windows https://cirrus-ci.com/task/6316260295180288 --- .cirrus.yml | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index eda8ac9596c..2d023a31d2d 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -397,7 +397,7 @@ task: - perl src/tools/msvc/mkvcbuild.pl build_script: - vcvarsall x64 - - msbuild %MSBFLAGS% pgsql.sln + - msbuild %MSBFLAGS% pgsql.sln |tee build.log tempinstall_script: # Installation on windows currently only completely works from src/tools/msvc - cd src/tools/msvc && perl install.pl %CIRRUS_WORKING_DIR%/tmp_install @@ -440,6 +440,11 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + always: + compiler_warnings_script: | + findstr "warning " build.log && exit /b 1 + exit /b 0 + on_failure: <<: *on_failure crashlog_artifacts: -- 2.17.1 --4e5ZDkbgLEOfWmLx Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-cirrus-upload-changed-html-docs-as-artifacts.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 3/7] cirrus/windows: add compiler_warnings_script @ 2022-02-20 21:01 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-02-20 21:01 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... ci-os-only: windows https://cirrus-ci.com/task/6183879907213312 https://cirrus-ci.com/task/4876271443247104 --- .cirrus.yml | 9 ++++++++- src/tools/ci/windows-compiler-warnings | 16 ++++++++++++++++ 2 files changed, 24 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index 5179353dc9d..a3af4a0a808 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -366,7 +366,8 @@ task: # ForceNoAlign prevents msbuild from introducing line-breaks for long lines # disable file tracker, we're never going to rebuild, and it slows down the # build - MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo + # -fileLoggerParameters1: write warnings to msbuild.warn.log. + MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log # If tests hang forever, cirrus eventually times out. In that case log # output etc is not uploaded, making the problem hard to debug. Of course @@ -447,6 +448,12 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + # These should be last, so all the important checks are always run + always: + # Success if the file doesn't exist or is empty, else fail + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings msbuild.warn.log + on_failure: <<: *on_failure crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..d6f9a1fc569 --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,16 @@ +#! /bin/sh +# Success if the given file doesn't exist or is empty, else fail +# This is a separate file only to avoid dealing with windows shell quoting and escaping. +set -e + +fn=$1 + +if [ -s "$fn" ] +then + # Display the file's content, then exit indicating failure + cat "$fn" + exit 1 +else + # Success + exit 0 +fi -- 2.17.1 --wLAMOaPNJ0fu1fTG Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0004-cirrus-code-coverage.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 01/23] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 ci-os-only: windows https://cirrus-ci.com/task/6183879907213312 https://cirrus-ci.com/task/4876271443247104 --- .cirrus.yml | 14 +++++++++++++- src/tools/ci/windows-compiler-warnings | 16 ++++++++++++++++ 2 files changed, 29 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index 81eb8a9996d..2be62791448 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -370,7 +370,14 @@ task: # ForceNoAlign prevents msbuild from introducing line-breaks for long lines # disable file tracker, we're never going to rebuild, and it slows down the # build - MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo + # -fileLoggerParameters1: write warnings to msbuild.warn.log. + MSBFLAGS: >- + "-consoleLoggerParameters:Summary;ForceNoAlign" + -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log + -m + -nologo + -p:TrackFileAccess=false + -verbosity:minimal # If tests hang forever, cirrus eventually times out. In that case log # output etc is not uploaded, making the problem hard to debug. Of course @@ -450,6 +457,11 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + # These should be last, so all the important checks are always run + always: + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings msbuild.warn.log + on_failure: <<: *on_failure crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..d6f9a1fc569 --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,16 @@ +#! /bin/sh +# Success if the given file doesn't exist or is empty, else fail +# This is a separate file only to avoid dealing with windows shell quoting and escaping. +set -e + +fn=$1 + +if [ -s "$fn" ] +then + # Display the file's content, then exit indicating failure + cat "$fn" + exit 1 +else + # Success + exit 0 +fi -- 2.17.1 --tKkaNMvYmhQvRCRK Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-vcregress-test-modules-contrib-with-NO_INSTAL.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 4/8] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 10 +++++++++- src/tools/ci/windows-compiler-warnings | 24 ++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index 9f2282471a9..99ac09dc679 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -451,12 +451,20 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build/meson-logs/build.txt + REM Since pipes lose exit status of the preceding command, rerun compilation, + REM without the pipe exiting now if it fails, rather than trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is always run + always: + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings build/meson-logs/build.txt build/meson-logs/build-warnings.txt + on_failure: <<: *on_failure_meson crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..685aa995d5c --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,24 @@ +#! /bin/sh +# Success if the given exists and is empty, else fail +# This is a separate file to avoid dealing with windows shell quoting and escaping, +# or giving the impression that one-liner scripts will work trivially. +set -e + +infile=$1 +outfile=$2 + +# Looks like: +# [19:39:27.130] c:\cirrus\src\backend\backup\basebackup_zstd.c(80) : warning C4715: 'bbsink_zstd_new': not all control paths return a value +# should include linker warnings? +grep ": warning " "$infile" > $outfile || + [ $? -eq 1 ] + +if [ -s "$outfile" ] +then + # Display the file's content, then exit indicating failure + cat "$outfile" + exit 1 +else + # Success + exit 0 +fi -- 2.25.1 --rPFbv4B2w2tcwGo5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-cirrus-build-docs-as-a-separate-task.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 01/10] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index f31923333ef..6ce4f393e29 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -547,12 +547,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.25.1 --sMkrXc3gAYLRVOjR Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-macos-switch-to-macos_instance-M1.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 01/19] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 ci-os-only: windows https://cirrus-ci.com/task/6183879907213312 https://cirrus-ci.com/task/4876271443247104 --- .cirrus.yml | 8 +++++++- src/tools/ci/windows-compiler-warnings | 16 ++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index f23d6cae552..bcb8d53db78 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -370,7 +370,8 @@ task: # ForceNoAlign prevents msbuild from introducing line-breaks for long lines # disable file tracker, we're never going to rebuild, and it slows down the # build - MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo + # -fileLoggerParameters1: write warnings to msbuild.warn.log. + MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log # If tests hang forever, cirrus eventually times out. In that case log # output etc is not uploaded, making the problem hard to debug. Of course @@ -450,6 +451,11 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + # These should be last, so all the important checks are always run + always: + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings msbuild.warn.log + on_failure: <<: *on_failure crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..d6f9a1fc569 --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,16 @@ +#! /bin/sh +# Success if the given file doesn't exist or is empty, else fail +# This is a separate file only to avoid dealing with windows shell quoting and escaping. +set -e + +fn=$1 + +if [ -s "$fn" ] +then + # Display the file's content, then exit indicating failure + cat "$fn" + exit 1 +else + # Success + exit 0 +fi -- 2.17.1 --Sw7tCqrGA+HQ0/zt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-vcregress-test-modules-contrib-with-NO_INSTAL.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 01/11] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 10 +++++++++- src/tools/ci/windows-compiler-warnings | 24 ++++++++++++++++++++++++ 2 files changed, 33 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index 9f2282471a9..7596b2058c6 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -451,12 +451,20 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build/meson-logs/build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is always run + always: + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings build/meson-logs/build.txt build/meson-logs/build-warnings.txt + on_failure: <<: *on_failure_meson crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..685aa995d5c --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,24 @@ +#! /bin/sh +# Success if the given exists and is empty, else fail +# This is a separate file to avoid dealing with windows shell quoting and escaping, +# or giving the impression that one-liner scripts will work trivially. +set -e + +infile=$1 +outfile=$2 + +# Looks like: +# [19:39:27.130] c:\cirrus\src\backend\backup\basebackup_zstd.c(80) : warning C4715: 'bbsink_zstd_new': not all control paths return a value +# should include linker warnings? +grep ": warning " "$infile" > $outfile || + [ $? -eq 1 ] + +if [ -s "$outfile" ] +then + # Display the file's content, then exit indicating failure + cat "$outfile" + exit 1 +else + # Success + exit 0 +fi -- 2.25.1 --I4VOKWutKNZEOIPu Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-build-docs-as-a-separate-task.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/6] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) The goal is to fail due to warnings only after running tests. (At least historically, it's too slow to run a separate windows VM to compile with -Werror.) https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 [email protected] [email protected] [email protected] [email protected] [email protected] ci-os-only: windows --- .cirrus.tasks.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index 33646faeadf..5a2b64f64c2 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -566,12 +566,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.42.0 --Rvs4meNCoh6yKfx/ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-WIP-cirrus-windows-ccache.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/6] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) The goal is to fail due to warnings only after running tests. (At least historically, it's too slow to run a separate windows VM to compile with -Werror.) https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 [email protected] [email protected] [email protected] [email protected] [email protected] ci-os-only: windows --- .cirrus.tasks.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index 33646faeadf..5a2b64f64c2 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -566,12 +566,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.42.0 --AZUs1t8VY6GH6Ct3 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-WIP-cirrus-windows-ccache.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/7] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) The goal is to fail due to warnings only after running tests. (At least historically, it's too slow to run a separate windows VM to compile with -Werror.) https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 [email protected] [email protected] [email protected] [email protected] [email protected] ci-os-only: windows --- .cirrus.tasks.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index 33646faeadf..5a2b64f64c2 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -566,12 +566,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.42.0 --jviKHM0HgsS9J3Sv Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-windows-ccache.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/6] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) The goal is to fail due to warnings only after running tests. (At least historically, it's too slow to run a separate windows VM to compile with -Werror.) https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.tasks.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index e4e1bcfeb99..2d397448851 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -561,12 +561,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.42.0 --r+4ArEqyVZuXSt++ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-002_pg_upgrade-exercise-link-and-clone.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/8] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) The goal is to fail due to warnings only after running tests. (At least historically, it's too slow to run a separate windows VM to compile with -Werror.) https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index d3f88821a85..13213ffd304 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -589,28 +589,37 @@ task: echo 127.0.0.2 pg-loadbalancetest >> c:\Windows\System32\Drivers\etc\hosts echo 127.0.0.3 pg-loadbalancetest >> c:\Windows\System32\Drivers\etc\hosts type c:\Windows\System32\Drivers\etc\hosts # Use /DEBUG:FASTLINK to avoid high memory usage during linking configure_script: | vcvarsall x64 meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: path: "crashlog-*.txt" type: text/plain task: << : *WINDOWS_ENVIRONMENT_BASE name: Windows - Server 2019, MinGW64 - Meson -- 2.34.1 --NvorufYXl2Cnpa+A Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/9] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index 1204824d2eb..eefc5c21fe6 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -566,12 +566,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.25.1 --61jdw2sOBCFtR2d/ Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/7] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index 69837bcd5ad..fd461048372 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -564,12 +564,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.25.1 --O98KdSgI27dgYlM5 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-macos-update-to-macos-ventura.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 01/21] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 ci-os-only: windows https://cirrus-ci.com/task/6183879907213312 https://cirrus-ci.com/task/4876271443247104 --- .cirrus.yml | 8 +++++++- src/tools/ci/windows-compiler-warnings | 16 ++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index f23d6cae552..bcb8d53db78 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -370,7 +370,8 @@ task: # ForceNoAlign prevents msbuild from introducing line-breaks for long lines # disable file tracker, we're never going to rebuild, and it slows down the # build - MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo + # -fileLoggerParameters1: write warnings to msbuild.warn.log. + MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log # If tests hang forever, cirrus eventually times out. In that case log # output etc is not uploaded, making the problem hard to debug. Of course @@ -450,6 +451,11 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + # These should be last, so all the important checks are always run + always: + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings msbuild.warn.log + on_failure: <<: *on_failure crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..d6f9a1fc569 --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,16 @@ +#! /bin/sh +# Success if the given file doesn't exist or is empty, else fail +# This is a separate file only to avoid dealing with windows shell quoting and escaping. +set -e + +fn=$1 + +if [ -s "$fn" ] +then + # Display the file's content, then exit indicating failure + cat "$fn" + exit 1 +else + # Success + exit 0 +fi -- 2.17.1 --STPqjqpCrtky8aYs Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-vcregress-test-modules-contrib-with-NO_INSTAL.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/6] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) The goal is to fail due to warnings only after running tests. (At least historically, it's too slow to run a separate windows VM to compile with -Werror.) https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 [email protected] [email protected] [email protected] [email protected] [email protected] ci-os-only: windows --- .cirrus.tasks.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.tasks.yml b/.cirrus.tasks.yml index 1ce6c443a8c..5dd37e07aae 100644 --- a/.cirrus.tasks.yml +++ b/.cirrus.tasks.yml @@ -576,12 +576,21 @@ task: build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: -- 2.42.0 --ndI8Dx/wEX5btThR Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-windows-ccache.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 1/8] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also easy to write something that doesn't work in posix sh, since windows shell is interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 https://cirrus-ci.com/task/6241060062494720 https://cirrus-ci.com/task/6496366607204352 ci-os-only: windows --- .cirrus.yml | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/.cirrus.yml b/.cirrus.yml index 505c50f3285..60c0efc2e63 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -563,28 +563,37 @@ task: setup_additional_packages_script: | REM choco install -y --no-progress ... # Use /DEBUG:FASTLINK to avoid high memory usage during linking configure_script: | vcvarsall x64 meson setup --backend ninja --buildtype debug -Dc_link_args=/DEBUG:FASTLINK -Dcassert=true -Db_pch=true -Dextra_lib_dirs=c:\openssl\1.1\lib -Dextra_include_dirs=c:\openssl\1.1\include -DTAR=%TAR% -DPG_TEST_EXTRA="%PG_TEST_EXTRA%" build build_script: | vcvarsall x64 - ninja -C build + ninja -C build |tee build.txt + REM Since pipes lose the exit status of the preceding command, rerun the compilation + REM without the pipe, exiting now if it fails, to avoid trying to run checks + ninja -C build > nul check_world_script: | vcvarsall x64 meson test %MTEST_ARGS% --num-processes %TEST_JOBS% + # This should be last, so check_world is run even if there are warnings + always: + compiler_warnings_script: + # this avoids using metachars which would be interpretted by the windows shell + - sh -c 'if grep ": warning " build.txt; then exit 1; fi; exit 0' + on_failure: <<: *on_failure_meson crashlog_artifacts: path: "crashlog-*.txt" type: text/plain task: << : *WINDOWS_ENVIRONMENT_BASE name: Windows - Server 2019, MinGW64 - Meson -- 2.34.1 --Wo0n/IWMQc9EwEbt Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-freebsd-run-with-more-CPUs-RAM-and-do-not-rep.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 01/21] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 ci-os-only: windows https://cirrus-ci.com/task/6183879907213312 https://cirrus-ci.com/task/4876271443247104 --- .cirrus.yml | 8 +++++++- src/tools/ci/windows-compiler-warnings | 16 ++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index f23d6cae552..bcb8d53db78 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -370,7 +370,8 @@ task: # ForceNoAlign prevents msbuild from introducing line-breaks for long lines # disable file tracker, we're never going to rebuild, and it slows down the # build - MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo + # -fileLoggerParameters1: write warnings to msbuild.warn.log. + MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log # If tests hang forever, cirrus eventually times out. In that case log # output etc is not uploaded, making the problem hard to debug. Of course @@ -450,6 +451,11 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + # These should be last, so all the important checks are always run + always: + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings msbuild.warn.log + on_failure: <<: *on_failure crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..d6f9a1fc569 --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,16 @@ +#! /bin/sh +# Success if the given file doesn't exist or is empty, else fail +# This is a separate file only to avoid dealing with windows shell quoting and escaping. +set -e + +fn=$1 + +if [ -s "$fn" ] +then + # Display the file's content, then exit indicating failure + cat "$fn" + exit 1 +else + # Success + exit 0 +fi -- 2.17.1 --mln0rGgUGuXEqmuI Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-vcregress-test-modules-contrib-with-NO_INSTAL.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
* [PATCH 01/25] cirrus/windows: add compiler_warnings_script @ 2022-05-26 02:53 Justin Pryzby <[email protected]> 0 siblings, 0 replies; 21+ messages in thread From: Justin Pryzby @ 2022-05-26 02:53 UTC (permalink / raw) I'm not sure how to write this test in windows shell; it's also not easy to write it in posix sh, since windows shell is somehow interpretting && and ||... https://www.postgresql.org/message-id/20220212212310.f645c6vw3njkgxka%40alap3.anarazel.de See also: 8a1ce5e54f6d144e4f8e19af7c767b026ee0c956 ci-os-only: windows https://cirrus-ci.com/task/6183879907213312 https://cirrus-ci.com/task/4876271443247104 --- .cirrus.yml | 8 +++++++- src/tools/ci/windows-compiler-warnings | 16 ++++++++++++++++ 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100755 src/tools/ci/windows-compiler-warnings diff --git a/.cirrus.yml b/.cirrus.yml index 81eb8a9996d..da16344341b 100644 --- a/.cirrus.yml +++ b/.cirrus.yml @@ -370,7 +370,8 @@ task: # ForceNoAlign prevents msbuild from introducing line-breaks for long lines # disable file tracker, we're never going to rebuild, and it slows down the # build - MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo + # -fileLoggerParameters1: write warnings to msbuild.warn.log. + MSBFLAGS: -m -verbosity:minimal "-consoleLoggerParameters:Summary;ForceNoAlign" /p:TrackFileAccess=false -nologo -fileLoggerParameters1:warningsonly;logfile=msbuild.warn.log # If tests hang forever, cirrus eventually times out. In that case log # output etc is not uploaded, making the problem hard to debug. Of course @@ -450,6 +451,11 @@ task: cd src/tools/msvc %T_C% perl vcregress.pl ecpgcheck + # These should be last, so all the important checks are always run + always: + compiler_warnings_script: + - sh src\tools\ci\windows-compiler-warnings msbuild.warn.log + on_failure: <<: *on_failure crashlog_artifacts: diff --git a/src/tools/ci/windows-compiler-warnings b/src/tools/ci/windows-compiler-warnings new file mode 100755 index 00000000000..d6f9a1fc569 --- /dev/null +++ b/src/tools/ci/windows-compiler-warnings @@ -0,0 +1,16 @@ +#! /bin/sh +# Success if the given file doesn't exist or is empty, else fail +# This is a separate file only to avoid dealing with windows shell quoting and escaping. +set -e + +fn=$1 + +if [ -s "$fn" ] +then + # Display the file's content, then exit indicating failure + cat "$fn" + exit 1 +else + # Success + exit 0 +fi -- 2.17.1 --IS0zKkzwUGydFO0o Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-cirrus-vcregress-test-modules-contrib-with-NO_INSTAL.patch" ^ permalink raw reply [nested|flat] 21+ messages in thread
end of thread, other threads:[~2022-05-26 02:53 UTC | newest] Thread overview: 21+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2020-07-15 22:35 [PATCH v13 6/6] snapshot scalability: cache snapshots using a xact completion counter. Andres Freund <[email protected]> 2022-02-13 00:59 [PATCH 2/8] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-02-13 00:59 [PATCH 2/3] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-02-20 21:01 [PATCH 3/7] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 01/19] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/6] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/6] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 01/25] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/6] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 01/11] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 4/8] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/6] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/7] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 01/21] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/8] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 01/23] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 01/10] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/7] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/8] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 1/9] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]> 2022-05-26 02:53 [PATCH 01/21] cirrus/windows: add compiler_warnings_script Justin Pryzby <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox