public inbox for [email protected]help / color / mirror / Atom feed
[PATCH v2 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. 12+ messages / 4 participants [nested] [flat]
* [PATCH v2 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. @ 2021-03-15 16:25 Andres Freund <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Andres Freund @ 2021-03-15 16:25 UTC (permalink / raw) --- .cirrus.yml | 388 ++++++++++++++++++++++++++++++++ .dockerignore | 3 + ci/README | 36 +++ ci/docker/linux_debian_bullseye | 13 ++ ci/docker/windows_vs_2019 | 111 +++++++++ ci/freebsd_gcp_repartition.sh | 28 +++ ci/pg_ci_base.conf | 12 + ci/windows_build_config.pl | 13 ++ 8 files changed, 604 insertions(+) create mode 100644 .cirrus.yml create mode 100644 .dockerignore create mode 100644 ci/README create mode 100644 ci/docker/linux_debian_bullseye create mode 100644 ci/docker/windows_vs_2019 create mode 100755 ci/freebsd_gcp_repartition.sh create mode 100644 ci/pg_ci_base.conf create mode 100644 ci/windows_build_config.pl diff --git a/.cirrus.yml b/.cirrus.yml new file mode 100644 index 00000000000..b3321a91ae6 --- /dev/null +++ b/.cirrus.yml @@ -0,0 +1,388 @@ +env: + # accelerate initial clone, but a bit of depth so that concurrent tasks work + CIRRUS_CLONE_DEPTH: 100 + # Useful to be able to analyse what in a script takes long + CIRRUS_LOG_TIMESTAMP: true + # target to test, for all but windows + CHECK: check-world + CHECKFLAGS: -Otarget + PGCTLTIMEOUT: 120 + CCACHE_MAXSIZE: "500M" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl + + +task: + name: FreeBSD + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-freebsd-13-0 + platform: freebsd + cpu: 2 + memory: 2G + disk: 50 + env: + CCACHE_DIR: "/tmp/ccache_dir" + + ccache_cache: + folder: "/tmp/ccache_dir" + sysinfo_script: + - export || true + sysconfig_script: + - sudo sysctl kern.corefile='/tmp/%N.%P.core' + repartition_script: + - ci/freebsd_gcp_repartition.sh + create_user_script: + - pw useradd postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/lib/tcl8.6/ \ + --with-uuid=bsd \ + \ + --with-includes=/usr/local/include --with-libs=/usr/local/lib \ + CC="ccache cc"' + build_script: + - su postgres -c 'gmake -s -j3 && gmake -s -j3 -C contrib' + upload_caches: + - ccache + + tests_script: + - su postgres -c 'time gmake -s -j2 ${CHECK} ${CHECKFLAGS}' + + on_failure: + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Linux + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-bullseye + platform: linux + cpu: 4 + memory: 2G + nested_virtualization: false + env: + CCACHE_DIR: "/tmp/ccache_dir" + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + ccache_cache: + folder: "/tmp/ccache_dir" + + sysinfo_script: + - id + - uname -a + - cat /proc/cmdline + - lsblk + - ulimit -a -H + - ulimit -a -S + - export + sysconfig_script: + - useradd -m postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf + - su postgres -c 'ulimit -l -H' + - su postgres -c 'ulimit -l -S' + - echo '/tmp/%e-%s-%p.core' > /proc/sys/kernel/core_pattern + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-llvm \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-systemd \ + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ \ + --with-uuid=e2fs \ + \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" CFLAGS="-O0 -ggdb"' + build_script: + - su postgres -c 'make -s -j4 && make -s -j4 -C contrib' + upload_caches: + - ccache + + tests_script: | + su postgres -c '\ + ulimit -c unlimited; \ + make -s ${CHECK} ${CHECKFLAGS} -j8 \ + ' + + on_failure: + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile ; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: macOS + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' + osx_instance: + image: big-sur-base + env: + CIRRUS_WORKING_DIR: ${HOME}/pgsql/ + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + CCACHE_DIR: ${HOME}/ccache + HOMEBREW_CACHE: ${HOME}/homebrew-cache + PERL5LIB: ${HOME}/perl5/lib/perl5 + + sysinfo_script: + - id + - export + ccache_cache: + folder: ${CCACHE_DIR} + homebrew_cache: + folder: ${HOMEBREW_CACHE} + perl_cache: + folder: ~/perl5 + + cpan_install_script: + - perl -mIPC::Run -e 1 || cpan -T IPC::Run + - perl -mIO::Pty -e 1 || cpan -T IO::Pty + upload_caches: + - perl + core_install_script: + - sudo chmod 777 /cores + homebrew_install_script: + - brew install make coreutils ccache icu4c lz4 tcl-tk openldap + upload_caches: + - homebrew + + configure_script: | + LIBS="/usr/local/lib:$LIBS" + INCLUDES="/usr/local/include:$INCLUDES" + + INCLUDES="/usr/local/opt/openssl/include:$INCLUDES" + LIBS="/usr/local/opt/openssl/lib:$LIBS" + + PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" + INCLUDES="/usr/local/opt/icu4c/include:$INCLUDES" + LIBS="/usr/local/opt/icu4c/lib:$LIBS" + + LIBS="/usr/local/opt/openldap/lib:$LIBS" + INCLUDES="/usr/local/opt/openldap/include:$INCLUDES" + + export PKG_CONFIG_PATH + + ./configure \ + --prefix=$HOME/install \ + --with-includes="$INCLUDES" \ + --with-libs="$LIBS" \ + \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + \ + --with-lz4 \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/opt/tcl-tk/lib/ \ + --with-uuid=e2fs \ + \ + CC="ccache gcc" CFLAGS="-O0 -ggdb" + build_script: + - gmake -s -j12 && gmake -s -j12 -C contrib + upload_caches: + - ccache + + tests_script: + - ulimit -c unlimited + - ulimit -n 1024 + - gmake -s -j12 ${CHECK} ${CHECKFLAGS} + + on_failure: + cores_script: | + for corefile in $(find /cores/ -name 'core.*' 2>/dev/null) ; do + lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' ; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Windows + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + windows_container: + dockerfile: ci/docker/windows_vs_2019 + cpu: 4 + memory: 4G + env: + PROVE_FLAGS: -j10 + # The default working dir is in a directory msbuild complains about + CIRRUS_WORKING_DIR: "c:/cirrus" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + # Avoid re-installing over and over + NO_TEMP_INSTALL: 1 + + sysinfo_script: + - chcp + - systeminfo + - powershell -Command get-psdrive -psprovider filesystem + - ps: Get-Item -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' + - set + + configure_script: + - copy ci\windows_build_config.pl src\tools\msvc\config.pl + - vcvarsall x64 + - perl src/tools/msvc/mkvcbuild.pl + build_script: + - vcvarsall x64 + # Disable file tracker, we're never going to rebuild... + - msbuild -m /p:TrackFileAccess=false pgsql.sln + tempinstall_script: + # Installation on windows currently only completely works from src\tools\msvc + - cd src\tools\msvc && perl .\install.pl %CIRRUS_WORKING_DIR%\tmp_install + + check_test_script: + - perl src/tools/msvc/vcregress.pl check parallel + startcreate_test_script: + - tmp_install\bin\pg_ctl.exe initdb -D tmp_check\db -l tmp_check\initdb.log + - echo include '%TEMP_CONFIG%' >> tmp_check\db\postgresql.conf + - tmp_install\bin\pg_ctl.exe start -D tmp_check\db -l tmp_check\postmaster.log + plcheck_test_script: + - perl src/tools/msvc/vcregress.pl plcheck + isolationcheck_test_script: + - perl src/tools/msvc/vcregress.pl isolationcheck + modulescheck_test_script: + - perl src/tools/msvc/vcregress.pl modulescheck + contribcheck_test_script: + - perl src/tools/msvc/vcregress.pl contribcheck + stop_test_script: + - tmp_install\bin\pg_ctl.exe stop -D tmp_check\db -l tmp_check\postmaster.log + ssl_test_script: + - set with_ssl=openssl + - perl src/tools/msvc/vcregress.pl taptest .\src\test\ssl\ + subscriptioncheck_test_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\subscription\ + authentication_test_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\authentication\ + recoverycheck_test_script: + - perl src/tools/msvc/vcregress.pl recoverycheck + bincheck_test_script: + - perl src/tools/msvc/vcregress.pl bincheck + upgradecheck_test_script: + - perl src/tools/msvc/vcregress.pl upgradecheck + ecpgcheck_test_script: + # tries to build additional stuff + - vcvarsall x64 + # References ecpg_regression.proj in the current dir + - cd src\tools\msvc + - perl vcregress.pl ecpgcheck + + on_failure: + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: CompilerWarnings + depends_on: + - Linux + # task that did not run count as a success, so we need to recheck Linux' condition here :/ + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + container: + dockerfile: ci/docker/linux_debian_bullseye + env: + CCACHE_SIZE: "4GB" + CCACHE_DIR: "/tmp/ccache_dir" + ccache_cache: + folder: "/tmp/ccache_dir" + setup_script: + - echo "COPT=-Werror" > src/Makefile.custom + - gcc -v + - clang -v + # gcc with asserts disabled + always: + gcc_warning_script: + - ./configure --cache gcc.cache CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # gcc with asserts enabled + always: + gcc_a_warning_script: + - ./configure --cache gcc.cache --enable-cassert CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts disabled + always: + clang_warning_script: + - ./configure --cache clang.cache CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts enabled + always: + clang_a_warning_script: + - ./configure --cache clang.cache --enable-cassert CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000000..3fceab2e97b --- /dev/null +++ b/.dockerignore @@ -0,0 +1,3 @@ +# Ignore everything, except ci/ +* +!ci/* diff --git a/ci/README b/ci/README new file mode 100644 index 00000000000..566fd9fbd78 --- /dev/null +++ b/ci/README @@ -0,0 +1,36 @@ +Postgres Continuous Integration (CI) +==================================== + +Postgres has two forms of CI: + +1) All supported branches in the main postgres repository are continuously + tested via the the the buildfarm. As this covers only the main repository, + it cannot be used during development of features. + + For details see https://buildfarm.postgresql.org/ + +2) For not yet merged development work, CI can be enabled for some git hosting + providers. This allows to test patches on a number of platforms before they + are merged (or even submitted). + + +Configuring CI on personal repositories +======================================= + +Currently postgres contains CI support utilizing cirrus-ci. cirrus-ci +currently is only available for github. + + +Enabling cirrus-ci in a github repository +========================================= + +To enable cirrus-ci on a repository, go to +https://github.com/marketplace/cirrus-ci and select "Public +Repositories". Then "Install it for free" and "Complete order". The next page +allows to configure which repositories cirrus-ci has access too. Choose the +relevant repository and "Install". + +Once enabled on a repository, future commits and pull-requests in that +repository will automatically trigger CI builds. These are visible from the +commit history / PRs, and can also be viewed in the cirrus-ci UI at +https://cirrus-ci.com/github/<username>/<reponame>/ diff --git a/ci/docker/linux_debian_bullseye b/ci/docker/linux_debian_bullseye new file mode 100644 index 00000000000..f6c1782f16b --- /dev/null +++ b/ci/docker/linux_debian_bullseye @@ -0,0 +1,13 @@ +FROM debian:bullseye +RUN \ + apt-get -y update && \ + apt-get -y upgrade && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + git build-essential gcc g++ libreadline-dev flex bison make perl libipc-run-perl \ + libio-pty-perl clang llvm-dev libperl-dev libpython3-dev tcl-dev libldap2-dev \ + libicu-dev docbook-xml docbook-xsl fop libxml2-utils xsltproc krb5-admin-server \ + krb5-kdc krb5-user slapd ldap-utils libssl-dev pkg-config locales-all liblz4-dev \ + libsystemd-dev libxml2-dev libxslt1-dev python3-dev libkrb5-dev libpam-dev \ + libkrb5-*-heimdal uuid-dev gettext \ + liburing-dev python3-distutils ccache && \ + apt-get clean diff --git a/ci/docker/windows_vs_2019 b/ci/docker/windows_vs_2019 new file mode 100644 index 00000000000..a4fcaceae96 --- /dev/null +++ b/ci/docker/windows_vs_2019 @@ -0,0 +1,111 @@ +# escape=` + +# We used to use the visual studio container, but it's too outdated now +FROM cirrusci/windowsservercore:2019 + +SHELL ["powershell", "-NoLogo", "-NoProfile", "-Command"] + + +# Install commandline debugger and log all crashes to c:\cirrus\crashlog.txt +# +# Done manually as doing this via chocolatey / the installer directly, ends up +# with a lot of unnecessary chaff, making the layer unnecessarily large. +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + setx PATH \"C:\Windows Kits\10\Debuggers\x64;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o 'windsdksetup.exe' https://download.microsoft.com/download/9/7/9/97982c1d-d687-41be-9dd3-6d01e52ceb68/windowssdk/winsdk... ; ` + echo 'starting sdk installation (for debugger)' ; ` + Start-Process -Wait -FilePath ".\windsdksetup.exe" ` + -ArgumentList '/Features OptionId.WindowsDesktopDebuggers /layout c:\t\sdk /quiet /norestart /log c:\t\sdk.log' ` + ; ` + ` + Start-Process -Wait -FilePath msiexec.exe ` + -ArgumentList '/a \"C:\t\sdk\Installers\X64 Debuggers And Tools-x64_en-us.msi\" /qb /log install2.log' ` + ; ` + C:\Windows` Kits\10\Debuggers\x64\cdb.exe -version ; ` + ` + Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Debugger' -Value '\"C:\Windows Kits\10\Debuggers\x64\cdb.exe\" -p %ld -e %ld -g -kqm -c \".lines -e; .symfix+ ;.logappend c:\cirrus\crashlog.txt ; !peb; ~*kP ; .logclose ; q \"' ; ` + New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Auto' -Value 1 -PropertyType DWord ; ` + Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name Debugger ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install perl, python, flex and bison. +# +# Done manually as choco takes a lot longer. I think it's download issues with +# powershell's download stuff? That's wy curl.exe is directly used here at least... +# +# Using perl 5.26.3.1 for now, as newer versions don't currently work correctly +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + echo 'adding to path, before setup below, so changes are not overwritten' ; ` + setx PATH \"C:\strawberry\perl\bin;C:\winflexbison;C:\Program Files\Git\usr\bin;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o perl.zip ` + https://strawberryperl.com/download/5.26.3.1/strawberry-perl-5.26.3.1-64bit-portable.zip ; ` + echo 'installing perl' ; ` + 7z.exe x .\perl.zip -xr!c -oc:\strawberry ; ` + ` + curl.exe -sSL -o python.exe https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe ; ` + echo 'installing python' ; ` + Start-Process -Wait -FilePath ".\python.exe" ` + -ArgumentList ` + '/quiet', 'SimpleInstall=1', 'PrependPath=1', 'CompileAll=1', ` + 'TargetDir=c:\python\', 'InstallAllUsers=1', 'Shortcuts=0', ` + 'Include_docs=0', 'Include_tcltk=0', 'Include_tests=0' ` + ; ` + ` + curl.exe -sSL -o winflexbison.zip ` + https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip ; ` + echo 'installing winflexbison' ; ` + 7z.exe x .\winflexbison.zip -oc:\winflexbison ; ` + Rename-Item -Path c:\winflexbison\win_flex.exe c:\winflexbison\flex.exe ; ` + Rename-Item -Path c:\winflexbison\win_bison.exe c:\winflexbison\bison.exe ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install openssl +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + curl.exe -o openssl-setup.exe -sSL https://slproweb.com/download/Win64OpenSSL-1_1_1L.exe ; ` + echo 'staring openssl installation' ; ` + Start-Process -Wait -FilePath ".\openssl-setup.exe" ` + -ArgumentList '/DIR=c:\openssl\1.1.1l\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install visual studio +# +# Adding VS path to vcvarsall.bat so user of container doesn't need to know the full path +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + setx PATH \"c:\BuildTools\VC\Auxiliary\Build;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o c:\t\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe ; ` + echo 'starting visual studio installation' ; ` + Start-Process -Wait ` + -FilePath c:\t\vs_buildtools.exe ` + -ArgumentList ` + '--quiet', '--wait', '--norestart', '--nocache', ` + '--installPath', 'c:\BuildTools', ` + '--add', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', ` + '--add', 'Microsoft.VisualStudio.Component.Windows10SDK.20348' ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse ; ` + Remove-Item -Force -Recurse ${Env:TEMP}\*; ` + Remove-Item -Force -Recurse \"${Env:ProgramData}\Package Cache\" diff --git a/ci/freebsd_gcp_repartition.sh b/ci/freebsd_gcp_repartition.sh new file mode 100755 index 00000000000..2d5e1738998 --- /dev/null +++ b/ci/freebsd_gcp_repartition.sh @@ -0,0 +1,28 @@ +#!/bin/sh + +set -e +set -x + +# The default filesystem on freebsd gcp images is very slow to run tests on, +# due to its 32KB block size +# +# XXX: It'd probably better to fix this in the image, using something like +# https://people.freebsd.org/~lidl/blog/re-root.html + +# fix backup partition table after resize +gpart recover da0 +gpart show da0 +# kill swap, so we can delete a partition +swapoff -a || true +# (apparently we can only have 4!?) +gpart delete -i 3 da0 +gpart add -t freebsd-ufs -l data8k -a 4096 da0 +gpart show da0 +newfs -U -b 8192 /dev/da0p3 + +# Migrate working directory +du -hs $CIRRUS_WORKING_DIR +mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig +mkdir $CIRRUS_WORKING_DIR +mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR +cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/ diff --git a/ci/pg_ci_base.conf b/ci/pg_ci_base.conf new file mode 100644 index 00000000000..637e3cfb343 --- /dev/null +++ b/ci/pg_ci_base.conf @@ -0,0 +1,12 @@ +# Tends to produce too many core files, taking a long time +restart_after_crash = false + +# So that tests using the "manually" started postgres on windows can use +# prepared statements +max_prepared_transactions=10 + +# Settings that make logs more useful +log_line_prefix='%m [%p][%b][%v:%x] ' +log_checkpoints = true +log_connections = true +log_disconnections = true diff --git a/ci/windows_build_config.pl b/ci/windows_build_config.pl new file mode 100644 index 00000000000..ba82b13d69a --- /dev/null +++ b/ci/windows_build_config.pl @@ -0,0 +1,13 @@ +use strict; +use warnings; + +our $config; + +$config->{"tap_tests"} = 1; +$config->{"asserts"} = 1; + +$config->{"openssl"} = "c:/openssl/1.1.1l/"; +$config->{"perl"} = "c:/strawberry/perl/"; +$config->{"python"} = "c:/python/"; + +1; -- 2.23.0.385.gbc12974a89 --ep4fmvj5ysdv22gg Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v2-0002-windows-Improve-crash-assert-exception-handling.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v3 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. @ 2021-03-15 16:25 Andres Freund <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Andres Freund @ 2021-03-15 16:25 UTC (permalink / raw) Author: Andres Freund <[email protected]> Author: Thomas Munro <[email protected]> Author: Melanie Plageman <[email protected]> Reviewed-By: Melanie Plageman <[email protected]> Discussion: https://postgr.es/m/[email protected] --- .cirrus.yml | 456 ++++++++++++++++++++++++++++++++ .dockerignore | 7 + ci/README | 50 ++++ ci/docker/linux_debian_bullseye | 67 +++++ ci/docker/windows_vs_2019 | 122 +++++++++ ci/gcp_freebsd_repartition.sh | 28 ++ ci/pg_ci_base.conf | 14 + ci/windows_build_config.pl | 13 + 8 files changed, 757 insertions(+) create mode 100644 .cirrus.yml create mode 100644 .dockerignore create mode 100644 ci/README create mode 100644 ci/docker/linux_debian_bullseye create mode 100644 ci/docker/windows_vs_2019 create mode 100755 ci/gcp_freebsd_repartition.sh create mode 100644 ci/pg_ci_base.conf create mode 100644 ci/windows_build_config.pl diff --git a/.cirrus.yml b/.cirrus.yml new file mode 100644 index 00000000000..7a068650206 --- /dev/null +++ b/.cirrus.yml @@ -0,0 +1,456 @@ +# What files to preserve in case tests fail +on_failure: &on_failure + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + +env: + # accelerate initial clone, but a bit of depth so that concurrent tasks work + CIRRUS_CLONE_DEPTH: 100 + # Useful to be able to analyse what in a script takes long + CIRRUS_LOG_TIMESTAMP: true + # target to test, for all but windows + CHECK: check-world + CHECKFLAGS: -Otarget + PGCTLTIMEOUT: 120 + CCACHE_MAXSIZE: "500M" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl + + +task: + name: FreeBSD + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-freebsd-13-0 + platform: freebsd + cpu: 2 + memory: 2G + disk: 50 + env: + CCACHE_DIR: "/tmp/ccache_dir" + + ccache_cache: + folder: "/tmp/ccache_dir" + sysinfo_script: + - export + sysconfig_script: + - sudo sysctl kern.corefile='/tmp/%N.%P.core' + repartition_script: + - ci/gcp_freebsd_repartition.sh + create_user_script: + - pw useradd postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/lib/tcl8.6/ \ + --with-uuid=bsd \ + \ + --with-includes=/usr/local/include \ + --with-libs=/usr/local/lib \ + \ + CC="ccache cc" CFLAGS="-O0 -ggdb"' + build_script: + - su postgres -c 'gmake -s -j3 && gmake -s -j3 -C contrib' + upload_caches: + - ccache + + # Use of script is to avoid make complaints about fcntl() + # https://savannah.gnu.org/bugs/?60774 + test_world_script: + - su postgres -c 'time script test.log gmake -s -j2 ${CHECK} ${CHECKFLAGS}' + + on_failure: + <<: *on_failure + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile; + done + + +task: + name: Linux + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-bullseye + platform: linux + cpu: 4 + memory: 2G + nested_virtualization: false + env: + CCACHE_DIR: "/tmp/ccache_dir" + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + ccache_cache: + folder: "/tmp/ccache_dir" + + sysinfo_script: + - id + - uname -a + - cat /proc/cmdline + - lsblk + - ulimit -a -H + - ulimit -a -S + - export + sysconfig_script: + - useradd -m postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf + - su postgres -c 'ulimit -l -H' + - su postgres -c 'ulimit -l -S' + - echo '/tmp/%e-%s-%p.core' > /proc/sys/kernel/core_pattern + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-llvm \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-selinux \ + --with-ssl=openssl \ + --with-systemd \ + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ \ + --with-uuid=e2fs \ + \ + CC="ccache gcc" CFLAGS="-O0 -ggdb" \ + CXX="ccache g++" CXXFLAGS="-O0 -ggdb" \ + CLANG="ccache clang" ' + build_script: + - su postgres -c 'make -s -j4 && make -s -j4 -C contrib' + upload_caches: + - ccache + + test_world_script: | + su postgres -c '\ + ulimit -c unlimited; \ + make -s ${CHECK} ${CHECKFLAGS} -j8 \ + ' + + on_failure: + <<: *on_failure + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile ; + done + + +task: + name: macOS + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' + osx_instance: + image: monterey-base + env: + CIRRUS_WORKING_DIR: ${HOME}/pgsql/ + CCACHE_DIR: ${HOME}/ccache + HOMEBREW_CACHE: ${HOME}/homebrew-cache + PERL5LIB: ${HOME}/perl5/lib/perl5 + + sysinfo_script: + - id + - export + ccache_cache: + folder: ${CCACHE_DIR} + homebrew_cache: + folder: ${HOMEBREW_CACHE} + perl_cache: + folder: ~/perl5 + + cpan_install_script: + - perl -mIPC::Run -e 1 || cpan -T IPC::Run + - perl -mIO::Pty -e 1 || cpan -T IO::Pty + upload_caches: + - perl + core_install_script: + - sudo chmod 777 /cores + + # XXX: Could we instead install homebrew into a cached directory? The + # homebrew installation takes a good bit of time every time, even if the + # packages do not need to be downloaded. + homebrew_install_script: | + brew install \ + ccache \ + coreutils \ + icu4c \ + krb5 \ + llvm \ + lz4 \ + make \ + openldap \ + openssl \ + [email protected] \ + tcl-tk + + brew cleanup -s + upload_caches: + - homebrew + + configure_script: | + LIBS="/usr/local/lib:$LIBS" + INCLUDES="/usr/local/include:$INCLUDES" + + PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" + INCLUDES="/usr/local/opt/icu4c/include:$INCLUDES" + LIBS="/usr/local/opt/icu4c/lib:$LIBS" + + PKG_CONFIG_PATH="/usr/local/opt/krb5/lib/pkgconfig:$PKG_CONFIG_PATH" + INCLUDES="/usr/local/opt/krb5/include:$INCLUDES" + LIBS="/usr/local/opt/krb5/lib:$LIBS" + + LIBS="/usr/local/opt/openldap/lib:$LIBS" + INCLUDES="/usr/local/opt/openldap/include:$INCLUDES" + + INCLUDES="/usr/local/opt/openssl/include:$INCLUDES" + LIBS="/usr/local/opt/openssl/lib:$LIBS" + + export PKG_CONFIG_PATH + + ./configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-bonjour \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-lz4 \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/opt/tcl-tk/lib/ \ + --with-uuid=e2fs \ + \ + --prefix=$HOME/install \ + --with-includes="$INCLUDES" \ + --with-libs="$LIBS" \ + \ + CC="ccache cc" CFLAGS="-O0 -ggdb" \ + CXX="ccache c++" CXXFLAGS="-O0 -ggdb" \ + CLANG="ccache /usr/local/opt/llvm/bin/ccache" \ + LLVM_CONFIG=/usr/local/opt/llvm/bin/llvm-config \ + PYTHON=/usr/local/opt/[email protected]/bin/python3 + build_script: + - gmake -s -j12 && gmake -s -j12 -C contrib + upload_caches: + - ccache + + test_world_script: + - ulimit -c unlimited + - ulimit -n 1024 + # See freebsd use of script + - script test.log gmake -s -j12 ${CHECK} ${CHECKFLAGS} + + on_failure: + <<: *on_failure + cores_script: | + for corefile in $(find /cores/ -name 'core.*' 2>/dev/null) ; do + lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' ; + done + + +task: + name: Windows + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + windows_container: + dockerfile: ci/docker/windows_vs_2019 + cpu: 4 + memory: 4G + env: + PROVE_FLAGS: -j10 + # The default cirrus working dir is in a directory msbuild complains about + CIRRUS_WORKING_DIR: "c:/cirrus" + # Avoid re-installing over and over + NO_TEMP_INSTALL: 1 + # git's tar doesn't deal with drive letters, see + # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net + TAR: "c:/windows/system32/tar.exe" + # Avoids port conflicts + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: "c:/cirrus/" + + sysinfo_script: + - chcp + - systeminfo + - powershell -Command get-psdrive -psprovider filesystem + - ps: Get-Item -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' + - set + + configure_script: + - copy ci\windows_build_config.pl src\tools\msvc\config.pl + - vcvarsall x64 + - perl src/tools/msvc/mkvcbuild.pl + build_script: + - vcvarsall x64 + # Disable file tracker, we're never going to rebuild... + - msbuild -m /p:TrackFileAccess=false pgsql.sln + tempinstall_script: + # Installation on windows currently only completely works from src\tools\msvc + - cd src\tools\msvc && perl .\install.pl %CIRRUS_WORKING_DIR%\tmp_install + + test_check_parallel_script: + - perl src/tools/msvc/vcregress.pl check parallel + startcreate_script: + - tmp_install\bin\pg_ctl.exe initdb -D tmp_check\db -l tmp_check\initdb.log + - echo include '%TEMP_CONFIG%' >> tmp_check\db\postgresql.conf + - tmp_install\bin\pg_ctl.exe start -D tmp_check\db -l tmp_check\postmaster.log + test_plcheck_script: + - perl src/tools/msvc/vcregress.pl plcheck + test_isolationcheck_script: + - perl src/tools/msvc/vcregress.pl isolationcheck + test_modulescheck_script: + - perl src/tools/msvc/vcregress.pl modulescheck + test_contribcheck_script: + - perl src/tools/msvc/vcregress.pl contribcheck + stop_script: + - tmp_install\bin\pg_ctl.exe stop -D tmp_check\db -l tmp_check\postmaster.log + test_ssl_script: + - set with_ssl=openssl + - perl src/tools/msvc/vcregress.pl taptest .\src\test\ssl\ + test_subscriptioncheck_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\subscription\ + test_authentication_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\authentication\ + test_recoverycheck_script: + - perl src/tools/msvc/vcregress.pl recoverycheck + test_bincheck_script: + - perl src/tools/msvc/vcregress.pl bincheck + test_upgradecheck_script: + - perl src/tools/msvc/vcregress.pl upgradecheck + test_ecpgcheck_script: + # tries to build additional stuff + - vcvarsall x64 + # References ecpg_regression.proj in the current dir + - cd src\tools\msvc + - perl vcregress.pl ecpgcheck + + on_failure: *on_failure + + +task: + name: CompilerWarnings + depends_on: + - Linux + # task that did not run count as a success, so we need to recheck Linux' + # condition here :/ + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + container: + dockerfile: ci/docker/linux_debian_bullseye + env: + # Use larger ccache cache, as this task compiles with multiple compilers / + # flag combinations + CCACHE_SIZE: "4GB" + CCACHE_DIR: "/tmp/ccache_dir" + ccache_cache: + folder: "/tmp/ccache_dir" + + sysinfo_script: + - gcc -v + - clang -v + + setup_script: + - echo "COPT=-Werror" > src/Makefile.custom + + ### + # Test that code can be built with gcc/clang without warnings + ### + + # Trace probes have a history of getting accidentally broken. Use the + # different compilers to build with different combinations of dtrace on/off + # and cassert on/off. + + # gcc, cassert off, dtrace on + always: + gcc_warning_script: | + time ./configure \ + --cache gcc.cache CC="ccache gcc" \ + --enable-dtrace + make -s -j4 clean + time make -s -j4 world-bin + # gcc, cassert on, dtrace off + always: + gcc_a_warning_script: | + time ./configure \ + --cache gcc.cache CC="ccache gcc" \ + --enable-cassert + make -s -j4 clean + time make -s -j4 world-bin + # clang, cassert off, dtrace off + always: + clang_warning_script: | + time ./configure \ + --cache clang.cache CC="ccache clang" + make -s -j4 clean + time make -s -j4 world-bin + # clang, cassert on, dtrace on + always: + clang_a_warning_script: | + time ./configure \ + --cache clang.cache CC="ccache clang" \ + --enable-dtrace \ + --enable-cassert + make -s -j4 clean + time make -s -j4 world-bin + # cross-compile to windows + always: + mingw_cross_warning_script: | + time ./configure \ + --host=x86_64-w64-mingw32 \ + --enable-cassert \ + CC="ccache x86_64-w64-mingw32-gcc" + make -s -j4 clean + time make -s -j4 world-bin + + ### + # Verify docs can be built + ### + + # XXX: Only do this if there have been changes in doc/ since last build + always: + docs_build_script: | + time ./configure \ + --cache gcc.cache CC="ccache gcc" + time make -s -j4 -C doc + + always: + upload_caches: + - ccache diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000000..f27b14d5c26 --- /dev/null +++ b/.dockerignore @@ -0,0 +1,7 @@ +# Ignore everything, except ci/ +# + +# cirrus-ci uses the whole repository as context when building dockerfiles, +# and the dockerfiles should not reference anything outside of ci/ +* +!ci/* diff --git a/ci/README b/ci/README new file mode 100644 index 00000000000..fda6eb39fb0 --- /dev/null +++ b/ci/README @@ -0,0 +1,50 @@ +Postgres Continuous Integration (CI) +==================================== + +Postgres has two forms of CI: + +1) All supported branches in the main postgres repository are continuously + tested via the the the buildfarm. As this covers only the main repository, + it cannot be used during development of features. + + For details see https://buildfarm.postgresql.org/ + +2) For not yet merged development work, CI can be enabled for some git hosting + providers. This allows to test patches on a number of platforms before they + are merged (or even submitted). + + +Configuring CI on personal repositories +======================================= + +Currently postgres contains CI support utilizing cirrus-ci. cirrus-ci +currently is only available for github. + + +Enabling cirrus-ci in a github repository +========================================= + +To enable cirrus-ci on a repository, go to +https://github.com/marketplace/cirrus-ci and select "Public +Repositories". Then "Install it for free" and "Complete order". The next page +allows to configure which repositories cirrus-ci has access too. Choose the +relevant repository and "Install". + +See also https://cirrus-ci.org/guide/quick-start/ + +Once enabled on a repository, future commits and pull-requests in that +repository will automatically trigger CI builds. These are visible from the +commit history / PRs, and can also be viewed in the cirrus-ci UI at +https://cirrus-ci.com/github/<username>/<reponame>/ + + +Controlling CI via commit messages +================================== + +The behavior of CI can be controlled by special content in commit +messages. Currently the following controls are available: + +- ci-os-only: {(freebsd|linux|macos|windows)} + + Only runs CI on operating systems specified. This can be useful when + addressing portability issues affecting only a subset of platforms. diff --git a/ci/docker/linux_debian_bullseye b/ci/docker/linux_debian_bullseye new file mode 100644 index 00000000000..bf48509c9e4 --- /dev/null +++ b/ci/docker/linux_debian_bullseye @@ -0,0 +1,67 @@ +FROM debian:bullseye +RUN \ + export DEBIAN_FRONTEND=noninteractive && \ + apt-get -y update && \ + apt-get -y upgrade --no-install-recommends && \ + apt-get -y install --no-install-recommends \ + \ + build-essential \ + gdb \ + make \ + git \ + meson \ + perl \ + pkg-config \ + \ + bison \ + ccache \ + clang \ + flex \ + g++ \ + gcc \ + gettext \ + \ + libio-pty-perl \ + libipc-run-perl \ + python3-distutils \ + \ + libicu-dev \ + libkrb5-*-heimdal \ + libkrb5-dev \ + libldap2-dev \ + liblz4-dev \ + libossp-uuid-dev \ + libpam-dev \ + libperl-dev \ + libpython3-dev \ + libreadline-dev \ + libselinux*-dev \ + libssl-dev \ + libsystemd-dev \ + liburing-dev \ + libxml2-dev \ + libxslt1-dev \ + llvm-dev \ + python3-dev \ + systemtap-sdt-dev \ + tcl-dev \ + uuid-dev \ + \ + docbook-xml \ + docbook-xsl \ + fop \ + libxml2-utils \ + xsltproc \ + \ + krb5-admin-server \ + krb5-kdc \ + krb5-user \ + ldap-utils \ + locales-all \ + slapd \ + \ + g++-mingw-w64-x86-64-win32 \ + gcc-mingw-w64-x86-64-win32 \ + libz-mingw-w64-dev \ + && \ + apt-get clean && rm -rf /var/lib/apt/lists/* diff --git a/ci/docker/windows_vs_2019 b/ci/docker/windows_vs_2019 new file mode 100644 index 00000000000..5e0ff91ab6e --- /dev/null +++ b/ci/docker/windows_vs_2019 @@ -0,0 +1,122 @@ +# escape=` + +# We used to use the visual studio container, but it's too outdated (&bloated) now +FROM cirrusci/windowsservercore:2019 + +SHELL ["powershell", "-NoLogo", "-NoProfile", "-Command"] + + +# Install commandline debugger and log all crashes to c:\cirrus\crashlog.txt +# +# Done manually as doing this via chocolatey / the installer directly, ends up +# with a lot of unnecessary chaff, making the layer unnecessarily large. +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + setx PATH \"C:\Windows Kits\10\Debuggers\x64;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o 'windsdksetup.exe' https://download.microsoft.com/download/9/7/9/97982c1d-d687-41be-9dd3-6d01e52ceb68/windowssdk/winsdk... ; ` + echo 'starting sdk installation (for debugger)' ; ` + Start-Process -Wait -FilePath ".\windsdksetup.exe" ` + -ArgumentList '/Features OptionId.WindowsDesktopDebuggers /layout c:\t\sdk /quiet /norestart /log c:\t\sdk.log' ` + ; ` + ` + Start-Process -Wait -FilePath msiexec.exe ` + -ArgumentList '/a \"C:\t\sdk\Installers\X64 Debuggers And Tools-x64_en-us.msi\" /qb /log install2.log' ` + ; ` + C:\Windows` Kits\10\Debuggers\x64\cdb.exe -version ; ` + ` + Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Debugger' -Value '\"C:\Windows Kits\10\Debuggers\x64\cdb.exe\" -p %ld -e %ld -g -kqm -c \".lines -e; .symfix+ ;.logappend c:\cirrus\crashlog.txt ; !peb; ~*kP ; .logclose ; q \"' ; ` + New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Auto' -Value 1 -PropertyType DWord ; ` + Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name Debugger ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install perl, python, flex and bison. +# +# Done manually as choco takes a lot longer. I think it's download issues with +# powershell's download stuff? That's why curl.exe is directly used here at +# least... +# +# Using perl 5.26.3.1 for now, as newer versions don't currently work +# correctly for plperl +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + echo 'adding to path, before setup below, so changes are not overwritten' ; ` + setx PATH \"C:\strawberry\perl\bin;C:\winflexbison;C:\Program Files\Git\usr\bin;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o perl.zip ` + https://strawberryperl.com/download/5.26.3.1/strawberry-perl-5.26.3.1-64bit-portable.zip ; ` + echo 'installing perl' ; ` + 7z.exe x .\perl.zip -xr!c -oc:\strawberry ; ` + ` + curl.exe -sSL -o python.exe https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe ; ` + echo 'installing python' ; ` + Start-Process -Wait -FilePath ".\python.exe" ` + -ArgumentList ` + '/quiet', 'SimpleInstall=1', 'PrependPath=1', 'CompileAll=1', ` + 'TargetDir=c:\python\', 'InstallAllUsers=1', 'Shortcuts=0', ` + 'Include_docs=0', 'Include_tcltk=0', 'Include_tests=0' ` + ; ` + ` + curl.exe -sSL -o winflexbison.zip ` + https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip ; ` + echo 'installing winflexbison' ; ` + 7z.exe x .\winflexbison.zip -oc:\winflexbison ; ` + Rename-Item -Path c:\winflexbison\win_flex.exe c:\winflexbison\flex.exe ; ` + Rename-Item -Path c:\winflexbison\win_bison.exe c:\winflexbison\bison.exe ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install openssl +# +# This is one of the windows binararies referenced by the openssl wiki. See +# https://www.openssl.org/community/binaries.html and +# https://wiki.openssl.org/index.php/Binaries +# +# It might be nicer to switch to the openssl built as part of curl-for-win, +# but recent releases only build openssl 3, and that still seems troublesome +# on windows, +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + curl.exe -o openssl-setup.exe -sSL https://slproweb.com/download/Win64OpenSSL-1_1_1L.exe ; ` + echo 'staring openssl installation' ; ` + Start-Process -Wait -FilePath ".\openssl-setup.exe" ` + -ArgumentList '/DIR=c:\openssl\1.1.1l\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install visual studio +# +# Adding path to vcvarsall.bat to PATH so user of container dont't need to +# know the full path +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + setx PATH \"c:\BuildTools\VC\Auxiliary\Build;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o c:\t\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe ; ` + echo 'starting visual studio installation' ; ` + Start-Process -Wait ` + -FilePath c:\t\vs_buildtools.exe ` + -ArgumentList ` + '--quiet', '--wait', '--norestart', '--nocache', ` + '--installPath', 'c:\BuildTools', ` + '--add', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', ` + '--add', 'Microsoft.VisualStudio.Component.Windows10SDK.20348' ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse ; ` + Remove-Item -Force -Recurse ${Env:TEMP}\*; ` + Remove-Item -Force -Recurse \"${Env:ProgramData}\Package Cache\" diff --git a/ci/gcp_freebsd_repartition.sh b/ci/gcp_freebsd_repartition.sh new file mode 100755 index 00000000000..2d5e1738998 --- /dev/null +++ b/ci/gcp_freebsd_repartition.sh @@ -0,0 +1,28 @@ +#!/bin/sh + +set -e +set -x + +# The default filesystem on freebsd gcp images is very slow to run tests on, +# due to its 32KB block size +# +# XXX: It'd probably better to fix this in the image, using something like +# https://people.freebsd.org/~lidl/blog/re-root.html + +# fix backup partition table after resize +gpart recover da0 +gpart show da0 +# kill swap, so we can delete a partition +swapoff -a || true +# (apparently we can only have 4!?) +gpart delete -i 3 da0 +gpart add -t freebsd-ufs -l data8k -a 4096 da0 +gpart show da0 +newfs -U -b 8192 /dev/da0p3 + +# Migrate working directory +du -hs $CIRRUS_WORKING_DIR +mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig +mkdir $CIRRUS_WORKING_DIR +mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR +cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/ diff --git a/ci/pg_ci_base.conf b/ci/pg_ci_base.conf new file mode 100644 index 00000000000..d8faa9c26c1 --- /dev/null +++ b/ci/pg_ci_base.conf @@ -0,0 +1,14 @@ +# Tends to produce too many core files, taking a long time +restart_after_crash = false + +# So that tests using the "manually" started postgres on windows can use +# prepared statements +max_prepared_transactions = 10 + +# Settings that make logs more useful +log_autovacuum_min_duration = 0 +log_checkpoints = true +log_connections = true +log_disconnections = true +log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] ' +log_lock_waits = true diff --git a/ci/windows_build_config.pl b/ci/windows_build_config.pl new file mode 100644 index 00000000000..ba82b13d69a --- /dev/null +++ b/ci/windows_build_config.pl @@ -0,0 +1,13 @@ +use strict; +use warnings; + +our $config; + +$config->{"tap_tests"} = 1; +$config->{"asserts"} = 1; + +$config->{"openssl"} = "c:/openssl/1.1.1l/"; +$config->{"perl"} = "c:/strawberry/perl/"; +$config->{"python"} = "c:/python/"; + +1; -- 2.34.0 --rbj3wvz2ukg3lhpx Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0002-windows-Improve-crash-assert-exception-handling.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v4 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. @ 2021-03-15 16:25 Andres Freund <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Andres Freund @ 2021-03-15 16:25 UTC (permalink / raw) Author: Andres Freund <[email protected]> Author: Thomas Munro <[email protected]> Author: Melanie Plageman <[email protected]> Reviewed-By: Melanie Plageman <[email protected]> Reviewed-By: Justin Pryzby <[email protected]> Reviewed-By: Thomas Munro <[email protected]> Reviewed-By: Peter Eisentraut <[email protected]> Discussion: https://postgr.es/m/[email protected] --- .cirrus.yml | 526 ++++++++++++++++++++++++ src/tools/ci/README | 63 +++ src/tools/ci/cores_backtrace.sh | 50 +++ src/tools/ci/gcp_freebsd_repartition.sh | 28 ++ src/tools/ci/pg_ci_base.conf | 14 + src/tools/ci/windows_build_config.pl | 13 + 6 files changed, 694 insertions(+) create mode 100644 .cirrus.yml create mode 100644 src/tools/ci/README create mode 100755 src/tools/ci/cores_backtrace.sh create mode 100755 src/tools/ci/gcp_freebsd_repartition.sh create mode 100644 src/tools/ci/pg_ci_base.conf create mode 100644 src/tools/ci/windows_build_config.pl diff --git a/.cirrus.yml b/.cirrus.yml new file mode 100644 index 00000000000..4116e0155dc --- /dev/null +++ b/.cirrus.yml @@ -0,0 +1,526 @@ +# CI configuration file for CI utilizing cirrus-ci.org +# +# See src/tools/ci/README for information. + +env: + # Source of images / containers + GCP_PROJECT: pg-ci-images + IMAGE_PROJECT: $GCP_PROJECT + CONTAINER_REPO: us-docker.pkg.dev/${GCP_PROJECT}/ci + + # accelerate initial clone, but a bit of depth so that concurrent tasks work + CIRRUS_CLONE_DEPTH: 100 + # Useful to be able to analyse what in a script takes long + CIRRUS_LOG_TIMESTAMP: true + + CCACHE_MAXSIZE: "500M" + + # target to test, for all but windows + PROVE_FLAGS: --timer + CHECK: check-world PROVE_FLAGS=$PROVE_FLAGS + CHECKFLAGS: -Otarget + PGCTLTIMEOUT: 120 + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/src/tools/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl + + +# What files to preserve in case tests fail +on_failure: &on_failure + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: FreeBSD - 13 + + env: + # FreeBSD on GCP is slow when running with larger number of CPUS / + # jobs. Using one more job than cpus seems to work best. + CPUS: 2 + BUILD_JOBS: 3 + TEST_JOBS: 3 + + CCACHE_DIR: /tmp/ccache_dir + + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*' + + compute_engine_instance: + image_project: $IMAGE_PROJECT + image: family/pg-ci-freebsd-13-0 + platform: freebsd + cpu: $CPUS + memory: 2G + disk: 50 + + sysinfo_script: | + id + uname -a + ulimit -a -H + ulimit -a -S + export + + ccache_cache: + folder: $CCACHE_DIR + # Workaround around performance issues due to 32KB block size + repartition_script: src/tools/ci/gcp_freebsd_repartition.sh + create_user_script: | + pw useradd postgres + chown -R postgres:postgres . + mkdir -p ${CCACHE_DIR} + chown -R postgres:postgres ${CCACHE_DIR} + setup_cores_script: | + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kern.corefile='/tmp/cores/%N.%P.core' + + # NB: Intentionally building without --with-llvm. The freebsd image size is + # already large enough to make task startup slow, and freebsd already takes + # longer than other platforms except for windows. + configure_script: | + su postgres <<-EOF + ./configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/lib/tcl8.6/ \ + --with-uuid=bsd \ + \ + --with-includes=/usr/local/include \ + --with-libs=/usr/local/lib \ + \ + CC="ccache cc" CFLAGS="-O0 -ggdb" + EOF + build_script: su postgres -c "gmake -s -j${BUILD_JOBS} world-bin" + upload_caches: ccache + + # Use of script is to avoid make -Otarget complaints about fcntl() on platforms + # without support for locking pipes. See also https://savannah.gnu.org/bugs/?60774 + # script uses a pseudo-tty, which do support locking. + test_world_script: + - su postgres -c "time script test.log gmake -s -j${TEST_JOBS} ${CHECK} ${CHECKFLAGS}" + + on_failure: + <<: *on_failure + cores_script: src/tools/ci/cores_backtrace.sh freebsd /tmp/cores + + +# This is shared between the task running the linux tests and the +# CompilerWarnings task +LINUX_CONFIGURE_FEATURES: &LINUX_CONFIGURE_FEATURES >- + --with-gssapi + --with-icu + --with-ldap + --with-libxml + --with-libxslt + --with-llvm + --with-lz4 + --with-pam + --with-perl + --with-python + --with-selinux + --with-ssl=openssl + --with-systemd + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ + --with-uuid=e2fs + +task: + name: Linux - Debian Bullseye + + env: + CPUS: 4 + BUILD_JOBS: 4 + TEST_JOBS: 8 # experimentally derived to be a decent choice + + CCACHE_DIR: /tmp/ccache_dir + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES + + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + + compute_engine_instance: + image_project: $IMAGE_PROJECT + image: family/pg-ci-bullseye + platform: linux + cpu: $CPUS + memory: 2G + + ccache_cache: + folder: ${CCACHE_DIR} + + sysinfo_script: | + id + uname -a + cat /proc/cmdline + lsblk + ulimit -a -H + ulimit -a -S + export + create_user_script: | + useradd -m postgres + chown -R postgres:postgres . + mkdir -p ${CCACHE_DIR} + chown -R postgres:postgres ${CCACHE_DIR} + echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf + su postgres -c 'ulimit -l -H' + su postgres -c 'ulimit -l -S' + setup_cores_script: | + mkdir -m 770 /tmp/cores + chown root:postgres /tmp/cores + sysctl kernel.core_pattern='/tmp/cores/%e-%s-%p.core' + + configure_script: | + su postgres <<-EOF + ./configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + ${LINUX_CONFIGURE_FEATURES} \ + \ + CC="ccache gcc" CFLAGS="-O0 -ggdb" \ + CXX="ccache g++" CXXFLAGS="-O0 -ggdb" \ + CLANG="ccache clang" + EOF + build_script: su postgres -c 'make -s -j${BUILD_JOBS} world-bin' + upload_caches: ccache + + test_world_script: | + su postgres <<-EOF + ulimit -c unlimited # default is 0 + make -s ${CHECK} ${CHECKFLAGS} -j${TEST_JOBS} + EOF + + on_failure: + <<: *on_failure + cores_script: src/tools/ci/cores_backtrace.sh linux /tmp/cores + + +task: + name: macOS - Monterey + + env: + CPUS: 12 # always get that much for cirrusci macOS instances + BUILD_JOBS: $CPUS + TEST_JOBS: $CPUS # already fast enough to not be worth tuning + + CIRRUS_WORKING_DIR: ${HOME}/pgsql/ + CCACHE_DIR: ${HOME}/ccache + HOMEBREW_CACHE: ${HOME}/homebrew-cache + PERL5LIB: ${HOME}/perl5/lib/perl5 + + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' + + osx_instance: + image: monterey-base + + sysinfo_script: | + id + export + ulimit -a -H + ulimit -a -S + + setup_cores_script: + - mkdir ${HOME}/cores + - sudo sysctl kern.corefile="${HOME}/cores/core.%P" + + perl_cache: + folder: ~/perl5 + cpan_install_script: + - perl -mIPC::Run -e 1 || cpan -T IPC::Run + - perl -mIO::Pty -e 1 || cpan -T IO::Pty + upload_caches: perl + + + # XXX: Could we instead install homebrew into a cached directory? The + # homebrew installation takes a good bit of time every time, even if the + # packages do not need to be downloaded. + homebrew_cache: + folder: $HOMEBREW_CACHE + homebrew_install_script: | + brew install \ + ccache \ + icu4c \ + krb5 \ + llvm \ + lz4 \ + make \ + openldap \ + openssl \ + python \ + tcl-tk + + brew cleanup -s # to reduce cache size + upload_caches: homebrew + + ccache_cache: + folder: $CCACHE_DIR + configure_script: | + LIBS="/usr/local/lib:${LIBS}" + INCLUDES="/usr/local/include:${INCLUDES}" + + brewpath=/usr/local/opt + for package in icu4c krb5 openldap openssl; do + PKG_CONFIG_PATH="${brewpath}/${package}/lib/pkgconfig:${PKG_CONFIG_PATH}" + INCLUDES="${brewpath}/${package}/include:${INCLUDES}" + LIBS="${brewpath}/${package}/lib:${LIBS}" + done + + export PKG_CONFIG_PATH + + ./configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-bonjour \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-lz4 \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=${brewpath}/tcl-tk/lib/ \ + --with-uuid=e2fs \ + \ + --prefix=${HOME}/install \ + --with-includes="${INCLUDES}" \ + --with-libs="${LIBS}" \ + \ + CC="ccache cc" CFLAGS="-O0 -ggdb" \ + CXX="ccache c++" CXXFLAGS="-O0 -ggdb" \ + CLANG="ccache ${brewpath}/llvm/bin/ccache" \ + LLVM_CONFIG=${brewpath}/llvm/bin/llvm-config \ + PYTHON=python3 + build_script: gmake -s -j${BUILD_JOBS} world-bin + upload_caches: ccache + + test_world_script: | + ulimit -c unlimited # default is 0 + ulimit -n 1024 # default is 256, pretty low + # See freebsd use of script for explanation + script test.log gmake -s -j${TEST_JOBS} ${CHECK} ${CHECKFLAGS} + + on_failure: + <<: *on_failure + cores_script: src/tools/ci/cores_backtrace.sh macos "${HOME}/cores" + + +task: + name: Windows - Server 2019, VS 2019 + + env: + # Half the allowed per-user CPU cores + CPUS: 4 + + # our windows infrastructure doesn't have test concurrency above the level + # of a single vcregress test target. Because of that it makes sense to use + # concurrency within prove. For other targets it's not really useful + # because two sources of concurrency can overload machines. + # + # The concrete choice of 10 is based on a small bit of experimentation and + # likely can be improved upon further. + PROVE_FLAGS: -j10 --timer + + # The default cirrus working dir is in a directory msbuild complains about + CIRRUS_WORKING_DIR: "c:/cirrus" + # Avoid re-installing over and over + NO_TEMP_INSTALL: 1 + # git's tar doesn't deal with drive letters, see + # https://postgr.es/m/b6782dc3-a7b0-ed56-175f-f8f54cb08d67%40dunslane.net + TAR: "c:/windows/system32/tar.exe" + # Avoids port conflicts between concurrent tap test runs + PG_TEST_USE_UNIX_SOCKETS: 1 + PG_REGRESS_SOCK_DIR: "c:/cirrus/" + # Use parallelism, disable file tracker, we're never going to rebuild... + MSBFLAGS: -m -verbosity:minimal /p:TrackFileAccess=false + + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + + windows_container: + image: $CONTAINER_REPO/windows_ci_vs_2019:latest + cpu: $CPUS + memory: 4G + + sysinfo_script: | + chcp + systeminfo + powershell -Command get-psdrive -psprovider filesystem + set + + configure_script: + # copy errors out when using forward slashes + - copy src\tools\ci\windows_build_config.pl src\tools\msvc\config.pl + - vcvarsall x64 + - perl src/tools/msvc/mkvcbuild.pl + build_script: + - vcvarsall x64 + - msbuild %MSBFLAGS% pgsql.sln + tempinstall_script: + # Installation on windows currently only completely works from src/tools/msvc + - cd src/tools/msvc && perl install.pl %CIRRUS_WORKING_DIR%/tmp_install + + test_regress_parallel_script: + - perl src/tools/msvc/vcregress.pl check parallel + startcreate_script: + # paths to binaries need backslashes + - tmp_install\bin\pg_ctl.exe initdb -D tmp_check/db -l tmp_check/initdb.log + - echo include '%TEMP_CONFIG%' >> tmp_check/db/postgresql.conf + - tmp_install\bin\pg_ctl.exe start -D tmp_check/db -l tmp_check/postmaster.log + test_pl_script: + - perl src/tools/msvc/vcregress.pl plcheck + test_isolation_script: + - perl src/tools/msvc/vcregress.pl isolationcheck + test_modules_script: + - perl src/tools/msvc/vcregress.pl modulescheck + test_contrib_script: + - perl src/tools/msvc/vcregress.pl contribcheck + stop_script: + - tmp_install\bin\pg_ctl.exe stop -D tmp_check/db -l tmp_check/postmaster.log + test_ssl_script: + - set with_ssl=openssl + - perl src/tools/msvc/vcregress.pl taptest ./src/test/ssl/ + test_subscription_script: + - perl src/tools/msvc/vcregress.pl taptest ./src/test/subscription/ + test_authentication_script: + - perl src/tools/msvc/vcregress.pl taptest ./src/test/authentication/ + test_recovery_script: + - perl src/tools/msvc/vcregress.pl recoverycheck + test_bin_script: + - perl src/tools/msvc/vcregress.pl bincheck + test_pg_upgrade_script: + - perl src/tools/msvc/vcregress.pl upgradecheck + test_ecpg_script: + # tries to build additional stuff + - vcvarsall x64 + # References ecpg_regression.proj in the current dir + - cd src/tools/msvc + - perl vcregress.pl ecpgcheck + + on_failure: *on_failure + + +task: + name: CompilerWarnings + + # To limit unneccessary work only run this once normal linux test succeeded + depends_on: + - Linux - Debian Bullseye + + env: + CPUS: 4 + BUILD_JOBS: 4 + + # Use larger ccache cache, as this task compiles with multiple compilers / + # flag combinations + CCACHE_SIZE: "4GB" + CCACHE_DIR: "/tmp/ccache_dir" + + LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES + + # task that did not run count as a success, so we need to recheck Linux' + # condition here :/ + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + + container: + image: $CONTAINER_REPO/linux_debian_bullseye_ci:latest + + sysinfo_script: | + id + uname -a + ulimit -a -H + ulimit -a -S + gcc -v + clang -v + export + + ccache_cache: + folder: $CCACHE_DIR + + ### + # Test that code can be built with gcc/clang without warnings + ### + + setup_script: echo "COPT=-Werror" > src/Makefile.custom + + # Trace probes have a history of getting accidentally broken. Use the + # different compilers to build with different combinations of dtrace on/off + # and cassert on/off. + + # gcc, cassert off, dtrace on + always: + gcc_warning_script: | + time ./configure \ + --cache gcc.cache CC="ccache gcc" \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} + make -s -j${BUILD_JOBS} clean + time make -s -j${BUILD_JOBS} world-bin + # gcc, cassert on, dtrace off + always: + gcc_a_warning_script: | + time ./configure \ + --cache gcc.cache CC="ccache gcc" \ + --enable-cassert \ + ${LINUX_CONFIGURE_FEATURES} + make -s -j${BUILD_JOBS} clean + time make -s -j${BUILD_JOBS} world-bin + # clang, cassert off, dtrace off + always: + clang_warning_script: | + time ./configure \ + --cache clang.cache CC="ccache clang" \ + ${LINUX_CONFIGURE_FEATURES} + make -s -j${BUILD_JOBS} clean + time make -s -j${BUILD_JOBS} world-bin + # clang, cassert on, dtrace on + always: + clang_a_warning_script: | + time ./configure \ + --cache clang.cache CC="ccache clang" \ + --enable-cassert \ + --enable-dtrace \ + ${LINUX_CONFIGURE_FEATURES} + make -s -j${BUILD_JOBS} clean + time make -s -j${BUILD_JOBS} world-bin + # cross-compile to windows + always: + mingw_cross_warning_script: | + time ./configure \ + --host=x86_64-w64-mingw32 \ + --enable-cassert \ + CC="ccache x86_64-w64-mingw32-gcc" + make -s -j${BUILD_JOBS} clean + time make -s -j${BUILD_JOBS} world-bin + + ### + # Verify docs can be built + ### + + # XXX: Only do this if there have been changes in doc/ since last build + always: + docs_build_script: | + time ./configure \ + --cache gcc.cache CC="ccache gcc" + time make -s -j${BUILD_JOBS} -C doc + + always: + upload_caches: ccache diff --git a/src/tools/ci/README b/src/tools/ci/README new file mode 100644 index 00000000000..88487e07561 --- /dev/null +++ b/src/tools/ci/README @@ -0,0 +1,63 @@ +Postgres Continuous Integration (CI) +==================================== + +Postgres has two forms of CI: + +1) All supported branches in the main postgres repository are continuously + tested via the the the buildfarm. As this covers only the main repository, + it cannot be used during development of features. + + For details see https://buildfarm.postgresql.org/ + +2) For not yet merged development work, CI can be enabled for some git hosting + providers. This allows developers to test patches on a number of platforms + before they are merged (or even submitted). + + +Configuring CI on personal repositories +======================================= + +Currently postgres contains CI support utilizing cirrus-ci. cirrus-ci +currently is only available for github. + + +Enabling cirrus-ci in a github repository +========================================= + +To enable cirrus-ci on a repository, go to +https://github.com/marketplace/cirrus-ci and select "Public +Repositories". Then "Install it for free" and "Complete order". The next page +allows to configure which repositories cirrus-ci has access too. Choose the +relevant repository and "Install". + +See also https://cirrus-ci.org/guide/quick-start/ + +Once enabled on a repository, future commits and pull-requests in that +repository will automatically trigger CI builds. These are visible from the +commit history / PRs, and can also be viewed in the cirrus-ci UI at +https://cirrus-ci.com/github/<username>/<reponame>/ + + +Controlling CI via commit messages +================================== + +The behavior of CI can be controlled by special content in commit +messages. Currently the following controls are available: + +- ci-os-only: {(freebsd|linux|macos|windows)} + + Only runs CI on operating systems specified. This can be useful when + addressing portability issues affecting only a subset of platforms. + + +Images used for CI +================== + +To keep CI times tolerable, most platforms use pre-generated images. Some +platforms use containers, others use full VMs. Images for both are generated +separately from CI runs, otherwise each git repository that is being tested +would need to build its own set of containers, which would be wasteful (both +in space and time. + +These images are built, on a daily basis, from the specifications in +github.com/anarazel/pg-vm-images/ diff --git a/src/tools/ci/cores_backtrace.sh b/src/tools/ci/cores_backtrace.sh new file mode 100755 index 00000000000..549e5afce2b --- /dev/null +++ b/src/tools/ci/cores_backtrace.sh @@ -0,0 +1,50 @@ +#! /bin/sh + +if [ $# -ne 2 ]; then + echo "cores_backtrace.sh <os> <directory>" + exit 1 +fi + +os=$1 +directory=$2 + +case $os in + freebsd|linux|macos) + ;; + *) + echo "unsupported operating system ${os}" + exit 1 + ;; +esac + +first=1 +for corefile in $(find "$directory" -type f) ; do + if [ "$first" -eq 1 ]; then + first=0 + else + # to make it easier to separate the different crash reports + echo -e '\n\n' + fi + + if [ "$os" = 'macos' ]; then + lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' + else + auxv=$(gdb --quiet --core ${corefile} --batch -ex 'info auxv' 2>/dev/null) + if [ $? -ne 0 ]; then + echo "could not process ${corefile}" + continue + fi + + if [ "$os" = 'freebsd' ]; then + binary=$(echo "$auxv" | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g") + elif [ "$os" = 'linux' ]; then + binary=$(echo "$auxv" | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g") + else + echo 'should not get here' + exit 1 + fi + + echo "dumping ${corefile} for ${binary}" + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" "$binary" "$corefile" 2>/dev/null + fi +done diff --git a/src/tools/ci/gcp_freebsd_repartition.sh b/src/tools/ci/gcp_freebsd_repartition.sh new file mode 100755 index 00000000000..2d5e1738998 --- /dev/null +++ b/src/tools/ci/gcp_freebsd_repartition.sh @@ -0,0 +1,28 @@ +#!/bin/sh + +set -e +set -x + +# The default filesystem on freebsd gcp images is very slow to run tests on, +# due to its 32KB block size +# +# XXX: It'd probably better to fix this in the image, using something like +# https://people.freebsd.org/~lidl/blog/re-root.html + +# fix backup partition table after resize +gpart recover da0 +gpart show da0 +# kill swap, so we can delete a partition +swapoff -a || true +# (apparently we can only have 4!?) +gpart delete -i 3 da0 +gpart add -t freebsd-ufs -l data8k -a 4096 da0 +gpart show da0 +newfs -U -b 8192 /dev/da0p3 + +# Migrate working directory +du -hs $CIRRUS_WORKING_DIR +mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig +mkdir $CIRRUS_WORKING_DIR +mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR +cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/ diff --git a/src/tools/ci/pg_ci_base.conf b/src/tools/ci/pg_ci_base.conf new file mode 100644 index 00000000000..d8faa9c26c1 --- /dev/null +++ b/src/tools/ci/pg_ci_base.conf @@ -0,0 +1,14 @@ +# Tends to produce too many core files, taking a long time +restart_after_crash = false + +# So that tests using the "manually" started postgres on windows can use +# prepared statements +max_prepared_transactions = 10 + +# Settings that make logs more useful +log_autovacuum_min_duration = 0 +log_checkpoints = true +log_connections = true +log_disconnections = true +log_line_prefix = '%m [%p][%b] %q[%a][%v:%x] ' +log_lock_waits = true diff --git a/src/tools/ci/windows_build_config.pl b/src/tools/ci/windows_build_config.pl new file mode 100644 index 00000000000..b0d4360c748 --- /dev/null +++ b/src/tools/ci/windows_build_config.pl @@ -0,0 +1,13 @@ +use strict; +use warnings; + +our $config; + +$config->{"tap_tests"} = 1; +$config->{"asserts"} = 1; + +$config->{"openssl"} = "c:/openssl/1.1/"; +$config->{"perl"} = "c:/strawberry/$ENV{DEFAULT_PERL_VERSION}/perl/"; +$config->{"python"} = "c:/python/"; + +1; -- 2.34.0 --mezqkkzp3cvymuis Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v4-0002-windows-Improve-crash-assert-exception-handling.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v1 2/6] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. @ 2021-03-15 16:25 Andres Freund <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Andres Freund @ 2021-03-15 16:25 UTC (permalink / raw) --- .cirrus.yml | 306 ++++++++++++++++++++++++++++++++ .dockerignore | 3 + ci/docker/linux_debian_bullseye | 9 + ci/docker/windows_vs_2019 | 82 +++++++++ ci/freebsd_gcp_repartition.sh | 28 +++ ci/pg_ci_base.conf | 12 ++ ci/windows_build_config.pl | 3 + 7 files changed, 443 insertions(+) create mode 100644 .cirrus.yml create mode 100644 .dockerignore create mode 100644 ci/docker/linux_debian_bullseye create mode 100644 ci/docker/windows_vs_2019 create mode 100755 ci/freebsd_gcp_repartition.sh create mode 100644 ci/pg_ci_base.conf create mode 100644 ci/windows_build_config.pl diff --git a/.cirrus.yml b/.cirrus.yml new file mode 100644 index 00000000000..9f513f7d10a --- /dev/null +++ b/.cirrus.yml @@ -0,0 +1,306 @@ +env: + # accelerate initial clone, but a bit of depth so that concurrent tasks work + CIRRUS_CLONE_DEPTH: 100 + # Useful to be able to analyse what in a script takes long + CIRRUS_LOG_TIMESTAMP: true + # target to test, for all but windows + CHECK: check-world + CHECKFLAGS: -Otarget + PGCTLTIMEOUT: 120 + CCACHE_MAXSIZE: "500M" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + + +task: + name: FreeBSD + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-freebsd-13-0 + platform: freebsd + cpu: 2 + memory: 2G + disk: 50 + env: + CCACHE_DIR: "/tmp/ccache_dir" + + ccache_cache: + folder: "/tmp/ccache_dir" + sysinfo_script: + - export || true + sysconfig_script: + - sudo sysctl kern.corefile='/tmp/%N.%P.core' + repartition_script: + - ci/freebsd_gcp_repartition.sh + create_user_script: + - pw useradd postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + + configure_script: + - su postgres -c './configure --enable-cassert --enable-debug --enable-tap-tests --with-includes=/usr/local/include --with-libs=/usr/local/lib CC="ccache cc"' + build_script: + - su postgres -c 'gmake -s -j3 && gmake -s -j3 -C contrib' + upload_caches: + - ccache + + check_world_script: + - su postgres -c 'time gmake -s -j2 ${CHECK} ${CHECKFLAGS}' + + on_failure: + core_script: + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; echo dumping $corefile for $binary ; gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile ; done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Linux + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-bullseye + platform: linux + cpu: 4 + memory: 8G + nested_virtualization: false + env: + CCACHE_DIR: "/tmp/ccache_dir" + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + TIMEOUT_CMD: timeout -s KILL -v 25m + ccache_cache: + folder: "/tmp/ccache_dir" + + sysinfo_script: + - id + - uname -a + - cat /proc/cmdline + - lsblk + - ulimit -a -H + - ulimit -a -S + - export + sysconfig_script: + - useradd -m postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf + - su postgres -c 'ulimit -l -H' + - su postgres -c 'ulimit -l -S' + - echo '/tmp/%e-%s-%p.core' > /proc/sys/kernel/core_pattern + + configure_script: + - echo "COPT=-O0 -ggdb" > src/Makefile.custom + - su postgres -c './configure --enable-cassert --enable-debug --enable-tap-tests --with-tcl --with-python --with-perl --with-ldap --with-openssl --with-icu --with-llvm CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang"' + build_script: + - su postgres -c 'make -s -j4 && make -s -j4 -C contrib' + upload_caches: + - ccache + + tests_script: + - su postgres -c 'ulimit -c unlimited ; ${TIMEOUT_CMD} make -s ${CHECK} ${CHECKFLAGS} -j8' + + on_failure: + cores_script: + - for corefile in $(find /tmp/ -name '*.core' 2>/dev/null) ; do binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; echo dumping $corefile for $binary ; gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile ; done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: macOS + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' + osx_instance: + image: big-sur-base + env: + CIRRUS_WORKING_DIR: ${HOME}/pgsql/ + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + CCACHE_DIR: ${HOME}/ccache + HOMEBREW_CACHE: ${HOME}/homebrew-cache + PERL5LIB: ${HOME}/perl5/lib/perl5 + + sysinfo_script: + - id + - export + ccache_cache: + folder: ${CCACHE_DIR} + homebrew_cache: + folder: ${HOMEBREW_CACHE} + perl_cache: + folder: ~/perl5 + + cpan_install_script: + - perl -mIPC::Run -e 1 || cpan -T IPC::Run + - perl -mIO::Pty -e 1 || cpan -T IO::Pty + upload_caches: + - perl + core_install_script: + - sudo chmod 777 /cores + homebrew_install_script: + - brew install make coreutils ccache + upload_caches: + - homebrew + + configure_script: + - ./configure --prefix=$HOME/install --enable-cassert --enable-debug --enable-tap-tests CC="ccache gcc" CFLAGS="-O0" + build_script: + - gmake -s -j12 && gmake -s -j12 -C contrib + upload_caches: + - ccache + + tests_script: + - ulimit -c unlimited + - ulimit -n 1024 + - gtimeout -s KILL -v 20m gmake -s -j12 ${CHECK} ${CHECKFLAGS} + + on_failure: + cores_script: + - for corefile in $(find /cores/ -name 'core.*' 2>/dev/null) ; do lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' ; done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Windows + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + windows_container: + dockerfile: ci/docker/windows_vs_2019 + cpu: 4 + memory: 4G + env: + PROVE_FLAGS: -j10 + # The default working dir is in a directory msbuild complains about + CIRRUS_WORKING_DIR: "c:/cirrus" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + # Avoid re-installing over and over + NO_TEMP_INSTALL: 1 + + sysinfo_script: + - chcp + - systeminfo + - powershell -Command get-psdrive -psprovider filesystem + - ps: Get-Item -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' + - set + + configure_script: + - copy ci\windows_build_config.pl src\tools\msvc\config.pl + - vcvarsall x64 + - perl src/tools/msvc/mkvcbuild.pl + build_script: + - vcvarsall x64 + # Disable file tracker, we're never going to rebuild... + - msbuild -m /p:TrackFileAccess=false pgsql.sln + tempinstall_script: + - perl src\tools\msvc\install.pl tmp_install + + check_script: + - perl src/tools/msvc/vcregress.pl check parallel + startcreate_script: + - tmp_install\bin\pg_ctl.exe initdb -D tmp_check\db -l tmp_check\initdb.log + - echo include '%TEMP_CONFIG%' >> tmp_check\db\postgresql.conf + - tmp_install\bin\pg_ctl.exe start -D tmp_check\db -l tmp_check\postmaster.log + isolationcheck_script: + - perl src/tools/msvc/vcregress.pl isolationcheck + modulescheck_script: + - perl src/tools/msvc/vcregress.pl modulescheck + contribcheck_script: + - perl src/tools/msvc/vcregress.pl contribcheck + plcheck_script: + - perl src/tools/msvc/vcregress.pl plcheck + stop_script: + - tmp_install\bin\pg_ctl.exe stop -D tmp_check\db -l tmp_check\postmaster.log + subscriptioncheck_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\subscription\ + authentication_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\authentication\ + recoverycheck_script: + - perl src/tools/msvc/vcregress.pl recoverycheck + bincheck_script: + - perl src/tools/msvc/vcregress.pl bincheck + upgradecheck_script: + - perl src/tools/msvc/vcregress.pl upgradecheck + ecpgcheck_script: + # tries to build additional stuff + - vcvarsall x64 + # References ecpg_regression.proj in the current dir + - cd src\tools\msvc + - perl vcregress.pl ecpgcheck + + always: + cat_dumps_script: + - cat crashlog.txt || true + dump_artifacts: + path: "crashlog.txt" + type: text/plain + + on_failure: + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: CompilerWarnings + depends_on: + - Linux + # not run task count as a success, so we need to recheck Linux' condition here :/ + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + container: + dockerfile: ci/docker/linux_debian_bullseye + env: + CCACHE_SIZE: "4GB" + CCACHE_DIR: "/tmp/ccache_dir" + ccache_cache: + folder: "/tmp/ccache_dir" + setup_script: + - echo "COPT=-Werror" > src/Makefile.custom + - gcc -v + - clang -v + # gcc with asserts disabled + always: + gcc_warning_script: + - ./configure --cache gcc.cache CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # gcc with asserts enabled + always: + gcc_a_warning_script: + - ./configure --cache gcc.cache --enable-cassert CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts disabled + always: + clang_warning_script: + - ./configure --cache clang.cache CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts enabled + always: + clang_a_warning_script: + - ./configure --cache clang.cache --enable-cassert CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000000..3fceab2e97b --- /dev/null +++ b/.dockerignore @@ -0,0 +1,3 @@ +# Ignore everything, except ci/ +* +!ci/* diff --git a/ci/docker/linux_debian_bullseye b/ci/docker/linux_debian_bullseye new file mode 100644 index 00000000000..3c1d1a4583a --- /dev/null +++ b/ci/docker/linux_debian_bullseye @@ -0,0 +1,9 @@ +FROM debian:bullseye +RUN apt-get -y update && apt-get -y upgrade && \ + DEBIAN_FRONTEND=noninteractive apt-get -y --no-install-recommends install \ + gcc g++ libreadline-dev flex bison make perl libipc-run-perl clang \ + llvm-dev libperl-dev libpython3-dev tcl-dev libldap2-dev libicu-dev \ + docbook-xml docbook-xsl fop libxml2-utils xsltproc krb5-admin-server \ + krb5-kdc krb5-user slapd ldap-utils libssl-dev pkg-config locales-all \ + liburing-dev python3-distutils ccache gdb && \ + apt-get clean diff --git a/ci/docker/windows_vs_2019 b/ci/docker/windows_vs_2019 new file mode 100644 index 00000000000..e5b4810ac9e --- /dev/null +++ b/ci/docker/windows_vs_2019 @@ -0,0 +1,82 @@ +# escape=` + +# We used to use the visual studio container, but it's too outdated now +FROM cirrusci/windowsservercore:2019 + +SHELL ["powershell", "-NoLogo", "-NoProfile", "-Command"] + + +RUN ` + New-Item -Path 'HKLM:\SOFTWARE\Policies\Microsoft\VisualStudio' ; ` + New-Item -Path 'HKLM:\SOFTWARE\Policies\Microsoft\VisualStudio\Setup' ; ` + New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\VisualStudio\Setup' -Name KeepDownloadedPayloads -Value 0 -PropertyType DWord + + +# Install commandline debugger and log all crashes to c:\cirrus\crashlog.txt +# +# Done manually as doing this via chocolatey / the installer directly, ends up +# with a lot of unnecessary chaff, making the layer unnecessarily large. +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + curl.exe -sSL -o 'windsdksetup.exe' https://download.microsoft.com/download/9/7/9/97982c1d-d687-41be-9dd3-6d01e52ceb68/windowssdk/winsdk... ; ` + Start-Process -FilePath ".\windsdksetup.exe" ` + -ArgumentList '/Features OptionId.WindowsDesktopDebuggers /layout c:\t\sdk /quiet /norestart /log c:\t\sdk.log' ` + -Wait ; ` + ` + Start-Process -FilePath msiexec.exe ` + -ArgumentList '/a \"C:\t\sdk\Installers\X64 Debuggers And Tools-x64_en-us.msi\" /qb /log install2.log' ` + -Wait ; ` + C:\Windows` Kits\10\Debuggers\x64\cdb.exe -version ; ` + ` + cd c:\ ; ` + Remove-Item C:\t\* -Force -Recurse ; ` + Remove-Item C:\t -Force -Recurse ; ` + Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Debugger' -Value '\"C:\Windows Kits\10\Debuggers\x64\cdb.exe\" -p %ld -e %ld -g -kqm -c \".lines -e; .symfix+ ;.logappend c:\cirrus\crashlog.txt ; !peb; ~*kP ; .logclose ; q \"' ; ` + New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Auto' -Value 1 -PropertyType DWord ; ` + Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name Debugger; ` + setx PATH \"C:\Windows Kits\10\Debuggers\x64;$Env:PATH\" /m + + +# Install perl and bison. +# +# Done manually as choco takes a lot longer. I think it's download issues with +# powershell's download stuff? That's wy curl.exe is directly used here at least... +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + curl.exe -sSL -o perl.zip ` + https://strawberryperl.com/download/5.30.3.1/strawberry-perl-5.30.3.1-64bit-portable.zip ; ` + 7z.exe x .\perl.zip -xr!c -oc:\strawberry ; ` + ` + curl.exe -sSL -o winflexbison.zip ` + https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip ; ` + 7z.exe x .\winflexbison.zip -oc:\winflexbison ; ` + Rename-Item -Path c:\winflexbison\win_flex.exe c:\winflexbison\flex.exe ; ` + Rename-Item -Path c:\winflexbison\win_bison.exe c:\winflexbison\bison.exe ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse ; ` + setx PATH \"C:\strawberry\perl\bin;C:\winflexbison;C:\Program Files\Git\usr\bin;$Env:PATH\" /m + + +# Install visual studio +# +# Adding VS path to vcvarsall.bat so user of container doesn't need to know the full path +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + curl.exe -sSL -o c:\t\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe ; ` + Start-Process -Wait ` + -FilePath c:\t\vs_buildtools.exe ` + -ArgumentList ` + '--quiet', '--wait', '--norestart', '--nocache', ` + '--installPath', 'c:\BuildTools', ` + '--add', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', ` + '--add', 'Microsoft.VisualStudio.Component.Windows10SDK.20348' ; ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse ; ` + Remove-Item -Force -Recurse ${Env:TEMP}\*; ` + Remove-Item -Force -Recurse \"${Env:ProgramData}\Package Cache\" ; ` + setx PATH \"c:\BuildTools\VC\Auxiliary\Build;$Env:PATH\" /m diff --git a/ci/freebsd_gcp_repartition.sh b/ci/freebsd_gcp_repartition.sh new file mode 100755 index 00000000000..2d5e1738998 --- /dev/null +++ b/ci/freebsd_gcp_repartition.sh @@ -0,0 +1,28 @@ +#!/bin/sh + +set -e +set -x + +# The default filesystem on freebsd gcp images is very slow to run tests on, +# due to its 32KB block size +# +# XXX: It'd probably better to fix this in the image, using something like +# https://people.freebsd.org/~lidl/blog/re-root.html + +# fix backup partition table after resize +gpart recover da0 +gpart show da0 +# kill swap, so we can delete a partition +swapoff -a || true +# (apparently we can only have 4!?) +gpart delete -i 3 da0 +gpart add -t freebsd-ufs -l data8k -a 4096 da0 +gpart show da0 +newfs -U -b 8192 /dev/da0p3 + +# Migrate working directory +du -hs $CIRRUS_WORKING_DIR +mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig +mkdir $CIRRUS_WORKING_DIR +mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR +cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/ diff --git a/ci/pg_ci_base.conf b/ci/pg_ci_base.conf new file mode 100644 index 00000000000..637e3cfb343 --- /dev/null +++ b/ci/pg_ci_base.conf @@ -0,0 +1,12 @@ +# Tends to produce too many core files, taking a long time +restart_after_crash = false + +# So that tests using the "manually" started postgres on windows can use +# prepared statements +max_prepared_transactions=10 + +# Settings that make logs more useful +log_line_prefix='%m [%p][%b][%v:%x] ' +log_checkpoints = true +log_connections = true +log_disconnections = true diff --git a/ci/windows_build_config.pl b/ci/windows_build_config.pl new file mode 100644 index 00000000000..a8abfa2a02f --- /dev/null +++ b/ci/windows_build_config.pl @@ -0,0 +1,3 @@ +$config->{"tap_tests"} = 1; +$config->{"asserts"} = 1; +1; -- 2.32.0.rc2 --ntohwnblr5nw7rqz Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v1-0003-Reference-test-binary-using-TESTDIR-in-001_libpq_.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v3 02/17] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. @ 2021-03-15 16:25 Andres Freund <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Andres Freund @ 2021-03-15 16:25 UTC (permalink / raw) --- .cirrus.yml | 395 ++++++++++++++++++++++++++++++++ .dockerignore | 3 + ci/docker/linux_debian_bullseye | 13 ++ ci/docker/windows_vs_2019 | 105 +++++++++ ci/freebsd_gcp_repartition.sh | 28 +++ ci/pg_ci_base.conf | 12 + ci/windows_build_config.pl | 10 + 7 files changed, 566 insertions(+) create mode 100644 .cirrus.yml create mode 100644 .dockerignore create mode 100644 ci/docker/linux_debian_bullseye create mode 100644 ci/docker/windows_vs_2019 create mode 100755 ci/freebsd_gcp_repartition.sh create mode 100644 ci/pg_ci_base.conf create mode 100644 ci/windows_build_config.pl diff --git a/.cirrus.yml b/.cirrus.yml new file mode 100644 index 00000000000..f75bdce6dec --- /dev/null +++ b/.cirrus.yml @@ -0,0 +1,395 @@ +env: + # accelerate initial clone, but a bit of depth so that concurrent tasks work + CIRRUS_CLONE_DEPTH: 100 + # Useful to be able to analyse what in a script takes long + CIRRUS_LOG_TIMESTAMP: true + # target to test, for all but windows + CHECK: check-world + CHECKFLAGS: -Otarget + PGCTLTIMEOUT: 120 + CCACHE_MAXSIZE: "500M" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl + + +task: + name: FreeBSD + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-freebsd-13-0 + platform: freebsd + cpu: 2 + memory: 2G + disk: 50 + env: + CCACHE_DIR: "/tmp/ccache_dir" + + ccache_cache: + folder: "/tmp/ccache_dir" + sysinfo_script: + - export || true + sysconfig_script: + - sudo sysctl kern.corefile='/tmp/%N.%P.core' + repartition_script: + - ci/freebsd_gcp_repartition.sh + create_user_script: + - pw useradd postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/lib/tcl8.6/ \ + --with-uuid=bsd \ + \ + --with-includes=/usr/local/include --with-libs=/usr/local/lib \ + CC="ccache cc"' + build_script: + - su postgres -c 'gmake -s -j3 && gmake -s -j3 -C contrib' + upload_caches: + - ccache + + tests_script: + - su postgres -c 'time gmake -s -j2 ${CHECK} ${CHECKFLAGS}' + + on_failure: + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Linux + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-bullseye + platform: linux + cpu: 4 + memory: 2G + nested_virtualization: false + env: + CCACHE_DIR: "/tmp/ccache_dir" + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + ccache_cache: + folder: "/tmp/ccache_dir" + + sysinfo_script: + - id + - uname -a + - cat /proc/cmdline + - lsblk + - ulimit -a -H + - ulimit -a -S + - export + sysconfig_script: + - useradd -m postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf + - su postgres -c 'ulimit -l -H' + - su postgres -c 'ulimit -l -S' + - echo '/tmp/%e-%s-%p.core' > /proc/sys/kernel/core_pattern + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-llvm \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-systemd \ + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ \ + --with-uuid=e2fs \ + \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" CFLAGS="-O0 -ggdb"' + build_script: + - su postgres -c 'make -s -j4 && make -s -j4 -C contrib' + upload_caches: + - ccache + + tests_script: | + su postgres -c '\ + ulimit -c unlimited; \ + make -s ${CHECK} ${CHECKFLAGS} -j8 \ + ' + + on_failure: + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile ; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: macOS + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' + osx_instance: + image: big-sur-base + env: + CIRRUS_WORKING_DIR: ${HOME}/pgsql/ + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + CCACHE_DIR: ${HOME}/ccache + HOMEBREW_CACHE: ${HOME}/homebrew-cache + PERL5LIB: ${HOME}/perl5/lib/perl5 + + sysinfo_script: + - id + - export + ccache_cache: + folder: ${CCACHE_DIR} + homebrew_cache: + folder: ${HOMEBREW_CACHE} + perl_cache: + folder: ~/perl5 + + cpan_install_script: + - perl -mIPC::Run -e 1 || cpan -T IPC::Run + - perl -mIO::Pty -e 1 || cpan -T IO::Pty + upload_caches: + - perl + core_install_script: + - sudo chmod 777 /cores + homebrew_install_script: + - brew install make coreutils ccache icu4c lz4 tcl-tk openldap + upload_caches: + - homebrew + + configure_script: | + LIBS="/usr/local/lib:$LIBS" + INCLUDES="/usr/local/include:$INCLUDES" + + INCLUDES="/usr/local/opt/openssl/include:$INCLUDES" + LIBS="/usr/local/opt/openssl/lib:$LIBS" + + PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" + INCLUDES="/usr/local/opt/icu4c/include:$INCLUDES" + LIBS="/usr/local/opt/icu4c/lib:$LIBS" + + LIBS="/usr/local/opt/openldap/lib:$LIBS" + INCLUDES="/usr/local/opt/openldap/include:$INCLUDES" + + export PKG_CONFIG_PATH + + ./configure \ + --prefix=$HOME/install \ + --with-includes="$INCLUDES" \ + --with-libs="$LIBS" \ + \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + \ + --with-lz4 \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/opt/tcl-tk/lib/ \ + --with-uuid=e2fs \ + \ + CC="ccache gcc" CFLAGS="-O0 -ggdb" + build_script: + - gmake -s -j12 && gmake -s -j12 -C contrib + upload_caches: + - ccache + + tests_script: + - ulimit -c unlimited + - ulimit -n 1024 + - gmake -s -j12 ${CHECK} ${CHECKFLAGS} + + on_failure: + cores_script: | + for corefile in $(find /cores/ -name 'core.*' 2>/dev/null) ; do + lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' ; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Windows + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + windows_container: + dockerfile: ci/docker/windows_vs_2019 + cpu: 4 + memory: 4G + env: + PROVE_FLAGS: -j10 + # The default working dir is in a directory msbuild complains about + CIRRUS_WORKING_DIR: "c:/cirrus" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + # Avoid re-installing over and over + NO_TEMP_INSTALL: 1 + + sysinfo_script: + - chcp + - systeminfo + - powershell -Command get-psdrive -psprovider filesystem + - ps: Get-Item -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' + - set + + configure_script: + - copy ci\windows_build_config.pl src\tools\msvc\config.pl + - vcvarsall x64 + - perl src/tools/msvc/mkvcbuild.pl + build_script: + - vcvarsall x64 + # Disable file tracker, we're never going to rebuild... + - msbuild -m /p:TrackFileAccess=false pgsql.sln + tempinstall_script: + # Installation on windows currently only completely works from src\tools\msvc + - cd src\tools\msvc && perl .\install.pl %CIRRUS_WORKING_DIR%\tmp_install + + check_test_script: + - perl src/tools/msvc/vcregress.pl check parallel + startcreate_test_script: + - tmp_install\bin\pg_ctl.exe initdb -D tmp_check\db -l tmp_check\initdb.log + - echo include '%TEMP_CONFIG%' >> tmp_check\db\postgresql.conf + - tmp_install\bin\pg_ctl.exe start -D tmp_check\db -l tmp_check\postmaster.log + plcheck_test_script: + - perl src/tools/msvc/vcregress.pl plcheck + isolationcheck_test_script: + - perl src/tools/msvc/vcregress.pl isolationcheck + modulescheck_test_script: + - perl src/tools/msvc/vcregress.pl modulescheck + contribcheck_test_script: + - perl src/tools/msvc/vcregress.pl contribcheck + stop_test_script: + - tmp_install\bin\pg_ctl.exe stop -D tmp_check\db -l tmp_check\postmaster.log + ssl_test_script: + - set with_ssl=openssl + - perl src/tools/msvc/vcregress.pl taptest .\src\test\ssl\ + subscriptioncheck_test_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\subscription\ + authentication_test_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\authentication\ + recoverycheck_test_script: + - perl src/tools/msvc/vcregress.pl recoverycheck + bincheck_test_script: + - perl src/tools/msvc/vcregress.pl bincheck + upgradecheck_test_script: + - perl src/tools/msvc/vcregress.pl upgradecheck + ecpgcheck_test_script: + # tries to build additional stuff + - vcvarsall x64 + # References ecpg_regression.proj in the current dir + - cd src\tools\msvc + - perl vcregress.pl ecpgcheck + + always: + cores_script: + - cat crashlog.txt || true + dump_artifacts: + path: "crashlog.txt" + type: text/plain + + on_failure: + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: CompilerWarnings + depends_on: + - Linux + # task that did not run count as a success, so we need to recheck Linux' condition here :/ + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + container: + dockerfile: ci/docker/linux_debian_bullseye + env: + CCACHE_SIZE: "4GB" + CCACHE_DIR: "/tmp/ccache_dir" + ccache_cache: + folder: "/tmp/ccache_dir" + setup_script: + - echo "COPT=-Werror" > src/Makefile.custom + - gcc -v + - clang -v + # gcc with asserts disabled + always: + gcc_warning_script: + - ./configure --cache gcc.cache CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # gcc with asserts enabled + always: + gcc_a_warning_script: + - ./configure --cache gcc.cache --enable-cassert CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts disabled + always: + clang_warning_script: + - ./configure --cache clang.cache CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts enabled + always: + clang_a_warning_script: + - ./configure --cache clang.cache --enable-cassert CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000000..3fceab2e97b --- /dev/null +++ b/.dockerignore @@ -0,0 +1,3 @@ +# Ignore everything, except ci/ +* +!ci/* diff --git a/ci/docker/linux_debian_bullseye b/ci/docker/linux_debian_bullseye new file mode 100644 index 00000000000..f6c1782f16b --- /dev/null +++ b/ci/docker/linux_debian_bullseye @@ -0,0 +1,13 @@ +FROM debian:bullseye +RUN \ + apt-get -y update && \ + apt-get -y upgrade && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + git build-essential gcc g++ libreadline-dev flex bison make perl libipc-run-perl \ + libio-pty-perl clang llvm-dev libperl-dev libpython3-dev tcl-dev libldap2-dev \ + libicu-dev docbook-xml docbook-xsl fop libxml2-utils xsltproc krb5-admin-server \ + krb5-kdc krb5-user slapd ldap-utils libssl-dev pkg-config locales-all liblz4-dev \ + libsystemd-dev libxml2-dev libxslt1-dev python3-dev libkrb5-dev libpam-dev \ + libkrb5-*-heimdal uuid-dev gettext \ + liburing-dev python3-distutils ccache && \ + apt-get clean diff --git a/ci/docker/windows_vs_2019 b/ci/docker/windows_vs_2019 new file mode 100644 index 00000000000..e09ca0d5825 --- /dev/null +++ b/ci/docker/windows_vs_2019 @@ -0,0 +1,105 @@ +# escape=` + +# We used to use the visual studio container, but it's too outdated now +FROM cirrusci/windowsservercore:2019 + +SHELL ["powershell", "-NoLogo", "-NoProfile", "-Command"] + + +RUN ` + New-Item -Path 'HKLM:\SOFTWARE\Policies\Microsoft\VisualStudio' ; ` + New-Item -Path 'HKLM:\SOFTWARE\Policies\Microsoft\VisualStudio\Setup' ; ` + New-ItemProperty -Path 'HKLM:\SOFTWARE\Policies\Microsoft\VisualStudio\Setup' -Name KeepDownloadedPayloads -Value 0 -PropertyType DWord + + +# Install commandline debugger and log all crashes to c:\cirrus\crashlog.txt +# +# Done manually as doing this via chocolatey / the installer directly, ends up +# with a lot of unnecessary chaff, making the layer unnecessarily large. +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + curl.exe -sSL -o 'windsdksetup.exe' https://download.microsoft.com/download/9/7/9/97982c1d-d687-41be-9dd3-6d01e52ceb68/windowssdk/winsdk... ; ` + Start-Process -Wait -FilePath ".\windsdksetup.exe" ` + -ArgumentList '/Features OptionId.WindowsDesktopDebuggers /layout c:\t\sdk /quiet /norestart /log c:\t\sdk.log' ` + ; ` + ` + Start-Process -Wait -FilePath msiexec.exe ` + -ArgumentList '/a \"C:\t\sdk\Installers\X64 Debuggers And Tools-x64_en-us.msi\" /qb /log install2.log' ` + ; ` + C:\Windows` Kits\10\Debuggers\x64\cdb.exe -version ; ` + ` + cd c:\ ; ` + Remove-Item C:\t\* -Force -Recurse ; ` + Remove-Item C:\t -Force -Recurse ; ` + Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Debugger' -Value '\"C:\Windows Kits\10\Debuggers\x64\cdb.exe\" -p %ld -e %ld -g -kqm -c \".lines -e; .symfix+ ;.logappend c:\cirrus\crashlog.txt ; !peb; ~*kP ; .logclose ; q \"' ; ` + New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Auto' -Value 1 -PropertyType DWord ; ` + Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name Debugger; ` + setx PATH \"C:\Windows Kits\10\Debuggers\x64;$Env:PATH\" /m + + +# Install perl, python, flex and bison. +# +# Done manually as choco takes a lot longer. I think it's download issues with +# powershell's download stuff? That's wy curl.exe is directly used here at least... +# +# Using perl 5.26.3.1 for now, as newer versions don't currently work correctly +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + curl.exe -sSL -o perl.zip ` + https://strawberryperl.com/download/5.26.3.1/strawberry-perl-5.26.3.1-64bit-portable.zip ; ` + 7z.exe x .\perl.zip -xr!c -oc:\strawberry ; ` + ` + curl.exe -sSL -o python.exe https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe ; ` + Start-Process -Wait -FilePath ".\python.exe" ` + -ArgumentList ` + '/quiet', 'SimpleInstall=1', 'PrependPath=1', 'CompileAll=1', ` + 'TargetDir=c:\python\', 'InstallAllUsers=1', 'Shortcuts=0', ` + 'Include_docs=0', 'Include_tcltk=0', 'Include_tests=0' ` + ; ` + ` + curl.exe -sSL -o winflexbison.zip ` + https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip ; ` + 7z.exe x .\winflexbison.zip -oc:\winflexbison ; ` + Rename-Item -Path c:\winflexbison\win_flex.exe c:\winflexbison\flex.exe ; ` + Rename-Item -Path c:\winflexbison\win_bison.exe c:\winflexbison\bison.exe ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse ; ` + setx PATH \"C:\strawberry\perl\bin;C:\winflexbison;C:\Program Files\Git\usr\bin;$Env:PATH\" /m + + +# Install visual studio +# +# Adding VS path to vcvarsall.bat so user of container doesn't need to know the full path +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + curl.exe -sSL -o c:\t\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe ; ` + Start-Process -Wait ` + -FilePath c:\t\vs_buildtools.exe ` + -ArgumentList ` + '--quiet', '--wait', '--norestart', '--nocache', ` + '--installPath', 'c:\BuildTools', ` + '--add', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', ` + '--add', 'Microsoft.VisualStudio.Component.Windows10SDK.20348' ; ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse ; ` + Remove-Item -Force -Recurse ${Env:TEMP}\*; ` + Remove-Item -Force -Recurse \"${Env:ProgramData}\Package Cache\" ; ` + setx PATH \"c:\BuildTools\VC\Auxiliary\Build;$Env:PATH\" /m + + +# Install openssl +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + curl.exe -o openssl-setup.exe -sSL https://slproweb.com/download/Win64OpenSSL-1_1_1L.exe ; ` + Start-Process -Wait -FilePath ".\openssl-setup.exe" ` + -ArgumentList '/DIR=c:\openssl\1.1.1l\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse diff --git a/ci/freebsd_gcp_repartition.sh b/ci/freebsd_gcp_repartition.sh new file mode 100755 index 00000000000..2d5e1738998 --- /dev/null +++ b/ci/freebsd_gcp_repartition.sh @@ -0,0 +1,28 @@ +#!/bin/sh + +set -e +set -x + +# The default filesystem on freebsd gcp images is very slow to run tests on, +# due to its 32KB block size +# +# XXX: It'd probably better to fix this in the image, using something like +# https://people.freebsd.org/~lidl/blog/re-root.html + +# fix backup partition table after resize +gpart recover da0 +gpart show da0 +# kill swap, so we can delete a partition +swapoff -a || true +# (apparently we can only have 4!?) +gpart delete -i 3 da0 +gpart add -t freebsd-ufs -l data8k -a 4096 da0 +gpart show da0 +newfs -U -b 8192 /dev/da0p3 + +# Migrate working directory +du -hs $CIRRUS_WORKING_DIR +mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig +mkdir $CIRRUS_WORKING_DIR +mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR +cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/ diff --git a/ci/pg_ci_base.conf b/ci/pg_ci_base.conf new file mode 100644 index 00000000000..637e3cfb343 --- /dev/null +++ b/ci/pg_ci_base.conf @@ -0,0 +1,12 @@ +# Tends to produce too many core files, taking a long time +restart_after_crash = false + +# So that tests using the "manually" started postgres on windows can use +# prepared statements +max_prepared_transactions=10 + +# Settings that make logs more useful +log_line_prefix='%m [%p][%b][%v:%x] ' +log_checkpoints = true +log_connections = true +log_disconnections = true diff --git a/ci/windows_build_config.pl b/ci/windows_build_config.pl new file mode 100644 index 00000000000..bf0660416fa --- /dev/null +++ b/ci/windows_build_config.pl @@ -0,0 +1,10 @@ +use strict; +use warnings; + +our $config; + +$config->{"tap_tests"} = 1; +$config->{"asserts"} = 1; +$config->{"openssl"} = "c:/openssl/1.1.1l/"; + +1; -- 2.23.0.385.gbc12974a89 --qozqt6hocm2wibt2 Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v3-0003-fixup-ci-Add-CI-for-FreeBSD-Linux-MacOS-and-Windo.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* [PATCH v5 02/16] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. @ 2021-03-15 16:25 Andres Freund <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: Andres Freund @ 2021-03-15 16:25 UTC (permalink / raw) --- .cirrus.yml | 395 ++++++++++++++++++++++++++++++++ .dockerignore | 3 + ci/docker/linux_debian_bullseye | 13 ++ ci/docker/windows_vs_2019 | 111 +++++++++ ci/freebsd_gcp_repartition.sh | 28 +++ ci/pg_ci_base.conf | 12 + ci/windows_build_config.pl | 13 ++ 7 files changed, 575 insertions(+) create mode 100644 .cirrus.yml create mode 100644 .dockerignore create mode 100644 ci/docker/linux_debian_bullseye create mode 100644 ci/docker/windows_vs_2019 create mode 100755 ci/freebsd_gcp_repartition.sh create mode 100644 ci/pg_ci_base.conf create mode 100644 ci/windows_build_config.pl diff --git a/.cirrus.yml b/.cirrus.yml new file mode 100644 index 00000000000..f75bdce6dec --- /dev/null +++ b/.cirrus.yml @@ -0,0 +1,395 @@ +env: + # accelerate initial clone, but a bit of depth so that concurrent tasks work + CIRRUS_CLONE_DEPTH: 100 + # Useful to be able to analyse what in a script takes long + CIRRUS_LOG_TIMESTAMP: true + # target to test, for all but windows + CHECK: check-world + CHECKFLAGS: -Otarget + PGCTLTIMEOUT: 120 + CCACHE_MAXSIZE: "500M" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + PG_TEST_EXTRA: kerberos ldap ssl + + +task: + name: FreeBSD + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*freebsd.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-freebsd-13-0 + platform: freebsd + cpu: 2 + memory: 2G + disk: 50 + env: + CCACHE_DIR: "/tmp/ccache_dir" + + ccache_cache: + folder: "/tmp/ccache_dir" + sysinfo_script: + - export || true + sysconfig_script: + - sudo sysctl kern.corefile='/tmp/%N.%P.core' + repartition_script: + - ci/freebsd_gcp_repartition.sh + create_user_script: + - pw useradd postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/lib/tcl8.6/ \ + --with-uuid=bsd \ + \ + --with-includes=/usr/local/include --with-libs=/usr/local/lib \ + CC="ccache cc"' + build_script: + - su postgres -c 'gmake -s -j3 && gmake -s -j3 -C contrib' + upload_caches: + - ccache + + tests_script: + - su postgres -c 'time gmake -s -j2 ${CHECK} ${CHECKFLAGS}' + + on_failure: + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECPATH | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Linux + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + compute_engine_instance: + image_project: pg-vm-images-aio + image: family/pg-aio-bullseye + platform: linux + cpu: 4 + memory: 2G + nested_virtualization: false + env: + CCACHE_DIR: "/tmp/ccache_dir" + DEBUGINFOD_URLS: "https://debuginfod.debian.net"; + + ccache_cache: + folder: "/tmp/ccache_dir" + + sysinfo_script: + - id + - uname -a + - cat /proc/cmdline + - lsblk + - ulimit -a -H + - ulimit -a -S + - export + sysconfig_script: + - useradd -m postgres + - chown -R postgres:postgres . + - mkdir -p /tmp/ccache_dir + - chown -R postgres:postgres /tmp/ccache_dir + - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf + - su postgres -c 'ulimit -l -H' + - su postgres -c 'ulimit -l -S' + - echo '/tmp/%e-%s-%p.core' > /proc/sys/kernel/core_pattern + + configure_script: | + su postgres -c './configure \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-gssapi \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + --with-llvm \ + --with-lz4 \ + --with-pam \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-systemd \ + --with-tcl --with-tclconfig=/usr/lib/tcl8.6/ \ + --with-uuid=e2fs \ + \ + CC="ccache gcc" CXX="ccache g++" CLANG="ccache clang" CFLAGS="-O0 -ggdb"' + build_script: + - su postgres -c 'make -s -j4 && make -s -j4 -C contrib' + upload_caches: + - ccache + + tests_script: | + su postgres -c '\ + ulimit -c unlimited; \ + make -s ${CHECK} ${CHECKFLAGS} -j8 \ + ' + + on_failure: + cores_script: | + for corefile in $(find /tmp -name '*.core' 2>/dev/null) ; do + binary=$(gdb -quiet -core $corefile -batch -ex 'info auxv' | grep AT_EXECFN | perl -pe "s/^.*\"(.*)\"\$/\$1/g") ; + echo dumping $corefile for $binary ; + gdb --batch --quiet -ex "thread apply all bt full" -ex "quit" $binary $corefile ; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: macOS + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*(macos|darwin|osx).*' + osx_instance: + image: big-sur-base + env: + CIRRUS_WORKING_DIR: ${HOME}/pgsql/ + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + CCACHE_DIR: ${HOME}/ccache + HOMEBREW_CACHE: ${HOME}/homebrew-cache + PERL5LIB: ${HOME}/perl5/lib/perl5 + + sysinfo_script: + - id + - export + ccache_cache: + folder: ${CCACHE_DIR} + homebrew_cache: + folder: ${HOMEBREW_CACHE} + perl_cache: + folder: ~/perl5 + + cpan_install_script: + - perl -mIPC::Run -e 1 || cpan -T IPC::Run + - perl -mIO::Pty -e 1 || cpan -T IO::Pty + upload_caches: + - perl + core_install_script: + - sudo chmod 777 /cores + homebrew_install_script: + - brew install make coreutils ccache icu4c lz4 tcl-tk openldap + upload_caches: + - homebrew + + configure_script: | + LIBS="/usr/local/lib:$LIBS" + INCLUDES="/usr/local/include:$INCLUDES" + + INCLUDES="/usr/local/opt/openssl/include:$INCLUDES" + LIBS="/usr/local/opt/openssl/lib:$LIBS" + + PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" + INCLUDES="/usr/local/opt/icu4c/include:$INCLUDES" + LIBS="/usr/local/opt/icu4c/lib:$LIBS" + + LIBS="/usr/local/opt/openldap/lib:$LIBS" + INCLUDES="/usr/local/opt/openldap/include:$INCLUDES" + + export PKG_CONFIG_PATH + + ./configure \ + --prefix=$HOME/install \ + --with-includes="$INCLUDES" \ + --with-libs="$LIBS" \ + \ + --enable-cassert --enable-debug --enable-tap-tests \ + --enable-nls \ + \ + --with-icu \ + --with-ldap \ + --with-libxml \ + --with-libxslt \ + \ + --with-lz4 \ + --with-perl \ + --with-python \ + --with-ssl=openssl \ + --with-tcl --with-tclconfig=/usr/local/opt/tcl-tk/lib/ \ + --with-uuid=e2fs \ + \ + CC="ccache gcc" CFLAGS="-O0 -ggdb" + build_script: + - gmake -s -j12 && gmake -s -j12 -C contrib + upload_caches: + - ccache + + tests_script: + - ulimit -c unlimited + - ulimit -n 1024 + - gmake -s -j12 ${CHECK} ${CHECKFLAGS} + + on_failure: + cores_script: | + for corefile in $(find /cores/ -name 'core.*' 2>/dev/null) ; do + lldb -c $corefile --batch -o 'thread backtrace all' -o 'quit' ; + done + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: Windows + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*windows.*' + windows_container: + dockerfile: ci/docker/windows_vs_2019 + cpu: 4 + memory: 4G + env: + PROVE_FLAGS: -j10 + # The default working dir is in a directory msbuild complains about + CIRRUS_WORKING_DIR: "c:/cirrus" + TEMP_CONFIG: ${CIRRUS_WORKING_DIR}/ci/pg_ci_base.conf + # Avoid re-installing over and over + NO_TEMP_INSTALL: 1 + + sysinfo_script: + - chcp + - systeminfo + - powershell -Command get-psdrive -psprovider filesystem + - ps: Get-Item -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' + - set + + configure_script: + - copy ci\windows_build_config.pl src\tools\msvc\config.pl + - vcvarsall x64 + - perl src/tools/msvc/mkvcbuild.pl + build_script: + - vcvarsall x64 + # Disable file tracker, we're never going to rebuild... + - msbuild -m /p:TrackFileAccess=false pgsql.sln + tempinstall_script: + # Installation on windows currently only completely works from src\tools\msvc + - cd src\tools\msvc && perl .\install.pl %CIRRUS_WORKING_DIR%\tmp_install + + check_test_script: + - perl src/tools/msvc/vcregress.pl check parallel + startcreate_test_script: + - tmp_install\bin\pg_ctl.exe initdb -D tmp_check\db -l tmp_check\initdb.log + - echo include '%TEMP_CONFIG%' >> tmp_check\db\postgresql.conf + - tmp_install\bin\pg_ctl.exe start -D tmp_check\db -l tmp_check\postmaster.log + plcheck_test_script: + - perl src/tools/msvc/vcregress.pl plcheck + isolationcheck_test_script: + - perl src/tools/msvc/vcregress.pl isolationcheck + modulescheck_test_script: + - perl src/tools/msvc/vcregress.pl modulescheck + contribcheck_test_script: + - perl src/tools/msvc/vcregress.pl contribcheck + stop_test_script: + - tmp_install\bin\pg_ctl.exe stop -D tmp_check\db -l tmp_check\postmaster.log + ssl_test_script: + - set with_ssl=openssl + - perl src/tools/msvc/vcregress.pl taptest .\src\test\ssl\ + subscriptioncheck_test_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\subscription\ + authentication_test_script: + - perl src/tools/msvc/vcregress.pl taptest .\src\test\authentication\ + recoverycheck_test_script: + - perl src/tools/msvc/vcregress.pl recoverycheck + bincheck_test_script: + - perl src/tools/msvc/vcregress.pl bincheck + upgradecheck_test_script: + - perl src/tools/msvc/vcregress.pl upgradecheck + ecpgcheck_test_script: + # tries to build additional stuff + - vcvarsall x64 + # References ecpg_regression.proj in the current dir + - cd src\tools\msvc + - perl vcregress.pl ecpgcheck + + always: + cores_script: + - cat crashlog.txt || true + dump_artifacts: + path: "crashlog.txt" + type: text/plain + + on_failure: + log_artifacts: + path: "**/**.log" + type: text/plain + regress_diffs_artifacts: + path: "**/**.diffs" + type: text/plain + tap_artifacts: + path: "**/regress_log_*" + type: text/plain + + +task: + name: CompilerWarnings + depends_on: + - Linux + # task that did not run count as a success, so we need to recheck Linux' condition here :/ + only_if: $CIRRUS_CHANGE_MESSAGE !=~ '.*\nci-os-only:.*' || $CIRRUS_CHANGE_MESSAGE =~ '.*\nci-os-only:[^\n]*linux.*' + container: + dockerfile: ci/docker/linux_debian_bullseye + env: + CCACHE_SIZE: "4GB" + CCACHE_DIR: "/tmp/ccache_dir" + ccache_cache: + folder: "/tmp/ccache_dir" + setup_script: + - echo "COPT=-Werror" > src/Makefile.custom + - gcc -v + - clang -v + # gcc with asserts disabled + always: + gcc_warning_script: + - ./configure --cache gcc.cache CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # gcc with asserts enabled + always: + gcc_a_warning_script: + - ./configure --cache gcc.cache --enable-cassert CC="ccache gcc" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts disabled + always: + clang_warning_script: + - ./configure --cache clang.cache CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 + # clang with asserts enabled + always: + clang_a_warning_script: + - ./configure --cache clang.cache --enable-cassert CC="ccache clang" + - time make -s -j4 clean && time make -s -j4 diff --git a/.dockerignore b/.dockerignore new file mode 100644 index 00000000000..3fceab2e97b --- /dev/null +++ b/.dockerignore @@ -0,0 +1,3 @@ +# Ignore everything, except ci/ +* +!ci/* diff --git a/ci/docker/linux_debian_bullseye b/ci/docker/linux_debian_bullseye new file mode 100644 index 00000000000..f6c1782f16b --- /dev/null +++ b/ci/docker/linux_debian_bullseye @@ -0,0 +1,13 @@ +FROM debian:bullseye +RUN \ + apt-get -y update && \ + apt-get -y upgrade && \ + DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + git build-essential gcc g++ libreadline-dev flex bison make perl libipc-run-perl \ + libio-pty-perl clang llvm-dev libperl-dev libpython3-dev tcl-dev libldap2-dev \ + libicu-dev docbook-xml docbook-xsl fop libxml2-utils xsltproc krb5-admin-server \ + krb5-kdc krb5-user slapd ldap-utils libssl-dev pkg-config locales-all liblz4-dev \ + libsystemd-dev libxml2-dev libxslt1-dev python3-dev libkrb5-dev libpam-dev \ + libkrb5-*-heimdal uuid-dev gettext \ + liburing-dev python3-distutils ccache && \ + apt-get clean diff --git a/ci/docker/windows_vs_2019 b/ci/docker/windows_vs_2019 new file mode 100644 index 00000000000..a4fcaceae96 --- /dev/null +++ b/ci/docker/windows_vs_2019 @@ -0,0 +1,111 @@ +# escape=` + +# We used to use the visual studio container, but it's too outdated now +FROM cirrusci/windowsservercore:2019 + +SHELL ["powershell", "-NoLogo", "-NoProfile", "-Command"] + + +# Install commandline debugger and log all crashes to c:\cirrus\crashlog.txt +# +# Done manually as doing this via chocolatey / the installer directly, ends up +# with a lot of unnecessary chaff, making the layer unnecessarily large. +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + setx PATH \"C:\Windows Kits\10\Debuggers\x64;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o 'windsdksetup.exe' https://download.microsoft.com/download/9/7/9/97982c1d-d687-41be-9dd3-6d01e52ceb68/windowssdk/winsdk... ; ` + echo 'starting sdk installation (for debugger)' ; ` + Start-Process -Wait -FilePath ".\windsdksetup.exe" ` + -ArgumentList '/Features OptionId.WindowsDesktopDebuggers /layout c:\t\sdk /quiet /norestart /log c:\t\sdk.log' ` + ; ` + ` + Start-Process -Wait -FilePath msiexec.exe ` + -ArgumentList '/a \"C:\t\sdk\Installers\X64 Debuggers And Tools-x64_en-us.msi\" /qb /log install2.log' ` + ; ` + C:\Windows` Kits\10\Debuggers\x64\cdb.exe -version ; ` + ` + Set-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Debugger' -Value '\"C:\Windows Kits\10\Debuggers\x64\cdb.exe\" -p %ld -e %ld -g -kqm -c \".lines -e; .symfix+ ;.logappend c:\cirrus\crashlog.txt ; !peb; ~*kP ; .logclose ; q \"' ; ` + New-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name 'Auto' -Value 1 -PropertyType DWord ; ` + Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug' -Name Debugger ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install perl, python, flex and bison. +# +# Done manually as choco takes a lot longer. I think it's download issues with +# powershell's download stuff? That's wy curl.exe is directly used here at least... +# +# Using perl 5.26.3.1 for now, as newer versions don't currently work correctly +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + echo 'adding to path, before setup below, so changes are not overwritten' ; ` + setx PATH \"C:\strawberry\perl\bin;C:\winflexbison;C:\Program Files\Git\usr\bin;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o perl.zip ` + https://strawberryperl.com/download/5.26.3.1/strawberry-perl-5.26.3.1-64bit-portable.zip ; ` + echo 'installing perl' ; ` + 7z.exe x .\perl.zip -xr!c -oc:\strawberry ; ` + ` + curl.exe -sSL -o python.exe https://www.python.org/ftp/python/3.10.0/python-3.10.0-amd64.exe ; ` + echo 'installing python' ; ` + Start-Process -Wait -FilePath ".\python.exe" ` + -ArgumentList ` + '/quiet', 'SimpleInstall=1', 'PrependPath=1', 'CompileAll=1', ` + 'TargetDir=c:\python\', 'InstallAllUsers=1', 'Shortcuts=0', ` + 'Include_docs=0', 'Include_tcltk=0', 'Include_tests=0' ` + ; ` + ` + curl.exe -sSL -o winflexbison.zip ` + https://github.com/lexxmark/winflexbison/releases/download/v2.5.24/win_flex_bison-2.5.24.zip ; ` + echo 'installing winflexbison' ; ` + 7z.exe x .\winflexbison.zip -oc:\winflexbison ; ` + Rename-Item -Path c:\winflexbison\win_flex.exe c:\winflexbison\flex.exe ; ` + Rename-Item -Path c:\winflexbison\win_bison.exe c:\winflexbison\bison.exe ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install openssl +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + ` + curl.exe -o openssl-setup.exe -sSL https://slproweb.com/download/Win64OpenSSL-1_1_1L.exe ; ` + echo 'staring openssl installation' ; ` + Start-Process -Wait -FilePath ".\openssl-setup.exe" ` + -ArgumentList '/DIR=c:\openssl\1.1.1l\ /VERYSILENT /SP- /SUPPRESSMSGBOXES' ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse + + +# Install visual studio +# +# Adding VS path to vcvarsall.bat so user of container doesn't need to know the full path +RUN ` + mkdir c:\t ; ` + cd c:\t ; ` + setx PATH \"c:\BuildTools\VC\Auxiliary\Build;$Env:PATH\" /m ; ` + ` + curl.exe -sSL -o c:\t\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe ; ` + echo 'starting visual studio installation' ; ` + Start-Process -Wait ` + -FilePath c:\t\vs_buildtools.exe ` + -ArgumentList ` + '--quiet', '--wait', '--norestart', '--nocache', ` + '--installPath', 'c:\BuildTools', ` + '--add', 'Microsoft.VisualStudio.Component.VC.Tools.x86.x64', ` + '--add', 'Microsoft.VisualStudio.Component.Windows10SDK.20348' ; ` + ` + cd c:\ ; ` + Remove-Item C:\t -Force -Recurse ; ` + Remove-Item -Force -Recurse ${Env:TEMP}\*; ` + Remove-Item -Force -Recurse \"${Env:ProgramData}\Package Cache\" diff --git a/ci/freebsd_gcp_repartition.sh b/ci/freebsd_gcp_repartition.sh new file mode 100755 index 00000000000..2d5e1738998 --- /dev/null +++ b/ci/freebsd_gcp_repartition.sh @@ -0,0 +1,28 @@ +#!/bin/sh + +set -e +set -x + +# The default filesystem on freebsd gcp images is very slow to run tests on, +# due to its 32KB block size +# +# XXX: It'd probably better to fix this in the image, using something like +# https://people.freebsd.org/~lidl/blog/re-root.html + +# fix backup partition table after resize +gpart recover da0 +gpart show da0 +# kill swap, so we can delete a partition +swapoff -a || true +# (apparently we can only have 4!?) +gpart delete -i 3 da0 +gpart add -t freebsd-ufs -l data8k -a 4096 da0 +gpart show da0 +newfs -U -b 8192 /dev/da0p3 + +# Migrate working directory +du -hs $CIRRUS_WORKING_DIR +mv $CIRRUS_WORKING_DIR $CIRRUS_WORKING_DIR.orig +mkdir $CIRRUS_WORKING_DIR +mount -o noatime /dev/da0p3 $CIRRUS_WORKING_DIR +cp -r $CIRRUS_WORKING_DIR.orig/* $CIRRUS_WORKING_DIR/ diff --git a/ci/pg_ci_base.conf b/ci/pg_ci_base.conf new file mode 100644 index 00000000000..637e3cfb343 --- /dev/null +++ b/ci/pg_ci_base.conf @@ -0,0 +1,12 @@ +# Tends to produce too many core files, taking a long time +restart_after_crash = false + +# So that tests using the "manually" started postgres on windows can use +# prepared statements +max_prepared_transactions=10 + +# Settings that make logs more useful +log_line_prefix='%m [%p][%b][%v:%x] ' +log_checkpoints = true +log_connections = true +log_disconnections = true diff --git a/ci/windows_build_config.pl b/ci/windows_build_config.pl new file mode 100644 index 00000000000..ba82b13d69a --- /dev/null +++ b/ci/windows_build_config.pl @@ -0,0 +1,13 @@ +use strict; +use warnings; + +our $config; + +$config->{"tap_tests"} = 1; +$config->{"asserts"} = 1; + +$config->{"openssl"} = "c:/openssl/1.1.1l/"; +$config->{"perl"} = "c:/strawberry/perl/"; +$config->{"python"} = "c:/python/"; + +1; -- 2.23.0.385.gbc12974a89 --q7iu5jwpuq7sbl3z Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="v5-0003-plpython-Drop-support-python2.patch" ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Considering additional sort specialisation functions for PG16 @ 2023-01-26 10:29 John Naylor <[email protected]> 0 siblings, 2 replies; 12+ messages in thread From: John Naylor @ 2023-01-26 10:29 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: PostgreSQL Developers <[email protected]>; Thomas Munro <[email protected]> On Tue, Aug 23, 2022 at 1:13 PM John Naylor <[email protected]> wrote: > > On Tue, Aug 23, 2022 at 11:24 AM David Rowley <[email protected]> wrote: > > > > On Tue, 23 Aug 2022 at 15:22, John Naylor <[email protected]> wrote: > > > Did you happen to see > > > > > > https://www.postgresql.org/message-id/CAFBsxsFhq8VUSkUL5YO17cFXbCPwtbbxBu%2Bd9MFrrsssfDXm3Q%40mail.g... > > > > I missed that. It looks like a much more promising idea than what I > > came up with. I've not looked at your code yet, but I'm interested and > > will aim to look soon. > > Note that I haven't actually implemented this idea yet, just tried to > model the effects by lobotomizing the current comparators. I think > it's worth pursuing and will try to come back to it this cycle, but if > you or anyone else wants to try, that's fine of course. Coming back to this, I wanted to sketch out this idea in a bit more detail. Have two memtuple arrays, one for first sortkey null and one for first sortkey non-null: - Qsort the non-null array, including whatever specialization is available. Existing specialized comparators could ignore nulls (and their ordering) taking less space in the binary. - Only if there is more than one sort key, qsort the null array. Ideally at some point we would have a method of ignoring the first sortkey (this is an existing opportunity that applies elsewhere as well). - To handle two arrays, grow_memtuples() would need some adjustment, as would any callers that read the final result of an in-memory sort -- they would need to retrieve the tuples starting with the appropriate array depending on NULLS FIRST/LAST behavior. I believe external merges wouldn't have to do anything different, since when writing out the tapes, we read from the arrays in the right order. (One could extend this idea further and have two pools of tapes for null and non-null first sortkey, that are merged separately, in the right order. That sounds like quite a bit more complexity than is worth, however.) -- John Naylor EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Considering additional sort specialisation functions for PG16 @ 2023-01-26 11:13 Pavel Borisov <[email protected]> parent: John Naylor <[email protected]> 1 sibling, 1 reply; 12+ messages in thread From: Pavel Borisov @ 2023-01-26 11:13 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: David Rowley <[email protected]>; PostgreSQL Developers <[email protected]>; Thomas Munro <[email protected]> Hi, John! Generally, I like the separation of non-null values before sorting and would like to join as a reviewer when we come to patch. I have only a small question: > - Only if there is more than one sort key, qsort the null array. Ideally at some point we would have a method of ignoring the first sortkey (this is an existing opportunity that applies elsewhere as well). Should we need to sort by the second sort key provided the first one in NULL by standard or by some part of the code relying on this? I suppose NULL values in the first sort key mean attribute values are undefined and there is no preferred order between these tuples, even if their second sort keys are different. And maybe (unlikely IMO) we need some analog of NULLS DISCTICNT/NOT DISTINCT in this scope? Kind regards, Pavel Borisov, Supabase. ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Considering additional sort specialisation functions for PG16 @ 2023-01-26 12:06 John Naylor <[email protected]> parent: Pavel Borisov <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: John Naylor @ 2023-01-26 12:06 UTC (permalink / raw) To: Pavel Borisov <[email protected]>; +Cc: David Rowley <[email protected]>; PostgreSQL Developers <[email protected]>; Thomas Munro <[email protected]> On Thu, Jan 26, 2023 at 6:14 PM Pavel Borisov <[email protected]> wrote: > > - Only if there is more than one sort key, qsort the null array. Ideally at some point we would have a method of ignoring the first sortkey (this is an existing opportunity that applies elsewhere as well). > Should we need to sort by the second sort key provided the first one > in NULL by standard or by some part of the code relying on this? I I'm not sure I quite understand the question. If there is more than one sort key, and the specialized comparison on the first key gives a definitive zero result, it falls back to comparing all keys from the full tuple. (The sorttuple struct only contains the first sortkey, which might actually be an abbreviated key.) A possible optimization, relevant here and also elsewhere, is to compare only using keys starting from key2. But note: if the first key is abbreviated, a zero result is not definitive, and we must check the first key's full value from the tuple. > suppose NULL values in the first sort key mean attribute values are > undefined and there is no preferred order between these tuples, even > if their second sort keys are different. There is in fact a preferred order between these tuples -- the second key is the tie breaker in this case. -- John Naylor EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Considering additional sort specialisation functions for PG16 @ 2023-01-26 12:14 David Rowley <[email protected]> parent: John Naylor <[email protected]> 1 sibling, 1 reply; 12+ messages in thread From: David Rowley @ 2023-01-26 12:14 UTC (permalink / raw) To: John Naylor <[email protected]>; +Cc: PostgreSQL Developers <[email protected]>; Thomas Munro <[email protected]> On Thu, 26 Jan 2023 at 23:29, John Naylor <[email protected]> wrote: > Coming back to this, I wanted to sketch out this idea in a bit more detail. > > Have two memtuple arrays, one for first sortkey null and one for first sortkey non-null: > - Qsort the non-null array, including whatever specialization is available. Existing specialized comparators could ignore nulls (and their ordering) taking less space in the binary. > - Only if there is more than one sort key, qsort the null array. Ideally at some point we would have a method of ignoring the first sortkey (this is an existing opportunity that applies elsewhere as well). > - To handle two arrays, grow_memtuples() would need some adjustment, as would any callers that read the final result of an in-memory sort -- they would need to retrieve the tuples starting with the appropriate array depending on NULLS FIRST/LAST behavior. Thanks for coming back to this. I've been thinking about this again recently due to what was discovered in [1]. Basically, the patch on that thread is trying to eliminate the need for the ORDER BY sort in a query such as: SELECT a,b,row_number() over (order by a) from ab order by a,b; the idea is to just perform the full sort on a,b for the WindowClause so save from having to do an Incremental Sort on b for the ORDER BY after evaluating the window funcs. Surprisingly (for me), we found a bunch of cases where the performance is better to do a sort on some of the keys, then do an Incremental sort on the remainder rather than just doing a single sort on everything. I don't really understand why this is fully yet, but one theory I have is that it might be down to work_mem being larger than the CPU's L3 cache and causing more cache line misses. With the more simple sort, there's less swapping of items in the array because the comparison function sees tuples as equal more often. I did find that the gap between the two is not as large with fewer tuples to sort. I think the slower sorts I found in [2] could also be partially caused by the current sort specialisation comparators re-comparing the leading column during a tie-break. I've not gotten around to disabling the sort specialisations to see if and how much this is a factor for that test. Why I'm bringing this up here is that I wondered, in addition to what you're mentioning above, if we're making some changes to allow multiple in-memory arrays, would it be worth going a little further and allowing it so we could have N arrays which we'd try to keep under L3 cache size. Maybe some new GUC could be used to know what a good value is. With fast enough disks, it's often faster to use smaller values of work_mem which don't exceed L3 to keep these batches smaller. Keeping them in memory would remove the need to write out tapes. I don't really know exactly if such a feature could easily be tagged on to what you propose above, but I feel like what you're talking about is part of the way towards that at least, maybe at the least, the additional work could be kept in mind when this is written so that it's easier to extend in the future. David [1] https://postgr.es/m/CAApHDvpAO5H_L84kn9gCJ_hihOavtmDjimKYyftjWtF69BJ=8Q@mail.gmail.com [2] https://postgr.es/m/CAApHDvqh%2BqOHk4sbvvy%3DQr2NjPqAAVYf82oXY0g%3DZ2hRpC2Vmg%40mail.gmail.com ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Considering additional sort specialisation functions for PG16 @ 2023-01-27 06:56 John Naylor <[email protected]> parent: David Rowley <[email protected]> 0 siblings, 1 reply; 12+ messages in thread From: John Naylor @ 2023-01-27 06:56 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: PostgreSQL Developers <[email protected]>; Thomas Munro <[email protected]> On Thu, Jan 26, 2023 at 7:15 PM David Rowley <[email protected]> wrote: > > On Thu, 26 Jan 2023 at 23:29, John Naylor <[email protected]> wrote: > > Coming back to this, I wanted to sketch out this idea in a bit more detail. > > > > Have two memtuple arrays, one for first sortkey null and one for first sortkey non-null: > > - Qsort the non-null array, including whatever specialization is available. Existing specialized comparators could ignore nulls (and their ordering) taking less space in the binary. > > - Only if there is more than one sort key, qsort the null array. Ideally at some point we would have a method of ignoring the first sortkey (this is an existing opportunity that applies elsewhere as well). > > - To handle two arrays, grow_memtuples() would need some adjustment, as would any callers that read the final result of an in-memory sort -- they would need to retrieve the tuples starting with the appropriate array depending on NULLS FIRST/LAST behavior. > > Thanks for coming back to this. I've been thinking about this again > recently due to what was discovered in [1]. That was indeed part of the motivation for bringing this up. > Why I'm bringing this up here is that I wondered, in addition to what > you're mentioning above, if we're making some changes to allow > multiple in-memory arrays, would it be worth going a little further > and allowing it so we could have N arrays which we'd try to keep under > L3 cache size. Maybe some new GUC could be used to know what a good > value is. With fast enough disks, it's often faster to use smaller > values of work_mem which don't exceed L3 to keep these batches > smaller. Keeping them in memory would remove the need to write out > tapes. That's interesting. I don't know enough to guess how complex it would be to make "external" merges agnostic about whether the tapes are on disk or in memory. If in-memory sorts were designed analogously to external ones, grow_memtuples would never have to repalloc, it could just allocate a new tape, which could further shave some cycles. My hunch in my last email was that having separate groups of tapes for each null/non-null first sortkey would be complex, because it increases the number of places that have to know about nulls and their ordering. If we wanted to go to N arrays instead of 2, and additionally wanted separate null/non-null treatment, it seems we would still need 2 sets of arrays, one with N non-null-first-sortkey tapes and one with M null-first-sortkey tapes. So using 2 arrays seems like the logical first step. I'd be curious to hear other possible development paths. > I think the slower sorts I found in [2] could also be partially caused > by the current sort specialisation comparators re-comparing the > leading column during a tie-break. I've not gotten around to disabling > the sort specialisations to see if and how much this is a factor for > that test. Right, that's worth addressing independently of the window function consideration. I'm still swapping this area back in my head, but I believe one issue is that state->base.onlyKey signals two things: "one sortkey, not abbreviated". We could add a separate branch for "first key unabbreviated, nkeys>1" -- I don't think we'd need to specialize, just branch -- and instead of state->base.comparetup, call a set of analogous functions that only handle keys 2 and above (comparetup_tail_* ? or possibly just add a boolean parameter compare_first). That would not pose a huge challenge, I think, since they're already written like this: /* Compare the leading sort key */ compare = ApplySortComparator(...); if (compare != 0) return compare; /* Compare additional sort keys */ ... The null/non-null separation would eliminate a bunch of branches in inlined comparators, so we could afford to add another branch for number of keys. I haven't thought through either of these ideas in the gory detail, but I don't yet see any big obstacles. -- John Naylor EDB: http://www.enterprisedb.com ^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Considering additional sort specialisation functions for PG16 @ 2023-01-30 10:32 John Naylor <[email protected]> parent: John Naylor <[email protected]> 0 siblings, 0 replies; 12+ messages in thread From: John Naylor @ 2023-01-30 10:32 UTC (permalink / raw) To: David Rowley <[email protected]>; +Cc: PostgreSQL Developers <[email protected]>; Thomas Munro <[email protected]> I wrote: > On Thu, Jan 26, 2023 at 7:15 PM David Rowley <[email protected]> wrote: > > I think the slower sorts I found in [2] could also be partially caused > > by the current sort specialisation comparators re-comparing the > > leading column during a tie-break. I've not gotten around to disabling > > the sort specialisations to see if and how much this is a factor for > > that test. > > Right, that's worth addressing independently of the window function consideration. I'm still swapping this area back in my head, but I believe one issue is that state->base.onlyKey signals two things: "one sortkey, not abbreviated". We could add a separate branch for "first key unabbreviated, nkeys>1" -- I don't think we'd need to specialize, just branch -- and instead of state->base.comparetup, call a set of analogous functions that only handle keys 2 and above (comparetup_tail_* ? or possibly just add a boolean parameter compare_first). That would not pose a huge challenge, I think, since they're already written like this: > > /* Compare the leading sort key */ > compare = ApplySortComparator(...); > if (compare != 0) > return compare; > > /* Compare additional sort keys */ > ... > > The null/non-null separation would eliminate a bunch of branches in inlined comparators, so we could afford to add another branch for number of keys. I gave this a go, and it turns out we don't need any extra branches in the inlined comparators -- the new fallbacks are naturally written to account for the "!onlyKey" case. If the first sortkey was abbreviated, call its full comparator, otherwise skip to the next sortkey (if there isn't one, we shouldn't have gotten here). The existing comparetup functions try the simple case and then call the fallback (which could be inlined for them but I haven't looked). Tests pass, but I'm not sure yet if we need more tests. I don't have a purpose-built benchmark at the moment, but I'll see if any of my existing tests exercise this code path. I can also try the window function case unless someone beats me to it. -- John Naylor EDB: http://www.enterprisedb.com Attachments: [text/x-patch] v1-0001-Split-out-fallback-functionality-from-comparetup-.patch (11.4K, ../../CAFBsxsFdFpzyBekxxkiA4vXnLpw-wcaQXz=EAP4pzkZMo91-MA@mail.gmail.com/3-v1-0001-Split-out-fallback-functionality-from-comparetup-.patch) download | inline diff: From 65b94223f8bdb726b97d695f0040c12197d5e65d Mon Sep 17 00:00:00 2001 From: John Naylor <[email protected]> Date: Mon, 30 Jan 2023 17:10:00 +0700 Subject: [PATCH v1] Split out fallback functionality from comparetup* functions Previously, if a specialized comparator find equal datum1 keys, the comparetup function would call the full comparator on the datum before proceeding with either the unabbreviated first key or the second key. Add a comparetup_fallback field for these that call special fallback functions. The existing comparetup functions just call ApplySortComparator where we can, than call our new fallback. --- src/backend/utils/sort/tuplesort.c | 6 +- src/backend/utils/sort/tuplesortvariants.c | 114 ++++++++++++++++----- src/include/utils/tuplesort.h | 6 ++ 3 files changed, 96 insertions(+), 30 deletions(-) diff --git a/src/backend/utils/sort/tuplesort.c b/src/backend/utils/sort/tuplesort.c index 9ca9835aab..54f64d6c2d 100644 --- a/src/backend/utils/sort/tuplesort.c +++ b/src/backend/utils/sort/tuplesort.c @@ -513,7 +513,7 @@ qsort_tuple_unsigned_compare(SortTuple *a, SortTuple *b, Tuplesortstate *state) if (state->base.onlyKey != NULL) return 0; - return state->base.comparetup(a, b, state); + return state->base.comparetup_fallback(a, b, state); } #if SIZEOF_DATUM >= 8 @@ -537,7 +537,7 @@ qsort_tuple_signed_compare(SortTuple *a, SortTuple *b, Tuplesortstate *state) if (state->base.onlyKey != NULL) return 0; - return state->base.comparetup(a, b, state); + return state->base.comparetup_fallback(a, b, state); } #endif @@ -561,7 +561,7 @@ qsort_tuple_int32_compare(SortTuple *a, SortTuple *b, Tuplesortstate *state) if (state->base.onlyKey != NULL) return 0; - return state->base.comparetup(a, b, state); + return state->base.comparetup_fallback(a, b, state); } /* diff --git a/src/backend/utils/sort/tuplesortvariants.c b/src/backend/utils/sort/tuplesortvariants.c index eb6cfcfd00..e65ac49d9d 100644 --- a/src/backend/utils/sort/tuplesortvariants.c +++ b/src/backend/utils/sort/tuplesortvariants.c @@ -47,18 +47,24 @@ static void removeabbrev_datum(Tuplesortstate *state, SortTuple *stups, int count); static int comparetup_heap(const SortTuple *a, const SortTuple *b, Tuplesortstate *state); +static int comparetup_heap_fallback(const SortTuple *a, const SortTuple *b, + Tuplesortstate *state); static void writetup_heap(Tuplesortstate *state, LogicalTape *tape, SortTuple *stup); static void readtup_heap(Tuplesortstate *state, SortTuple *stup, LogicalTape *tape, unsigned int len); static int comparetup_cluster(const SortTuple *a, const SortTuple *b, Tuplesortstate *state); +static int comparetup_cluster_fallback(const SortTuple *a, const SortTuple *b, + Tuplesortstate *state); static void writetup_cluster(Tuplesortstate *state, LogicalTape *tape, SortTuple *stup); static void readtup_cluster(Tuplesortstate *state, SortTuple *stup, LogicalTape *tape, unsigned int tuplen); static int comparetup_index_btree(const SortTuple *a, const SortTuple *b, Tuplesortstate *state); +static int comparetup_index_btree_fallback(const SortTuple *a, const SortTuple *b, + Tuplesortstate *state); static int comparetup_index_hash(const SortTuple *a, const SortTuple *b, Tuplesortstate *state); static void writetup_index(Tuplesortstate *state, LogicalTape *tape, @@ -67,6 +73,8 @@ static void readtup_index(Tuplesortstate *state, SortTuple *stup, LogicalTape *tape, unsigned int len); static int comparetup_datum(const SortTuple *a, const SortTuple *b, Tuplesortstate *state); +static int comparetup_datum_fallback(const SortTuple *a, const SortTuple *b, + Tuplesortstate *state); static void writetup_datum(Tuplesortstate *state, LogicalTape *tape, SortTuple *stup); static void readtup_datum(Tuplesortstate *state, SortTuple *stup, @@ -165,6 +173,7 @@ tuplesort_begin_heap(TupleDesc tupDesc, base->removeabbrev = removeabbrev_heap; base->comparetup = comparetup_heap; + base->comparetup_fallback = comparetup_heap_fallback; base->writetup = writetup_heap; base->readtup = readtup_heap; base->haveDatum1 = true; @@ -242,6 +251,7 @@ tuplesort_begin_cluster(TupleDesc tupDesc, base->removeabbrev = removeabbrev_cluster; base->comparetup = comparetup_cluster; + base->comparetup_fallback = comparetup_cluster_fallback; base->writetup = writetup_cluster; base->readtup = readtup_cluster; base->freestate = freestate_cluster; @@ -351,6 +361,7 @@ tuplesort_begin_index_btree(Relation heapRel, base->removeabbrev = removeabbrev_index; base->comparetup = comparetup_index_btree; + base->comparetup_fallback = comparetup_index_btree_fallback; base->writetup = writetup_index; base->readtup = readtup_index; base->haveDatum1 = true; @@ -431,6 +442,7 @@ tuplesort_begin_index_hash(Relation heapRel, base->removeabbrev = removeabbrev_index; base->comparetup = comparetup_index_hash; + base->comparetup_fallback = NULL; base->writetup = writetup_index; base->readtup = readtup_index; base->haveDatum1 = true; @@ -476,6 +488,7 @@ tuplesort_begin_index_gist(Relation heapRel, base->removeabbrev = removeabbrev_index; base->comparetup = comparetup_index_btree; + base->comparetup_fallback = comparetup_index_btree_fallback; base->writetup = writetup_index; base->readtup = readtup_index; base->haveDatum1 = true; @@ -546,6 +559,7 @@ tuplesort_begin_datum(Oid datumType, Oid sortOperator, Oid sortCollation, base->removeabbrev = removeabbrev_datum; base->comparetup = comparetup_datum; + base->comparetup_fallback = comparetup_datum_fallback; base->writetup = writetup_datum; base->readtup = readtup_datum; base->haveDatum1 = true; @@ -931,16 +945,7 @@ comparetup_heap(const SortTuple *a, const SortTuple *b, Tuplesortstate *state) { TuplesortPublic *base = TuplesortstateGetPublic(state); SortSupport sortKey = base->sortKeys; - HeapTupleData ltup; - HeapTupleData rtup; - TupleDesc tupDesc; - int nkey; int32 compare; - AttrNumber attno; - Datum datum1, - datum2; - bool isnull1, - isnull2; /* Compare the leading sort key */ @@ -951,6 +956,26 @@ comparetup_heap(const SortTuple *a, const SortTuple *b, Tuplesortstate *state) return compare; /* Compare additional sort keys */ + return comparetup_heap_fallback(a, b, state); +} + +/* Compare the first sortkey only if it was abbreviated, then compare the rest. */ +static int +comparetup_heap_fallback(const SortTuple *a, const SortTuple *b, Tuplesortstate *state) +{ + TuplesortPublic *base = TuplesortstateGetPublic(state); + SortSupport sortKey = base->sortKeys; + HeapTupleData ltup; + HeapTupleData rtup; + TupleDesc tupDesc; + int nkey; + int32 compare; + AttrNumber attno; + Datum datum1, + datum2; + bool isnull1, + isnull2; + ltup.t_len = ((MinimalTuple) a->tuple)->t_len + MINIMAL_TUPLE_OFFSET; ltup.t_data = (HeapTupleHeader) ((char *) a->tuple - MINIMAL_TUPLE_OFFSET); rtup.t_len = ((MinimalTuple) b->tuple)->t_len + MINIMAL_TUPLE_OFFSET; @@ -1061,6 +1086,27 @@ removeabbrev_cluster(Tuplesortstate *state, SortTuple *stups, int count) static int comparetup_cluster(const SortTuple *a, const SortTuple *b, Tuplesortstate *state) +{ + TuplesortPublic *base = TuplesortstateGetPublic(state); + SortSupport sortKey = base->sortKeys; + int32 compare; + + /* Compare the leading sort key, if it's simple */ + if (base->haveDatum1) + { + compare = ApplySortComparator(a->datum1, a->isnull1, + b->datum1, b->isnull1, + sortKey); + if (compare != 0) + return compare; + } + + return comparetup_cluster_fallback(a, b, state); +} + +static int +comparetup_cluster_fallback(const SortTuple *a, const SortTuple *b, + Tuplesortstate *state) { TuplesortPublic *base = TuplesortstateGetPublic(state); TuplesortClusterArg *arg = (TuplesortClusterArg *) base->arg; @@ -1075,7 +1121,6 @@ comparetup_cluster(const SortTuple *a, const SortTuple *b, bool isnull1, isnull2; - /* Be prepared to compare additional sort keys */ ltup = (HeapTuple) a->tuple; rtup = (HeapTuple) b->tuple; tupDesc = arg->tupDesc; @@ -1083,12 +1128,6 @@ comparetup_cluster(const SortTuple *a, const SortTuple *b, /* Compare the leading sort key, if it's simple */ if (base->haveDatum1) { - compare = ApplySortComparator(a->datum1, a->isnull1, - b->datum1, b->isnull1, - sortKey); - if (compare != 0) - return compare; - if (sortKey->abbrev_converter) { AttrNumber leading = arg->indexInfo->ii_IndexAttrNumbers[0]; @@ -1100,6 +1139,9 @@ comparetup_cluster(const SortTuple *a, const SortTuple *b, datum2, isnull2, sortKey); } + else + compare = 0; + if (compare != 0 || base->nKeys == 1) return compare; /* Compare additional columns the hard way */ @@ -1269,6 +1311,25 @@ comparetup_index_btree(const SortTuple *a, const SortTuple *b, * treatment for equal keys at the end. */ TuplesortPublic *base = TuplesortstateGetPublic(state); + SortSupport sortKey = base->sortKeys; + int32 compare; + + /* Compare the leading sort key */ + compare = ApplySortComparator(a->datum1, a->isnull1, + b->datum1, b->isnull1, + sortKey); + if (compare != 0) + return compare; + + /* Compare additional sort keys */ + return comparetup_index_btree_fallback(a, b, state); +} + +static int +comparetup_index_btree_fallback(const SortTuple *a, const SortTuple *b, + Tuplesortstate *state) +{ + TuplesortPublic *base = TuplesortstateGetPublic(state); TuplesortIndexBTreeArg *arg = (TuplesortIndexBTreeArg *) base->arg; SortSupport sortKey = base->sortKeys; IndexTuple tuple1; @@ -1283,15 +1344,6 @@ comparetup_index_btree(const SortTuple *a, const SortTuple *b, bool isnull1, isnull2; - - /* Compare the leading sort key */ - compare = ApplySortComparator(a->datum1, a->isnull1, - b->datum1, b->isnull1, - sortKey); - if (compare != 0) - return compare; - - /* Compare additional sort keys */ tuple1 = (IndexTuple) a->tuple; tuple2 = (IndexTuple) b->tuple; keysz = base->nKeys; @@ -1527,13 +1579,21 @@ comparetup_datum(const SortTuple *a, const SortTuple *b, Tuplesortstate *state) return compare; /* if we have abbreviations, then "tuple" has the original value */ + return comparetup_datum_fallback(a, b, state); +} + +static int +comparetup_datum_fallback(const SortTuple *a, const SortTuple *b, Tuplesortstate *state) +{ + TuplesortPublic *base = TuplesortstateGetPublic(state); if (base->sortKeys->abbrev_converter) - compare = ApplySortAbbrevFullComparator(PointerGetDatum(a->tuple), a->isnull1, + return ApplySortAbbrevFullComparator(PointerGetDatum(a->tuple), a->isnull1, PointerGetDatum(b->tuple), b->isnull1, base->sortKeys); - return compare; + /* shouldn't get here */ + return 0; } static void diff --git a/src/include/utils/tuplesort.h b/src/include/utils/tuplesort.h index 12578e42bc..86fa57c870 100644 --- a/src/include/utils/tuplesort.h +++ b/src/include/utils/tuplesort.h @@ -162,6 +162,12 @@ typedef struct */ SortTupleComparator comparetup; + /* + * Only compares the first sortkey if it was abbreviated and we need a + * tiebreaker. Otherwise, only compare second and later sortkeys. + */ + SortTupleComparator comparetup_fallback; + /* * Alter datum1 representation in the SortTuple's array back from the * abbreviated key to the first column value. -- 2.39.0 ^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2023-01-30 10:32 UTC | newest] Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2021-03-15 16:25 [PATCH v3 02/17] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. Andres Freund <[email protected]> 2021-03-15 16:25 [PATCH v2 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. Andres Freund <[email protected]> 2021-03-15 16:25 [PATCH v3 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. Andres Freund <[email protected]> 2021-03-15 16:25 [PATCH v1 2/6] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. Andres Freund <[email protected]> 2021-03-15 16:25 [PATCH v5 02/16] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. Andres Freund <[email protected]> 2021-03-15 16:25 [PATCH v4 1/2] ci: Add CI for FreeBSD, Linux, MacOS and Windows, utilizing cirrus-ci. Andres Freund <[email protected]> 2023-01-26 10:29 Re: Considering additional sort specialisation functions for PG16 John Naylor <[email protected]> 2023-01-26 11:13 ` Re: Considering additional sort specialisation functions for PG16 Pavel Borisov <[email protected]> 2023-01-26 12:06 ` Re: Considering additional sort specialisation functions for PG16 John Naylor <[email protected]> 2023-01-26 12:14 ` Re: Considering additional sort specialisation functions for PG16 David Rowley <[email protected]> 2023-01-27 06:56 ` Re: Considering additional sort specialisation functions for PG16 John Naylor <[email protected]> 2023-01-30 10:32 ` Re: Considering additional sort specialisation functions for PG16 John Naylor <[email protected]>
This inbox is served by agora; see mirroring instructions for how to clone and mirror all data and code used for this inbox