pgjdbc/pgjdbc GitHub issues and pull requests (mirror)
help / color / mirror / Atom feed[pgjdbc/pgjdbc] PR #3257: test: add CI testing against PostgreSQL HEAD
11+ messages / 3 participants
[nested] [flat]
* [pgjdbc/pgjdbc] PR #3257: test: add CI testing against PostgreSQL HEAD
@ 2024-05-21 19:39 "vlsi (@vlsi)" <[email protected]>
0 siblings, 0 replies; 11+ messages in thread
From: vlsi (@vlsi) @ 2024-05-21 19:39 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
## What
Adds PostgreSQL HEAD to the main and omni CI matrices as another `pg_version` value. When `matrix.pg_version == 'HEAD'`, the shared `docker/postgres-server` compose builds a devel image inline from apt.postgresql.org's `*-pgdg-snapshot` suite and runs the matrix row against it through the existing healthcheck and `docker compose up --wait` flow.
HEAD is added to the matrix only on branch builds (`refs/heads/*`); pull requests are deliberately not exercised against HEAD so PR validation is not blocked by breakage in unreleased PostgreSQL. `omni.yml` always includes HEAD because it triggers on schedule and workflow_dispatch.
Other pieces that come with this change:
- The Dockerfile auto-discovers the highest `postgresql-N` available in pgdg-snapshot at build time, so no `PG_MAJOR` bump is needed when a new major appears. Pass `--build-arg PG_MAJOR=18` to pin a specific major. Stable paths come from a `/usr/lib/postgresql/current` symlink; `PG_MAJOR` is exported by the shared entrypoint from `postgres --version` at runtime.
- The shared compose gains a TCP healthcheck (a bash `/dev/tcp` probe that works across versions, auth modes and SSL; `pg_isready` cannot serve as the probe because the Debian `pg_wrapper` fails to resolve it for servers older than 9.3). `main.yml` and `omni.yml` wait via `docker compose up --wait` instead of polling `pg_isready`.
- `pg_basebackup --checkpoint=fast` is added for faster replica creation.
## Why
The matrix previously had no first-class HEAD coverage; a narrow `test-pg-head` job in `omni.yml` only ran on the nightly/manual schedule and used a separate, narrow setup. Treating HEAD as just another `pg_version` brings it to every branch build through the regular matrix path, so breakage in unreleased PostgreSQL shows up early — without making PRs depend on a moving target.
The build path is fast enough to do per-job (~30 s on a clean runner), so no shared image needs publishing: nothing is pushed to a registry and no retention has to be managed. apt-pgdg packages bring the full released feature set (`--with-gssapi`, `--with-llvm`, `--with-icu`, `--with-lz4`, `--with-zstd`, `--with-perl`, `--with-python`, `--with-tcl`), so the image covers feature coverage that a hand-rolled source build would have to opt into.
## How to verify
- After this PR lands, branch builds on `pgjdbc/pgjdbc` show a `PG HEAD` row in the CI matrix that goes green.
- The HEAD row runs on `ubuntu-latest` (per `matrix.imply`).
- The HEAD row's `Start PostgreSQL` step builds the image (`docker compose up -d --wait`), reaches `Healthy`, and the gradle tests run against it.
- The CI run for this pull request itself does **not** include a HEAD row (HEAD is gated to `refs/heads/*`).
- No new workflows or registry pushes appear.
diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 3f7b20bb0f..eb3692405d 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -88,7 +88,7 @@ jobs:
fetch-depth: 50
- name: Start PostgreSQL
working-directory: docker/postgres-server
- run: docker compose up -d && docker compose logs
+ run: docker compose up -d --wait || { docker compose logs; exit 1; }
- name: 'Set up JDK 21'
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
@@ -162,6 +162,8 @@ jobs:
SSL: ${{ matrix.ssl }}
SCRAM: ${{ matrix.scram }}
CREATE_REPLICAS: ${{ matrix.replication }}
+ PG_IMAGE: ${{ matrix.pg_version == 'HEAD' && 'pgjdbc/postgres-devel:ci-head' || '' }}
+ PG_PULL_POLICY: ${{ matrix.pg_version == 'HEAD' && 'build' || 'missing' }}
# The below run command is long, however, it is intentional, and it makes the output nicer in GitHub UI
# language=bash
run: |
@@ -169,7 +171,7 @@ jobs:
docker compose down -v --rmi local || true
sed -i -r '/- (543[3-4]):\1/d' docker-compose.yml
- docker compose up -d
+ docker compose up -d --wait || { docker compose logs; exit 1; }
docker compose logs
- name: Start PostgreSQL PGV=${{ matrix.pg_version }}
if: ${{ runner.os != 'Linux' }}
diff --git a/.github/workflows/matrix.mjs b/.github/workflows/matrix.mjs
index c62824f748..c851ccb1b2 100644
--- a/.github/workflows/matrix.mjs
+++ b/.github/workflows/matrix.mjs
@@ -72,6 +72,11 @@ matrix.addAxis({
]
});
+// Test with PostgreSQL HEAD for branch-based builds only.
+if ((process.env.GITHUB_REF || '').startsWith('refs/heads/')) {
+ matrix.axisByName.pg_version.values.push('HEAD');
+}
+
matrix.addAxis({
name: 'tz',
title: x => 'client_tz ' + x,
@@ -287,6 +292,8 @@ matrix.exclude({java_distribution: {value: 'semeru'}, java_version: '21'})
matrix.imply({gss: {value: 'yes'}}, {os: {value: 'ubuntu-latest'}})
// ikalnytskyi/action-setup-postgres supports PostgreSQL 14+ only
matrix.exclude({os: {value: ['windows-latest', 'macos-latest']}, pg_version: lessThan('14')});
+// HEAD is built from pgdg-snapshot inside Docker, which only runs on Linux.
+matrix.imply({pg_version: 'HEAD'}, {os: {value: 'ubuntu-latest'}});
// cleanupSavepoints is not relevant when autosave=never
matrix.imply({autosave: {value: 'never'}}, {cleanupSavepoints: {value: 'false'}});
diff --git a/.github/workflows/omni.yml b/.github/workflows/omni.yml
index c75bdeac93..aaa803ffc6 100644
--- a/.github/workflows/omni.yml
+++ b/.github/workflows/omni.yml
@@ -123,7 +123,7 @@ jobs:
const test_group = opts.test_group ?? DEFAULT_TEST_GROUP;
const create_replicas = opts.create_replicas ?? false;
- const isAtLeast = (minVersion) => Number(pg_version) >= Number(minVersion);
+ const isAtLeast = (minVersion) => pg_version === 'HEAD' || Number(pg_version) >= Number(minVersion);
const scramSupported = isAtLeast('10');
const sslSupported = isAtLeast('9.3');
@@ -186,6 +186,11 @@ jobs:
});
}
+ // Latest JDK x PostgreSQL HEAD (built inline from pgdg-snapshot).
+ addItem({
+ pg_version: 'HEAD',
+ });
+
// Latest PG version x each remaining JDK
for (const jdk of JDK_OPTIONS) {
if (jdk == LATEST_JDK) {
@@ -301,9 +306,11 @@ jobs:
- name: Start PostgreSQL
working-directory: docker/postgres-server
- run: docker compose up -d && docker compose logs
+ run: docker compose up -d --wait || { docker compose logs; exit 1; }
env:
PGV: ${{ matrix.pg_version }}
+ PG_IMAGE: ${{ matrix.pg_version == 'HEAD' && 'pgjdbc/postgres-devel:ci-head' || '' }}
+ PG_PULL_POLICY: ${{ matrix.pg_version == 'HEAD' && 'build' || 'missing' }}
# Install built-in JDK
- name: 'Set up JDK ${{ matrix.jdk_version }} / ${{ matrix.jdk_distribution }}'
@@ -339,12 +346,6 @@ jobs:
PGDATABASE: postgres
PGHOST: localhost
run: |
- if ! docker/bin/wait_for_pg_isready; then
- # Server is not online so dump some logs for debugging
- docker ps
- cd docker/postgres-server
- docker compose logs
- fi
psql -c 'SELECT version()'
- name: Prepare local properties
@@ -374,54 +375,3 @@ jobs:
with:
file: ./build/reports/jacoco/jacocoReport/jacocoReport.xml
- test-pg-head:
- name: Zulu 17 x PG HEAD
- runs-on: ubuntu-latest
- if: github.event_name != 'schedule' || vars.ENABLE_SCHEDULED_JOBS == 'true'
- continue-on-error: true
- steps:
- - uses: actions/checkout@df4cb1c069e1874edd31b4311f1884172cec0e10 # v6.0.3
- with:
- fetch-depth: 50
- - name: Compile and start PostgreSQL
- working-directory: docker/postgres-head
- run: docker compose up -d && docker compose logs
- - name: Set up JDK
- uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
- with:
- distribution: zulu
- java-version: 17
- architecture: x64
- - name: Java version
- run: |
- java -version
- - name: PostgreSQL version
- env:
- PGUSER: postgres
- PGDATABASE: postgres
- PGHOST: localhost
- run: |
- if ! docker/bin/wait_for_pg_isready; then
- # Server is not online so dump some logs for debugging
- docker ps
- cd docker/postgres-head
- docker compose logs
- fi
- psql -c 'SELECT version()'
- - name: Prepare local properties
- run: |
- cat <<EOF >ssltest.local.properties
- enable_ssl_tests=false
- EOF
- cat <<EOF >build.local.properties
- EOF
- - name: Test
- uses: burrunan/gradle-cache-action@4b67497abd37a511d6c1dc6299bdd84ff39f7bf5 # v3.0.2
- with:
- arguments: --no-parallel --no-daemon --scan jandex test jacocoReport -PskipJavadoc
- properties: |
- includeTestTags=!org.postgresql.test.SlowTests & !replication
- - name: 'Upload code coverage'
- uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v6
- with:
- file: ./build/reports/jacoco/jacocoReport/jacocoReport.xml
diff --git a/docker/bin/postgres-head b/docker/bin/postgres-head
index b350f19e69..b97f043cef 100755
--- a/docker/bin/postgres-head
+++ b/docker/bin/postgres-head
@@ -1,45 +1,31 @@
#!/usr/bin/env bash
set -euo pipefail
-# Helper script to start a postgres container built from source for testing the PGJDBC driver.
+# Build the PostgreSQL devel image from apt.postgresql.org's *-pgdg-snapshot
+# suite and start it via the shared postgres-server compose.
+#
+# PG_MAJOR = 19 | 18 | ... - PostgreSQL major to install
+# (unset => the build picks the
+# highest available in pgdg-snapshot)
+# PG_IMAGE = image tag to build and run - defaults to pgjdbc/postgres-devel:local
log () {
echo "$@" 1>&2
}
main () {
- local publish_port="${PG_PUBLISH_PORT:-5432}"
-
-
- local branch_name="${PG_BRANCH_NAME:-master}"
- local branch_json
- branch_json=$(curl -q -s -f "https://api.github.com/repos/postgres/postgres/branches/${branch_name}")
- branch_json_field () {
- jq -r -c <<<"${branch_json}" "${1}"
- }
- local branch_sha
- branch_sha="$(branch_json_field .commit.sha)"
- local branch_message
- branch_message="$(branch_json_field .commit.commit.message)"
-
- log "================================================================================"
- log "Will build branch ${branch_name} with SHA=${branch_sha}"
- log ""
- log "${branch_message}"
- log ""
- log "================================================================================"
-
- # Determine our current directory and change to be in the same directory as the compose file
- local script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
- cd "${script_dir}/../postgres-head"
-
- log "Building container for PostgreSQL built from source"
- docker-compose build \
- --build-arg "GIT_SHA=${branch_sha}" \
- --build-arg "GIT_TAG=${branch_name}"
-
- log "Starting Postgres built from source and mapping to local port ${publish_port}"
- exec docker-compose run --rm --publish=${publish_port}:5432 pgdb
+ local image="${PG_IMAGE:-pgjdbc/postgres-devel:local}"
+ local script_dir
+ script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )"
+
+ log "Building ${image} (PG_MAJOR=${PG_MAJOR:-auto from pgdg-snapshot})"
+ docker build \
+ --build-arg "PG_MAJOR=${PG_MAJOR:-}" \
+ --tag "${image}" \
+ "${script_dir}/../postgres-head"
+
+ log "Starting the devel server through the shared postgres-server compose"
+ PG_IMAGE="${image}" exec "${script_dir}/postgres-server"
}
main "$@"
diff --git a/docker/postgres-head/.dockerignore b/docker/postgres-head/.dockerignore
index 88aaf4229c..a9b303d31e 100644
--- a/docker/postgres-head/.dockerignore
+++ b/docker/postgres-head/.dockerignore
@@ -1,5 +1,3 @@
-# Ignore everything
+# The build only needs the Dockerfile; sources are git-cloned and the entrypoint
+# is fetched from the official image during the build.
**
-
-# Explicitly included
-!scripts/**
diff --git a/docker/postgres-head/Dockerfile b/docker/postgres-head/Dockerfile
index 77a100453e..5c2eb159d7 100644
--- a/docker/postgres-head/Dockerfile
+++ b/docker/postgres-head/Dockerfile
@@ -1,83 +1,80 @@
-FROM ubuntu:26.04@sha256:f3d28607ddd78734bb7f71f117f3c6706c666b8b76cbff7c9ff6e5718d46ff64
+# syntax=docker/dockerfile:1
+# PostgreSQL devel image built from apt.postgresql.org's `*-pgdg-snapshot`.
+# Drop-in compatible with the official `postgres:*` image, so the shared
+# docker/postgres-server compose runs it via PG_IMAGE.
-RUN export DEBIAN_FRONTEND=noninteractive \
- && apt-get update \
- && apt-get -y install \
- clang \
- bison \
- build-essential \
- dumb-init \
- flex \
- git \
- libperl-dev \
- libreadline-dev \
- libssl-dev \
- libxml2-dev \
- libxml2-utils \
- libxslt-dev \
- llvm \
- locales \
- pkg-config \
- python3-dev \
- tcl-dev \
- xsltproc \
- zlib1g-dev \
- && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8
+FROM debian:trixie-slim
-ARG GIT_URL="https://github.com/postgres/postgres";
-ARG GIT_TAG="master"
-# If specified, this allows us to help Docker cache the clone step
-ARG GIT_SHA
+# Empty default => pick the highest postgresql-N available in pgdg-snapshot
+# at build time, so the image follows new major releases without a Dockerfile
+# bump. Pass `--build-arg PG_MAJOR=18` to pin a specific major.
+ARG PG_MAJOR=""
-RUN mkdir -p /build/postgres \
- && git clone -b "${GIT_TAG}" --single-branch "${GIT_URL}" --depth 10 /build/postgres \
- && cd /build/postgres \
- && [ -z "${GIT_SHA:-}" ] || git reset --hard "${GIT_SHA:-}"
+# PATH points at a stable symlink populated below; PG_MAJOR is exported by the
+# shared docker/postgres-server entrypoint from `postgres --version` at runtime.
+ENV PATH=/usr/lib/postgresql/current/bin:$PATH \
+ PGDATA=/var/lib/postgresql/data \
+ LANG=en_US.utf8
-# Default configure options that can be overridden with build arg
-ARG CONFIGURE_OPTS=" \
- --enable-debug \
- --enable-cassert \
- --with-llvm \
- --with-tcl \
- --with-perl \
- --with-python \
- --with-ssl=openssl \
- --with-libxml \
- "
-# Additional configure options to append to defaults
-ARG CONFIGURE_ADD_OPTS
-
-WORKDIR /build/postgres
-RUN ./configure ${CONFIGURE_OPTS} ${CONFIGURE_ADD_OPTS:-}
-ARG MAKE_OPTS="-j 8"
-ARG MAKE_ADD_OPTS
-RUN make ${MAKE_OPTS} ${MAKE_ADD_OPTS}
-RUN make install
-
-ARG MAKE_CONTRIB_OPTS
-ARG MAKE_ADD_OPTS
-
-WORKDIR /build/postgres/contrib
-RUN make ${MAKE_CONTRIB_OPTS:-${MAKE_OPTS}} ${MAKE_CONTRIB_ADD_OPTS}
-RUN make install
+# Match the official image's postgres user (uid/gid 999) so volumes and bind
+# mounts stay portable between this image and the official one.
+RUN set -eux; \
+ groupadd -r postgres --gid=999; \
+ useradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres
-ENV LANG en_US.utf8
-ENV GIT_SHA=$GIT_SHA
-ENV GIT_TAG=$GIT_TAG
-ENV PATH $PATH:/usr/local/pgsql/bin
-ENV PGDATA /var/lib/postgresql/data
+# Cache mounts keep apt lists/debs out of the image layer, so no rm needed.
+# docker-clean is removed so apt keeps the downloaded .debs in the cache.
+ARG PG_MAJOR
+RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
+ --mount=type=cache,target=/var/lib/apt,sharing=locked \
+ set -eux; \
+ export DEBIAN_FRONTEND=noninteractive; \
+ rm -f /etc/apt/apt.conf.d/docker-clean; \
+ apt-get update -qq; \
+ # stdout->/dev/null silences dpkg's per-package "Selecting/Unpacking/Setting up"
+ # chatter that apt-get -qq does not reach; errors stay on stderr.
+ apt-get -y -qq install --no-install-recommends \
+ ca-certificates curl gosu locales >/dev/null; \
+ install -d /usr/share/postgresql-common/pgdg; \
+ curl -fsS https://www.postgresql.org/media/keys/ACCC4CF8.asc \
+ -o /usr/share/postgresql-common/pgdg/apt.postgresql.org.asc; \
+ . /etc/os-release; \
+ # Enable every numeric component (one per major) that the snapshot ships,
+ # so apt-cache search can find postgresql-N for any major.
+ COMPONENTS="$(curl -fsS "http://apt.postgresql.org/pub/repos/apt/dists/$VERSION_CODENAME-pgdg-snapshot/Release"; \
+ | awk '/^Components:/ {for (i=2; i<=NF; i++) print $i}' | tr '\n' ' ')"; \
+ echo "deb [signed-by=/usr/share/postgresql-common/pgdg/apt.postgresql.org.asc] \
+http://apt.postgresql.org/pub/repos/apt $VERSION_CODENAME-pgdg-snapshot $COMPONENTS" \
+ > /etc/apt/sources.list.d/pgdg-snapshot.list; \
+ apt-get update -qq; \
+ if [ -z "${PG_MAJOR:-}" ]; then \
+ PG_MAJOR="$(apt-cache search --names-only '^postgresql-[0-9]+$' | awk '{print $1}' | sed 's/postgresql-//' | sort -n | tail -n1)"; \
+ fi; \
+ [ -n "$PG_MAJOR" ] || { echo "Could not determine PG_MAJOR"; exit 1; }; \
+ echo "Installing PostgreSQL major $PG_MAJOR"; \
+ # -t targets the snapshot suite explicitly; pgdg-snapshot has lower apt
+ # priority by default, so without -t apt prefers older stable libpq5 and
+ # the snapshot's version dependency can't be satisfied.
+ apt-get -y -qq install --no-install-recommends \
+ -t "$VERSION_CODENAME-pgdg-snapshot" \
+ "postgresql-$PG_MAJOR" "postgresql-contrib-$PG_MAJOR" >/dev/null; \
+ localedef -i en_US -c -f UTF-8 en_US.UTF-8; \
+ ln -sfn "/usr/lib/postgresql/$PG_MAJOR" /usr/lib/postgresql/current; \
+ ln -sfn "/usr/share/postgresql/$PG_MAJOR" /usr/share/postgresql/current; \
+ dpkg-query -W "postgresql-$PG_MAJOR" | tee /usr/local/share/postgresql-version
-# explicitly set user/group IDs
RUN set -eux; \
- groupadd -r postgres --gid=999; \
- useradd -r -g postgres --uid=999 --home-dir=/var/lib/postgresql --shell=/bin/bash postgres; \
- mkdir -p /var/lib/postgresql; \
- chown -R postgres:postgres /var/lib/postgresql; \
- mkdir -p "$PGDATA" ; \
- chown -R postgres:postgres "$PGDATA"
+ mkdir -p /var/lib/postgresql "$PGDATA" /docker-entrypoint-initdb.d /var/run/postgresql; \
+ chown -R postgres:postgres /var/lib/postgresql "$PGDATA" /var/run/postgresql; \
+ chmod 1777 /var/run/postgresql; \
+ echo "listen_addresses = '*'" >> /usr/share/postgresql/current/postgresql.conf.sample
+
+# Reuse the official entrypoint (pinned commit) rather than maintaining a fork.
+ADD --chmod=0755 https://raw.githubusercontent.com/docker-library/postgres/dc8f7ae06a43a6c9647d9b7ca3b270bd148307fb/d... /usr/local/bin/docker-entrypoint.sh
-USER postgres
-ADD scripts/entrypoint.sh /
-ENTRYPOINT ["/usr/bin/dumb-init", "--"]
-CMD [ "/entrypoint.sh" ]
+WORKDIR /var/lib/postgresql
+VOLUME /var/lib/postgresql/data
+EXPOSE 5432
+STOPSIGNAL SIGINT
+ENTRYPOINT ["docker-entrypoint.sh"]
+CMD ["postgres"]
diff --git a/docker/postgres-head/README.md b/docker/postgres-head/README.md
index e22aaf96f8..6b888f752f 100644
--- a/docker/postgres-head/README.md
+++ b/docker/postgres-head/README.md
@@ -1 +1,11 @@
-Docker Compose script helps to start a PostgreSQL instance that builds from source.
+PostgreSQL development image built from apt.postgresql.org's `*-pgdg-snapshot`
+suite. Drop-in compatible with the official `postgres:*` image.
+
+The build picks the highest `postgresql-N` available in pgdg-snapshot at build
+time; pass `--build-arg PG_MAJOR=18` to pin a specific major. The configure
+flags match the released pgdg builds (`--with-gssapi`, `--with-llvm`,
+`--with-icu`, `--with-lz4`, `--with-zstd`, `--with-perl`, `--with-python`,
+`--with-tcl`).
+
+See [docker/postgres-server/README.md](../postgres-server/README.md) for how
+to run this image through the shared compose.
diff --git a/docker/postgres-head/docker-compose.yml b/docker/postgres-head/docker-compose.yml
deleted file mode 100644
index 211e60c0e8..0000000000
--- a/docker/postgres-head/docker-compose.yml
+++ /dev/null
@@ -1,7 +0,0 @@
-services:
- pgdb:
- build:
- dockerfile: Dockerfile
- context: .
- ports:
- - 5432:5432
diff --git a/docker/postgres-head/scripts/entrypoint.sh b/docker/postgres-head/scripts/entrypoint.sh
deleted file mode 100755
index 1d3cf2d803..0000000000
--- a/docker/postgres-head/scripts/entrypoint.sh
+++ /dev/null
@@ -1,57 +0,0 @@
-#!/usr/bin/env bash
-set -euo pipefail
-
-log () {
- echo "$(date) - $@" 1>&2
-}
-
-gen_pg_hba_conf () {
- cat <<EOF
-# LOCAL
-local all all trust
-local replication all trust
-# HOST
-host all postgres 0.0.0.0/0 trust
-host authtest nobody all trust
-host authtest pword all password
-host authtest md51 all md5
-host authtest scram all scram-sha-256
-host all auth_plugin_test_md5 0.0.0.0/0 md5
-host all /test_password_md5.* 0.0.0.0/0 md5
-host all /test_password_scram.* 0.0.0.0/0 scram-sha-256
-host all auth_plugin_test_scram 0.0.0.0/0 scram-sha-256
-host all all 0.0.0.0/0 scram-sha-256
-host replication postgres 0.0.0.0/0 trust
-EOF
-}
-
-gen_postgresql_conf () {
- cat <<EOF
-listen_addresses = '*'
-fsync = off
-synchronous_commit = on
-full_page_writes = off
-max_wal_senders = 10
-max_replication_slots = 10
-wal_level = logical
-EOF
-}
-
-main () {
- log "Initializing with PGDATA=${PGDATA}"
-
- initdb --no-sync
- gen_pg_hba_conf >"${PGDATA}/pg_hba.conf"
- gen_postgresql_conf >>"${PGDATA}/postgresql.conf"
-
- log "Creating test user and database"
- postgres --single -D /var/lib/postgresql/data/ <<<"
- CREATE USER test WITH PASSWORD 'test';
- CREATE DATABASE test WITH OWNER test;
- "
-
- log "Starting PostgreSQL server for GIT_SHA=${GIT_SHA:-} GIT_TAG=${GIT_TAG:-}"
- exec postgres -D "${PGDATA}"
-}
-
-main "$@"
diff --git a/docker/postgres-server/README.md b/docker/postgres-server/README.md
index 8c342a9344..0bb59cdec6 100644
--- a/docker/postgres-server/README.md
+++ b/docker/postgres-server/README.md
@@ -27,6 +27,23 @@ Example usages
docker compose down && PGV=latest docker compose up
+Testing against PostgreSQL HEAD
+===============================
+
+To run the driver tests against an unreleased PostgreSQL, build the devel
+image (from `apt.postgresql.org`'s `pgdg-snapshot` suite) inline via the
+shared compose:
+
+ PG_IMAGE=pgjdbc/postgres-devel:local docker compose up --build
+
+Or, equivalently, without the `--build` flag:
+
+ PG_IMAGE=pgjdbc/postgres-devel:local PG_PULL_POLICY=build docker compose up
+
+The build picks the highest `postgresql-N` available in `pgdg-snapshot`. Pass
+`PG_MAJOR=18` to pin a specific major. See [docker/postgres-head/README.md](../postgres-head/README.md)
+for the Dockerfile.
+
Alternative Foreground Helper
=============================
Run a database server configured for testing the driver in the foreground via:
diff --git a/docker/postgres-server/docker-compose.yml b/docker/postgres-server/docker-compose.yml
index 1108aa819e..e47b527fb1 100644
--- a/docker/postgres-server/docker-compose.yml
+++ b/docker/postgres-server/docker-compose.yml
@@ -1,6 +1,13 @@
services:
pgdb:
- image: postgres:${PGV:-latest}
+ image: ${PG_IMAGE:-postgres:${PGV:-latest}}
+ # With the default pull_policy (missing) this section is ignored and the
+ # official postgres:<PGV> is pulled; force a build with PG_PULL_POLICY=build.
+ build:
+ context: ../postgres-head
+ args:
+ PG_MAJOR: ${PG_MAJOR:-}
+ pull_policy: ${PG_PULL_POLICY:-missing}
ports:
- 5432:5432
- 5433:5433
@@ -31,3 +38,14 @@ services:
- GITHUB_ACTIONS=true
command: >-
postgres
+ healthcheck:
+ # Probe the TCP listener with bash, not the socket: the entrypoint runs a
+ # socket-only temp server while initialising, so the listener is reachable
+ # only once the real server is up. This is version-, auth- and SSL-agnostic,
+ # unlike pg_isready (the Debian pg_wrapper cannot resolve it for servers
+ # older than 9.3). It lets consumers `docker compose up --wait`.
+ test: ["CMD-SHELL", "bash -c 'exec 3<>/dev/tcp/127.0.0.1/5432'"]
+ interval: 5s
+ timeout: 5s
+ retries: 30
+ start_period: 30s
diff --git a/docker/postgres-server/scripts/entrypoint.sh b/docker/postgres-server/scripts/entrypoint.sh
index be598817e6..ccb5c469f5 100755
--- a/docker/postgres-server/scripts/entrypoint.sh
+++ b/docker/postgres-server/scripts/entrypoint.sh
@@ -4,6 +4,12 @@ set -euo pipefail
. /custom/scripts/common.sh
main () {
+ # The official postgres image exports PG_MAJOR; the from-source image does
+ # not. Derive a bare integer major from the server binary when unset so the
+ # version checks below (and the upstream docker-entrypoint.sh) work.
+ : "${PG_MAJOR:=$(postgres --version | grep -oE '[0-9]+' | head -n1)}"
+ export PG_MAJOR
+
# Make a copy of certdir so we can edit the files
cp -r /custom/certdir /home/certdir
chown postgres:postgres /home/certdir/*.key
diff --git a/docker/postgres-server/scripts/post-startup.sh b/docker/postgres-server/scripts/post-startup.sh
index 6b98942335..d2ada9895a 100755
--- a/docker/postgres-server/scripts/post-startup.sh
+++ b/docker/postgres-server/scripts/post-startup.sh
@@ -38,7 +38,8 @@ create_replica () {
CREATE USER ${replication_user} WITH REPLICATION PASSWORD '${replication_pass}';
SELECT * FROM pg_create_physical_replication_slot('${replication_slot_name}');
"
- pg_basebackup -D "${replica_data_dir}" -S "${replication_slot_name}" -X stream -P -Fp -R
+ # checkpoint=fast avoids the "checkpoint starting: force wait" delay
+ pg_basebackup --checkpoint=fast -D "${replica_data_dir}" -S "${replication_slot_name}" -X stream -P -Fp -R
if is_pg_version_less_than "10"; then
cat <<EOF >>"${replica_data_dir}/postgresql.conf"
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2024-05-21 20:43 ` "davecramer (@davecramer)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: davecramer (@davecramer) @ 2024-05-21 20:43 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
docker supports HEAD ?
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2024-05-21 20:48 ` "vlsi (@vlsi)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: vlsi (@vlsi) @ 2024-05-21 20:48 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
There's an extra `Dockerfile` that executes the relevant commands to build from source: https://github.com/pgjdbc/pgjdbc/blob/450488c142fdc368cab54e8257407603acc18c4f/docker/postgres-head/...
So if you execute `docker-compose up` from `docker/postgresql-head`, then it would build and spawn PG from sources.
I think this PR is good to go. We will have both `HEAD` (build from source) and `latest` (the latest from DockerHub == the latest released one).
Building PG from source takes ~5 minutes, and we can possibly afford that on a per-PR basis.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2024-05-21 20:54 ` "davecramer (@davecramer)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: davecramer (@davecramer) @ 2024-05-21 20:54 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
cool, then I can test the ssl stuff, although I'm getting an error with the certificates. Have you tried that lately ?
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2024-05-21 20:59 ` "vlsi (@vlsi)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: vlsi (@vlsi) @ 2024-05-21 20:59 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
It looks like `postgresql-head` misses ssl configuration (e.g. it does not customize `pg_hba.conf` and so on).
It should probably reuse `post-startup.sh` and so on. I'll have a look into it.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2024-05-22 09:51 ` "davecramer (@davecramer)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: davecramer (@davecramer) @ 2024-05-22 09:51 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
more specifically, the makefile in the cert dir fails on my mac using openssl 3.3.0
```
openssl pkcs8 -topk8 -in goodclient.key -out goodclient.pk8 -outform DER -v1 PBE-MD5-DES -passout pass:sslpwd
Error encrypting key
00E163E201000000:error:0308010C:digital envelope routines:inner_evp_generic_fetch:unsupported:crypto/evp/evp_fetch.c:355:Global default library context, Algorithm (PBKDF1 : 0), Properties (<null>)
00E163E201000000:error:11800067:PKCS12 routines:PKCS12_item_i2d_encrypt_ex:encrypt error:crypto/pkcs12/p12_decr.c:199:
00E163E201000000:error:11800067:PKCS12 routines:PKCS8_set0_pbe_ex:encrypt error:crypto/pkcs12/p12_p8e.c:80:
```
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2024-05-22 12:45 ` "sehrope (@sehrope)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: sehrope (@sehrope) @ 2024-05-22 12:45 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
(on .github/workflows/main.yml:154)
There's a lot of random failures building and testing against HEAD so we should ignore errors. We'll see have the details in actions to see what happened, but no point in CI randomly turning red because HEAD itself is broken.
That's what the omni action would do: https://github.com/pgjdbc/pgjdbc/blob/450488c142fdc368cab54e8257407603acc18c4f/.github/workflows/omn...
Also, we should probably disable HEAD entirely for PRs and enable it always for master and any daily runs.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2025-01-28 13:23 ` "vlsi (@vlsi)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: vlsi (@vlsi) @ 2025-01-28 13:23 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
I've rebased this, and made it so it runs `HEAD` tests on push to the branches only.
So the PR does not spend time on building `HEAD` (~5m), and it does not suffer from occasional PG HEAD failures.
Here's a sample run in a fork to verify branch-based builds: https://github.com/vlsi/pgjdbc/actions/runs/13010465219/job/36286871109
It fails for Java 21 as follows (it looks like SSL exception is different in Java 21 in this case):
```
org.postgresql.test.ssl.SslTest > run(Hostname, TestDatabase, SslMode, SslNegotiation, ClientCertificate, ClientRootCertificate, GSSEncMode)[1761], host=BAD, db=certdb sslMode=REQUIRE, sslNegotiation=DIRECT, clientCert=GOOD, clientRootCert=GOOD, gssEncMode=PREFER failure marker
FAILURE 0,0sec, org.postgresql.test.ssl.SslTest > run(Hostname, TestDatabase, SslMode, SslNegotiation, ClientCertificate, ClientRootCertificate, GSSEncMode)[1761], host=BAD, db=certdb sslMode=REQUIRE, sslNegotiation=DIRECT, clientCert=GOOD, clientRootCert=GOOD, gssEncMode=PREFER
org.opentest4j.AssertionFailedError: SQLException present when it was not expected
at app//org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:38)
at app//org.junit.jupiter.api.Assertions.fail(Assertions.java:138)
at app//org.postgresql.test.ssl.SslTest.checkErrorCodes(SslTest.java:281)
at app//org.postgresql.test.ssl.SslTest.run(SslTest.java:492)
Caused by: org.postgresql.util.PSQLException: SSL error: Remote host terminated the handshake
at app//org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:53)
at app//org.postgresql.core.v3.ConnectionFactoryImpl.enableSSL(ConnectionFactoryImpl.java:594)
at app//org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:205)
at app//org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
at app//org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:54)
at app//org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:274)
at app//org.postgresql.Driver.makeConnection(Driver.java:448)
at app//org.postgresql.Driver.connect(Driver.java:298)
at platform/[email protected]/java.sql.DriverManager.getConnection(DriverManager.java:683)
at platform/[email protected]/java.sql.DriverManager.getConnection(DriverManager.java:191)
at app//org.postgresql.test.TestUtil.openDB(TestUtil.java:401)
at app//org.postgresql.test.ssl.SslTest.run(SslTest.java:482)
Caused by: javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1714)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1514)
at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:455)
at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:426)
at org.postgresql.ssl.MakeSSL.convert(MakeSSL.java:51)
... 11 more
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:494)
at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:483)
at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:111)
at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1506)
```
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2025-01-28 13:28 ` "davecramer (@davecramer)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: davecramer (@davecramer) @ 2025-01-28 13:28 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
any idea what is causing the SSL errors ?
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2025-01-28 13:55 ` "vlsi (@vlsi)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: vlsi (@vlsi) @ 2025-01-28 13:55 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
>any idea what is causing the SSL errors ?
It turns out `docker/postgres-head` does not configure SSL at the database side. I suggest skipping HEAD+SSL combo for now, so we fix it later.
^ permalink raw reply [nested|flat] 11+ messages in thread
* Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD
@ 2025-01-28 13:57 ` "vlsi (@vlsi)" <[email protected]>
9 siblings, 0 replies; 11+ messages in thread
From: vlsi (@vlsi) @ 2025-01-28 13:57 UTC (permalink / raw)
To: pgjdbc/pgjdbc <[email protected]>
(on .github/workflows/main.yml:154)
@sehrope, I've altered the PR so it executes HEAD build in the branch-based builds only. Does it resolve your concern?
^ permalink raw reply [nested|flat] 11+ messages in thread
end of thread, other threads:[~2025-01-28 13:57 UTC | newest]
Thread overview: 11+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-05-21 19:39 [pgjdbc/pgjdbc] PR #3257: test: add CI testing against PostgreSQL HEAD "vlsi (@vlsi)" <[email protected]>
2024-05-21 20:43 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "davecramer (@davecramer)" <[email protected]>
2024-05-21 20:48 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "vlsi (@vlsi)" <[email protected]>
2024-05-21 20:54 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "davecramer (@davecramer)" <[email protected]>
2024-05-21 20:59 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "vlsi (@vlsi)" <[email protected]>
2024-05-22 09:51 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "davecramer (@davecramer)" <[email protected]>
2024-05-22 12:45 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "sehrope (@sehrope)" <[email protected]>
2025-01-28 13:23 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "vlsi (@vlsi)" <[email protected]>
2025-01-28 13:28 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "davecramer (@davecramer)" <[email protected]>
2025-01-28 13:55 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "vlsi (@vlsi)" <[email protected]>
2025-01-28 13:57 ` Re: [pgjdbc/pgjdbc] PR #3257: test: add GitHub CI testing with PostgreSQL 15 and HEAD "vlsi (@vlsi)" <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox