public inbox for [email protected]
help / color / mirror / Atom feedFrom: Dave Page <[email protected]>
To: pgadmin-hackers <[email protected]>
Subject: Re: UBI based container patch
Date: Fri, 5 Feb 2021 13:47:11 +0000
Message-ID: <CA+OCxoyNaokyPNgJi0yi4dBSxJR-FH8ueE62_VSPfKSQyBZxUQ@mail.gmail.com> (raw)
In-Reply-To: <CA+OCxoz0JKkgw5tkiqK1FemLhB3O4FuXu7MOBF0B=6EYyvBqQA@mail.gmail.com>
References: <CA+OCxoz0JKkgw5tkiqK1FemLhB3O4FuXu7MOBF0B=6EYyvBqQA@mail.gmail.com>
* Do not commit this! *
For the archives....
I reworked the patch to minimise the size of the resulting container. I
managed to knock 52MB off, taking it down to 410MB. In parallel, I also
managed to reduce the size of the Alpine container by about 25MB as I
realised we were shipping some files that aren't required.
To reduce the size of the UBI container, I essentially built a 'ubi-micro'
container, by installing only the RPMs that are actually required into an
alternate root directory from within the standard UBI container, and then
copied that directory into the root of a 'FROM scratch' container.
Unfortunately it seems that the issue is essentially the RPM packaging;
pulling in things like Postfix also pulls in a huge amount of dependencies
that aren't really required (for pgAdmin). Alpine avoids this by minimising
their package dependencies.
So, I think it's not feasible to move to UBI, without adding a lot to the
size of the resulting container.
Here's the patch anyway!
On Mon, Feb 1, 2021 at 4:21 PM Dave Page <[email protected]> wrote:
> * Do not commit this! *
>
> The dev team have discussed moving the pgAdmin container to be based on
> Redhat's UBI instead of Alpine Linux. I got some time today to work on
> that, and a patch is attached for interest/archives.
>
> Whilst the build process is arguably cleaner with this patch, plus it
> would be easier to maintain the versions of the PostgreSQL tools that are
> included, unfortunately using UBI takes the size of the container from
> ~278MB to ~462MB.
>
> So... unless someone finds a huge error in the patch that causes this
> bloat, I think we have no choice but to stick with Alpine Linux.
>
> Any thoughts/comments?
>
> --
> Dave Page
> Blog: http://pgsnake.blogspot.com
> Twitter: @pgsnake
>
> EDB: http://www.enterprisedb.com
>
>
--
Dave Page
Blog: http://pgsnake.blogspot.com
Twitter: @pgsnake
EDB: http://www.enterprisedb.com
Attachments:
[application/octet-stream] ubi-container_v2.diff (12.3K, 3-ubi-container_v2.diff)
download | inline diff:
diff --git a/.dockerignore b/.dockerignore
index 4c3d8b0ab..5feb2bae6 100644
--- a/.dockerignore
+++ b/.dockerignore
@@ -1,4 +1,6 @@
.git
+docs/en_US/_build/html/_sources
+docs/en_US/_build/html/_static/*.png
web/node_modules
web/*.log
web/regression
diff --git a/Dockerfile b/Dockerfile
index 1413bb10c..f28bac787 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -12,20 +12,12 @@
# and clean up the web/ source code
#########################################################################
-FROM node:14-alpine3.12 AS app-builder
-
-RUN apk add --no-cache \
- autoconf \
- automake \
- bash \
- g++ \
- libc6-compat \
- libjpeg-turbo-dev \
- libpng-dev \
- make \
- nasm \
- git \
- zlib-dev
+FROM registry.redhat.io/ubi8/ubi AS app-builder
+
+# Install dependencies
+RUN dnf module install -y nodejs:14
+RUN dnf install -y git
+RUN npm install --global yarn
# Create the /pgadmin4 directory and copy the source into it. Explicitly
# remove the node_modules directory as we'll recreate a clean version, as well
@@ -60,145 +52,92 @@ RUN npm install && \
karma.conf.js \
./pgadmin/static/js/generated/.cache
-#########################################################################
-# Now, create a documentation build container for the Sphinx docs
-#########################################################################
+##########################################################################
+## Now, create the virtual environment
+##########################################################################
-FROM python:3.9-alpine3.12 as docs-builder
+FROM registry.redhat.io/ubi8/ubi AS env-builder
# Install dependencies
+RUN dnf install -y gcc make python3-devel python3-pip postgresql-devel krb5-devel
+
COPY requirements.txt /
-RUN apk add --no-cache \
- make \
- build-base \
- openssl-dev \
- libffi-dev \
- postgresql-dev \
- krb5-dev && \
- pip install --no-cache-dir \
- sphinx && \
- pip install --no-cache-dir -r requirements.txt
+RUN python3 -m venv /venv
+RUN /venv/bin/pip3 install --upgrade pip
+RUN /venv/bin/pip3 install -r requirements.txt
+RUN /venv/bin/pip3 install gunicorn
+
+##########################################################################
+## Now, create a documentation build container for the Sphinx docs
+##########################################################################
+
+FROM registry.redhat.io/ubi8/ubi AS doc-builder
+
+# Install dependencies
+RUN dnf install -y make python3
+COPY --from=env-builder /venv /venv
+RUN /venv/bin/pip3 install sphinx
# Copy the docs from the local tree. Explicitly remove any existing builds that
-# may be present
+# may be present. We don't use the .dockerignore for this as we need to copy
+# the _build directory later
COPY docs /pgadmin4/docs
COPY web /pgadmin4/web
RUN rm -rf /pgadmin4/docs/en_US/_build
# Build the docs
-RUN LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make -C /pgadmin4/docs/en_US -f Makefile.sphinx html
+RUN source /venv/bin/activate && \
+ LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make -C /pgadmin4/docs/en_US -f Makefile.sphinx html
-# Cleanup unwanted files
-RUN rm -rf /pgadmin4/docs/en_US/_build/html/_sources
-RUN rm -rf /pgadmin4/docs/en_US/_build/html/_static/*.png
+##########################################################################
+## Now, create the base image
+##########################################################################
-#########################################################################
-# Create additional builders to get all of the PostgreSQL utilities
-#########################################################################
+FROM registry.redhat.io/ubi8/ubi AS base-env
-FROM postgres:9.6-alpine as pg96-builder
-FROM postgres:10-alpine as pg10-builder
-FROM postgres:11-alpine as pg11-builder
-FROM postgres:12-alpine as pg12-builder
-FROM postgres:13-alpine as pg13-builder
-
-FROM alpine:3.11 as tool-builder
-
-# Copy the PG binaries
-
-COPY --from=pg96-builder /usr/local/bin/pg_dump /usr/local/pgsql/pgsql-9.6/
-COPY --from=pg96-builder /usr/local/bin/pg_dumpall /usr/local/pgsql/pgsql-9.6/
-COPY --from=pg96-builder /usr/local/bin/pg_restore /usr/local/pgsql/pgsql-9.6/
-COPY --from=pg96-builder /usr/local/bin/psql /usr/local/pgsql/pgsql-9.6/
-
-COPY --from=pg10-builder /usr/local/bin/pg_dump /usr/local/pgsql/pgsql-10/
-COPY --from=pg10-builder /usr/local/bin/pg_dumpall /usr/local/pgsql/pgsql-10/
-COPY --from=pg10-builder /usr/local/bin/pg_restore /usr/local/pgsql/pgsql-10/
-COPY --from=pg10-builder /usr/local/bin/psql /usr/local/pgsql/pgsql-10/
-
-COPY --from=pg11-builder /usr/local/bin/pg_dump /usr/local/pgsql/pgsql-11/
-COPY --from=pg11-builder /usr/local/bin/pg_dumpall /usr/local/pgsql/pgsql-11/
-COPY --from=pg11-builder /usr/local/bin/pg_restore /usr/local/pgsql/pgsql-11/
-COPY --from=pg11-builder /usr/local/bin/psql /usr/local/pgsql/pgsql-11/
-
-COPY --from=pg12-builder /usr/local/bin/pg_dump /usr/local/pgsql/pgsql-12/
-COPY --from=pg12-builder /usr/local/bin/pg_dumpall /usr/local/pgsql/pgsql-12/
-COPY --from=pg12-builder /usr/local/bin/pg_restore /usr/local/pgsql/pgsql-12/
-COPY --from=pg12-builder /usr/local/bin/psql /usr/local/pgsql/pgsql-12/
-
-COPY --from=pg13-builder /usr/local/bin/pg_dump /usr/local/pgsql/pgsql-13/
-COPY --from=pg13-builder /usr/local/bin/pg_dumpall /usr/local/pgsql/pgsql-13/
-COPY --from=pg13-builder /usr/local/bin/pg_restore /usr/local/pgsql/pgsql-13/
-COPY --from=pg13-builder /usr/local/bin/psql /usr/local/pgsql/pgsql-13/
+# Build the base OS
+RUN mkdir /output
+RUN rpm -i https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm
+RUN dnf -y install --downloadonly --downloaddir=/tmp --releasever 8 --setopt=install_weak_deps=false --nodocs glibc-minimal-langpack coreutils-single postfix postgresql13 postgresql12 postgresql11 postgresql10 postgresql96 python3 sed sudo
+RUN dnf -y install --installroot=/output --releasever 8 --setopt=install_weak_deps=false --nodocs /tmp/*.rpm
+RUN dnf -y --installroot /output --releasever 8 clean all
#########################################################################
# Assemble everything into the final container.
#########################################################################
-FROM python:3.9-alpine3.12
-
-COPY --from=tool-builder /usr/local/pgsql /usr/local/
+FROM scratch
WORKDIR /pgadmin4
ENV PYTHONPATH=/pgadmin4
# Copy in the code and docs
+COPY --from=base-env /output /
COPY --from=app-builder /pgadmin4/web /pgadmin4
-COPY --from=docs-builder /pgadmin4/docs/en_US/_build/html/ /pgadmin4/docs
-COPY requirements.txt /pgadmin4/requirements.txt
-
-# License files
-COPY LICENSE /pgadmin4/LICENSE
-COPY DEPENDENCIES /pgadmin4/DEPENDENCIES
-
-# Install build-dependencies, build & install C extensions and purge deps in
-# one RUN step
-RUN apk add --no-cache --virtual \
- build-deps \
- build-base \
- postgresql-dev \
- libffi-dev \
- krb5-dev \
- e2fsprogs-dev \
- krb5-server-ldap \
- linux-headers && \
- apk add \
- postfix \
- postgresql-client \
- postgresql-libs \
- krb5-libs \
- shadow \
- sudo \
- libcap && \
- pip install --upgrade pip && \
- pip install --no-cache-dir -r requirements.txt && \
- pip install --no-cache-dir gunicorn && \
- apk del --no-cache build-deps && \
- echo "pgadmin ALL = NOPASSWD: /usr/sbin/postfix start" > /etc/sudoers.d/postfix
-
-# We need the v13 libpq
-COPY --from=pg13-builder /usr/local/lib/libpq.so.5.13 /usr/lib/
-RUN ln -sf /usr/lib/libpq.so.5.13 /usr/lib/libpq.so.5
-
-# Copy the various scripts
+COPY --from=doc-builder /pgadmin4/docs/en_US/_build/html/ /pgadmin4/docs
+COPY --from=env-builder /venv /venv
COPY pkg/docker/run_pgadmin.py /pgadmin4
COPY pkg/docker/gunicorn_config.py /pgadmin4
COPY pkg/docker/entrypoint.sh /entrypoint.sh
-# Precompile and optimize python code to save time and space on startup
-RUN python -O -m compileall -x node_modules /pgadmin4
-
-RUN groupadd -g 5050 pgadmin && \
- useradd -r -u 5050 -g pgadmin pgadmin && \
+# Perform all the OS-level setup. Do this in one RUN command to minimise the
+# number of layers
+RUN echo "pgadmin:x:5050:" >> /etc/group && \
+ echo "pgadmin:!::" >> /etc/gshadow && \
+ echo "pgadmin:x:5050:5050::/pgadmin4:/bin/bash" >> /etc/passwd && \
+ echo "pgadmin:!!:18608::::::" >> /etc/shadow && \
mkdir -p /var/lib/pgadmin && \
chown pgadmin:pgadmin /var/lib/pgadmin && \
touch /pgadmin4/config_distro.py && \
- chown pgadmin:pgadmin /pgadmin4/config_distro.py && \
- setcap CAP_NET_BIND_SERVICE=+eip /usr/local/bin/python3.9
+ chown pgadmin:pgadmin /pgadmin4/config_distro.py # && \
+ setcap CAP_NET_BIND_SERVICE=+eip /usr/libexec/platform-python3.6 && \
+ sed -i 's/inet_protocols = .*/inet_protocols = ipv4/g' /etc/postfix/main.cf && \
+ echo "pgadmin ALL = NOPASSWD: /usr/sbin/postfix start" > /etc/sudoers.d/postfix && \
+
USER pgadmin
# Finish up
VOLUME /var/lib/pgadmin
EXPOSE 80 443
-ENTRYPOINT ["/entrypoint.sh"]
+ENTRYPOINT ["/entrypoint.sh"]
\ No newline at end of file
diff --git a/pkg/docker/entrypoint.sh b/pkg/docker/entrypoint.sh
index 84457d379..a9314bccf 100755
--- a/pkg/docker/entrypoint.sh
+++ b/pkg/docker/entrypoint.sh
@@ -34,7 +34,7 @@ if [ ! -f /var/lib/pgadmin/pgadmin4.db ]; then
# Initialize DB before starting Gunicorn
# Importing pgadmin4 (from this script) is enough
- python run_pgadmin.py
+ /venv/bin/python run_pgadmin.py
export PGADMIN_SERVER_JSON_FILE=${PGADMIN_SERVER_JSON_FILE:-/pgadmin4/servers.json}
# Pre-load any required servers
@@ -42,9 +42,9 @@ if [ ! -f /var/lib/pgadmin/pgadmin4.db ]; then
# When running in Desktop mode, no user is created
# so we have to import servers anonymously
if [ "${PGADMIN_CONFIG_SERVER_MODE}" = "False" ]; then
- /usr/local/bin/python /pgadmin4/setup.py --load-servers "${PGADMIN_SERVER_JSON_FILE}"
+ /venv/bin/python /pgadmin4/setup.py --load-servers "${PGADMIN_SERVER_JSON_FILE}"
else
- /usr/local/bin/python /pgadmin4/setup.py --load-servers "${PGADMIN_SERVER_JSON_FILE}" --user ${PGADMIN_DEFAULT_EMAIL}
+ /venv/bin/python /pgadmin4/setup.py --load-servers "${PGADMIN_SERVER_JSON_FILE}" --user ${PGADMIN_DEFAULT_EMAIL}
fi
fi
fi
@@ -56,13 +56,13 @@ fi
# Get the session timeout from the pgAdmin config. We'll use this (in seconds)
# to define the Gunicorn worker timeout
-TIMEOUT=$(cd /pgadmin4 && python -c 'import config; print(config.SESSION_EXPIRATION_TIME * 60 * 60 * 24)')
+TIMEOUT=$(cd /pgadmin4 && /venv/bin/python -c 'import config; print(config.SESSION_EXPIRATION_TIME * 60 * 60 * 24)')
# NOTE: currently pgadmin can run only with 1 worker due to sessions implementation
# Using --threads to have multi-threaded single-process worker
if [ ! -z ${PGADMIN_ENABLE_TLS} ]; then
- exec gunicorn --timeout ${TIMEOUT} --bind ${PGADMIN_LISTEN_ADDRESS:-[::]}:${PGADMIN_LISTEN_PORT:-443} -w 1 --threads ${GUNICORN_THREADS:-25} --access-logfile ${GUNICORN_ACCESS_LOGFILE:--} --keyfile /certs/server.key --certfile /certs/server.cert -c gunicorn_config.py run_pgadmin:app
+ exec /venv/bin/gunicorn --timeout ${TIMEOUT} --bind ${PGADMIN_LISTEN_ADDRESS:-[::]}:${PGADMIN_LISTEN_PORT:-443} -w 1 --threads ${GUNICORN_THREADS:-25} --access-logfile ${GUNICORN_ACCESS_LOGFILE:--} --keyfile /certs/server.key --certfile /certs/server.cert -c gunicorn_config.py run_pgadmin:app
else
- exec gunicorn --timeout ${TIMEOUT} --bind ${PGADMIN_LISTEN_ADDRESS:-[::]}:${PGADMIN_LISTEN_PORT:-80} -w 1 --threads ${GUNICORN_THREADS:-25} --access-logfile ${GUNICORN_ACCESS_LOGFILE:--} -c gunicorn_config.py run_pgadmin:app
+ exec /venv/bin/gunicorn --timeout ${TIMEOUT} --bind ${PGADMIN_LISTEN_ADDRESS:-[::]}:${PGADMIN_LISTEN_PORT:-80} -w 1 --threads ${GUNICORN_THREADS:-25} --access-logfile ${GUNICORN_ACCESS_LOGFILE:--} -c gunicorn_config.py run_pgadmin:app
fi
diff --git a/web/config.py b/web/config.py
index 8fd8b0391..84792f58c 100644
--- a/web/config.py
+++ b/web/config.py
@@ -350,7 +350,7 @@ SESSION_COOKIE_NAME = 'pga4_session'
# These settings are used when running in web server mode for confirming
# and resetting passwords etc.
# See: http://pythonhosted.org/Flask-Mail/ for more info
-MAIL_SERVER = 'localhost'
+MAIL_SERVER = '127.0.0.1'
MAIL_PORT = 25
MAIL_USE_SSL = False
MAIL_USE_TLS = False
view thread (2+ messages)
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected]
Subject: Re: UBI based container patch
In-Reply-To: <CA+OCxoyNaokyPNgJi0yi4dBSxJR-FH8ueE62_VSPfKSQyBZxUQ@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox