supabase-postgres.git / summary / log / commit / refs
commit cbf34f088ed912d6cdb989ab4f87c1ecb755b4ae
Author: Manuel Mendez <708570+mmlb@users.noreply.github.com>
Commit: GitHub <noreply@github.com>
Date: Mon Aug 31 21:09:35 2026 +0000
AMI: Clean up messy apt handling (#2351)
* qemu: Report disk usage
Just like the AMIs, good to know here too.
* images: report all packages installed at end
* packer: Use better apt fallback config
It turns out that most of the code deleted here was actually not doing
anything useful. For example, switch_mirror modifies
/etc/apt/sources.list in place but the file doesn't have any mirrors
configured there so it's really a no-op! Here's the contents from an
instance I just fired up[^1]:
ubuntu@ip-172-31-26-227:~$ tail -n+1 /etc/apt/sources.list /etc/apt/sources.list.d/*
==> /etc/apt/sources.list <==
# Ubuntu sources have moved to the /etc/apt/sources.list.d/ubuntu.sources
# file, which uses the deb822 format. Use deb822-formatted .sources files
# to manage package sources in the /etc/apt/sources.list.d/ directory.
# See the sources.list(5) manual page for details.
==> /etc/apt/sources.list.d/ubuntu.sources <==
## Note, this file is written by cloud-init on first boot of an instance
## modifications made here will not survive a re-bundle.
--- 8< ---
Types: deb
URIs: http://us-east-1.ec2.ports.ubuntu.com/ubuntu-ports/
Suites: noble noble-updates noble-backports
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
## Ubuntu security updates. Aside from URIs and Suites,
## this should mirror your choices in the previous section.
Types: deb
URIs: http://ports.ubuntu.com/ubuntu-ports
Suites: noble-security
Components: main universe restricted multiverse
Signed-By: /usr/share/keyrings/ubuntu-archive-keyring.gpg
We get fallback handling by apt itself by adding multiple mirrors in
URIs, apt tries first (the regional) and falls back to global ubuntu
repos if there's an issue. We also setup a temporary apt config that
forces the APT_OPTIONS for all apt calls so we don't need to do it in
every call site.
I ended up dropping the in-repo sources file since they are strictly
worse than what we get from AWS in the build. There's no change in
suites or components between old and new, just that we get ubuntu
upstream as a fallback. Well technically there's a slight difference
since we are basing off of cloud-init generated files and they can
theoretically change under us but I'll gamble that it'll be fine or
better off. Besides, one day we'll be on NixOS as the ultimate "make
sure we know everything in the instance" ;).
I also got rid of the `add-apt-repository --yes universe` because
universe is already enabled.
[^1]: AMI=ubuntu/images/hvm-ssd-gp3/ubuntu-noble-24.04-amd64-server-20260604
* packer: Minimize and improve apt-get calls
We are wasting a bunch of time and brain power keeping track of package
installs, so lets minimize them while keeping the grouping/messages
intact.
I did move the package installs from setup_apparmor && setup_grub into
update_install_packages because why not just install everything in one
place and leave the _setup_ to somewhere else. I moved them out of
because package installation should all be done early and ASAP IMO so it
can fail fast if its going to fail.
Also I moved ec2-hibinit-agent, ec2-instance-connect, hibagent here
since this is only run for AMIs and the comment isn't currently
accurate. Either through `export DEBIAN_FRONTEND=noninteractive`,
`disable_services` or maybe it was treated as a bug and is now fixed in
the package I'm not sure but I ran this out of AWS and it was fine.
* packer: Refactor apt usage
Lets be a little cleaner about setup/update/cleanup, also drop the
wrapper functions to plain old apt-get calls since they no longer do
anything useful.
I created setup_apt because I want to move the Install-Recommends=False
into it later and configure all apt calls to use it seamlessly, this way
we will have a better/more visible list of packages being installed.
* packer: Remove dead code from cleanup scripts
No yum handling necessary and commented out code should be deleted.
* packer: Drop all but one apt-get remove invocation from cleanup scripts
One is enough.
* packer: Bookend and otherwise minimize apt-get update/upgrade
There's no point run updates/upgrades unnecessarily while also not
ensuring that we start with latest and end with latest. So I've dropped
any unecessary updates and upgrades from middle of runs. Now we update
and upgrade after start up ({surrogate,qemu}-bootstrap-nix, chroot was
already good) and then once at the end (qemu-bootstrap-nix and
nix-provision). Ansible's apt_repository module does update when
necessary and has not changed.
.github/workflows/test.yml | 8 +
amazon-amd64-nix.pkr.hcl | 5 -
amazon-arm64-nix.pkr.hcl | 5 -
ansible/playbook.yml | 13 -
ansible/tasks/internal/install-salt.yml | 1 -
ansible/tasks/setup-postgrest.yml | 1 -
ansible/tasks/setup-system.yml | 8 -
ansible/tasks/setup-tmux.yml | 1 -
ansible/tasks/setup-tuned.yml | 1 -
ansible/tasks/test-image.yml | 1 -
ebssurrogate/files/sources-arm64.cfg | 10 -
ebssurrogate/files/sources.cfg | 10 -
ebssurrogate/scripts/chroot-bootstrap-nix.sh | 305 ++++++------------------
ebssurrogate/scripts/cleanup-qemu.sh | 152 +++++++-----
ebssurrogate/scripts/cleanup.sh | 50 ++--
ebssurrogate/scripts/nix-provision.sh | 41 ++--
ebssurrogate/scripts/qemu-bootstrap-nix.sh | 104 ++++++--
ebssurrogate/scripts/surrogate-bootstrap-nix.sh | 169 +++++--------
18 files changed, 359 insertions(+), 526 deletions(-)
diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml
index 7d93b2d1..c484a295 100644
--- a/.github/workflows/test.yml
+++ b/.github/workflows/test.yml
@@ -39,21 +39,27 @@ jobs:
steps:
- name: Checkout Repo
uses: supabase/postgres/.github/actions/shared-checkout@HEAD
+
- uses: ./.github/actions/nix-install-ephemeral
+
- name: Set PostgreSQL version environment variable
run: echo "POSTGRES_MAJOR_VERSION=${{ matrix.postgres_version }}" >> $GITHUB_ENV
+
- name: Strip quotes from pg major and set env var
run: |
stripped_version=$(echo "${{ matrix.postgres_version }}" | sed 's/^"\(.*\)"$/\1/')
echo "PGMAJOR=$stripped_version" >> $GITHUB_ENV
+
- name: Generate common-nix.vars.pkr.hcl
run: |
PG_VERSION=$(nix run nixpkgs#yq-go -- '.postgres_release["postgres'${{ matrix.postgres_version }}'"]' ansible/vars.yml)
PG_VERSION=$(echo $PG_VERSION | tr -d '"') # Remove any surrounding quotes
echo 'postgres-version = "'$PG_VERSION'"' > common-nix.vars.pkr.hcl
echo "" >> common-nix.vars.pkr.hcl
+
- id: settings
run: sed -r 's/(\s|\")+//g' common-nix.vars.pkr.hcl >> $GITHUB_OUTPUT
+
- name: Generate args
id: args
run: |
@@ -61,11 +67,13 @@ jobs:
echo "result<<EOF" >> $GITHUB_OUTPUT
echo "$ARGS" >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
+
- name: Resolve git sha
id: resolve-git-sha
uses: ./.github/actions/resolve-git-sha
with:
flake_input: ${{ vars.GIT_SHA_FROM_FLAKE_INPUT }}
+
- name: verify schema.sql is committed
run: |
nix run github:supabase/postgres/${{ steps.resolve-git-sha.outputs.sha }}#dbmate-tool -- --version ${{ env.PGMAJOR }} --flake-url github:supabase/postgres/${{ steps.resolve-git-sha.outputs.sha }}
diff --git a/amazon-amd64-nix.pkr.hcl b/amazon-amd64-nix.pkr.hcl
index 44546218..57e36e89 100644
--- a/amazon-amd64-nix.pkr.hcl
+++ b/amazon-amd64-nix.pkr.hcl
@@ -170,11 +170,6 @@ source "amazon-ebssurrogate" "source" {
build {
sources = ["source.amazon-ebssurrogate.source"]
- provisioner "file" {
- source = "ebssurrogate/files/sources.cfg"
- destination = "/tmp/sources.list"
- }
-
provisioner "file" {
source = "ebssurrogate/files/ebsnvme-id"
destination = "/tmp/ebsnvme-id"
diff --git a/amazon-arm64-nix.pkr.hcl b/amazon-arm64-nix.pkr.hcl
index 7bca54bb..422886c0 100644
--- a/amazon-arm64-nix.pkr.hcl
+++ b/amazon-arm64-nix.pkr.hcl
@@ -170,11 +170,6 @@ source "amazon-ebssurrogate" "source" {
build {
sources = ["source.amazon-ebssurrogate.source"]
- provisioner "file" {
- source = "ebssurrogate/files/sources-arm64.cfg"
- destination = "/tmp/sources.list"
- }
-
provisioner "file" {
source = "ebssurrogate/files/ebsnvme-id"
destination = "/tmp/ebsnvme-id"
diff --git a/ansible/playbook.yml b/ansible/playbook.yml
index 5202fe98..d9bb6e1c 100644
--- a/ansible/playbook.yml
+++ b/ansible/playbook.yml
@@ -141,25 +141,12 @@
when: not stage2
import_tasks: tasks/setup-fail2ban.yml
- # Install EC2 instance connect
- # Only for AWS images
- - name: install EC2 instance connect
- when: not qemu
- become: yes
- apt:
- pkg:
- - ec2-instance-connect
- tags:
- - aws-only
-
# Install this at the end to prevent it from kicking in during the apt process, causing conflicts
- name: Install security tools
become: yes
apt:
pkg:
- unattended-upgrades
- update_cache: yes
- cache_valid_time: 3600
- name: Run migrations
when: stage2
diff --git a/ansible/tasks/internal/install-salt.yml b/ansible/tasks/internal/install-salt.yml
index 8df927ef..70ee3a79 100644
--- a/ansible/tasks/internal/install-salt.yml
+++ b/ansible/tasks/internal/install-salt.yml
@@ -29,7 +29,6 @@
- salt-common={{ salt_release }}
- salt-minion={{ salt_release }}
state: present
- update_cache: yes
- name: Pin salt packages at major version
ansible.builtin.copy:
diff --git a/ansible/tasks/setup-postgrest.yml b/ansible/tasks/setup-postgrest.yml
index 33afbc2f..6d217c55 100644
--- a/ansible/tasks/setup-postgrest.yml
+++ b/ansible/tasks/setup-postgrest.yml
@@ -27,7 +27,6 @@
package:
- libnuma-dev
- libpq5
- update_cache: true
state: 'present'
- name: PostgREST - grab the list of installed packages
diff --git a/ansible/tasks/setup-system.yml b/ansible/tasks/setup-system.yml
index fda2627b..d23ab03d 100644
--- a/ansible/tasks/setup-system.yml
+++ b/ansible/tasks/setup-system.yml
@@ -1,15 +1,8 @@
- name: Execute stage1 tasks
when: not stage2
block:
- - name: System - apt update and apt upgrade
- ansible.builtin.apt:
- update_cache: true
- upgrade: true
- # SEE http://archive.vn/DKJjs#parameter-upgrade
-
- name: Install required security updates Ansible dependencies, security tools, and other useful things
ansible.builtin.apt:
- cache_valid_time: 3600
pkg:
- acl
- bwm-ng
@@ -23,7 +16,6 @@
- sysstat
- tzdata
- vim
- update_cache: true
# SEE https://github.com/georchestra/ansible/issues/55#issuecomment-588313638
# Without this, a similar error is faced
diff --git a/ansible/tasks/setup-tmux.yml b/ansible/tasks/setup-tmux.yml
index 93b69ff7..658afdda 100644
--- a/ansible/tasks/setup-tmux.yml
+++ b/ansible/tasks/setup-tmux.yml
@@ -2,7 +2,6 @@
ansible.builtin.apt:
name: 'tmux'
state: 'present'
- update_cache: true
become: true
- name: tmux - Configure tmux alias
diff --git a/ansible/tasks/setup-tuned.yml b/ansible/tasks/setup-tuned.yml
index 1a02becc..1d4fb16c 100644
--- a/ansible/tasks/setup-tuned.yml
+++ b/ansible/tasks/setup-tuned.yml
@@ -6,7 +6,6 @@
name: 'tuned'
policy_rc_d: 101
state: 'present'
- update_cache: true
become: true
- name: 'tuned - Create a tuned profile directory' # noqa: name[casing]
diff --git a/ansible/tasks/test-image.yml b/ansible/tasks/test-image.yml
index 3acd25ce..493eed11 100644
--- a/ansible/tasks/test-image.yml
+++ b/ansible/tasks/test-image.yml
@@ -65,7 +65,6 @@
ansible.builtin.apt:
name: 'python3-psycopg2'
state: 'present'
- update_cache: true
become: true
- name: Reset db stats
diff --git a/ebssurrogate/files/sources-arm64.cfg b/ebssurrogate/files/sources-arm64.cfg
deleted file mode 100644
index eed6c0f1..00000000
--- a/ebssurrogate/files/sources-arm64.cfg
+++ /dev/null
@@ -1,10 +0,0 @@
-deb http://REGION.clouds.ports.ubuntu.com/ubuntu-ports/ noble main restricted
-deb http://REGION.clouds.ports.ubuntu.com/ubuntu-ports/ noble-updates main restricted
-deb http://REGION.clouds.ports.ubuntu.com/ubuntu-ports/ noble universe
-deb http://REGION.clouds.ports.ubuntu.com/ubuntu-ports/ noble-updates universe
-deb http://REGION.clouds.ports.ubuntu.com/ubuntu-ports/ noble multiverse
-deb http://REGION.clouds.ports.ubuntu.com/ubuntu-ports/ noble-updates multiverse
-deb http://REGION.clouds.ports.ubuntu.com/ubuntu-ports/ noble-backports main restricted universe multiverse
-deb http://ports.ubuntu.com/ubuntu-ports noble-security main restricted
-deb http://ports.ubuntu.com/ubuntu-ports noble-security universe
-deb http://ports.ubuntu.com/ubuntu-ports noble-security multiverse
diff --git a/ebssurrogate/files/sources.cfg b/ebssurrogate/files/sources.cfg
deleted file mode 100644
index a27be052..00000000
--- a/ebssurrogate/files/sources.cfg
+++ /dev/null
@@ -1,10 +0,0 @@
-deb http://REGION.ec2.archive.ubuntu.com/ubuntu/ noble main restricted
-deb http://REGION.ec2.archive.ubuntu.com/ubuntu/ noble-updates main restricted
-deb http://REGION.ec2.archive.ubuntu.com/ubuntu/ noble universe
-deb http://REGION.ec2.archive.ubuntu.com/ubuntu/ noble-updates universe
-deb http://REGION.ec2.archive.ubuntu.com/ubuntu/ noble multiverse
-deb http://REGION.ec2.archive.ubuntu.com/ubuntu/ noble-updates multiverse
-deb http://REGION.ec2.archive.ubuntu.com/ubuntu/ noble-backports main restricted universe multiverse
-deb http://security.ubuntu.com/ubuntu noble-security main restricted
-deb http://security.ubuntu.com/ubuntu noble-security universe
-deb http://security.ubuntu.com/ubuntu noble-security multiverse
diff --git a/ebssurrogate/scripts/chroot-bootstrap-nix.sh b/ebssurrogate/scripts/chroot-bootstrap-nix.sh
index 140f6802..1c0e2cfa 100755
--- a/ebssurrogate/scripts/chroot-bootstrap-nix.sh
+++ b/ebssurrogate/scripts/chroot-bootstrap-nix.sh
@@ -8,237 +8,87 @@ set -o errexit
set -o pipefail
set -o xtrace
-# Switch to a different mirror
-function switch_mirror {
- local new_mirror=$1
- local sources_file=/etc/apt/sources.list
-
- echo "Switching to mirror: $new_mirror"
- if [[ $ARCH == amd64 ]]; then
- sed -i "s|http://[^/]*/ubuntu/|http://$new_mirror/ubuntu/|g" "$sources_file"
- else
- sed -i "s|http://[^/]*/ubuntu-ports/|http://$new_mirror/ubuntu-ports/|g" "$sources_file"
- fi
-
- # Show what we're using
- echo "Current sources.list configuration:"
- grep -E '^deb ' "$sources_file" | head -3
-}
-
-# Get list of mirrors to try
-function get_mirror_list {
- local sources_file=/etc/apt/sources.list
- local -a mirrors=()
-
- # Priority order:
- # 1. Country-specific mirror (most reliable)
- # 2. Regional CDN (can be inconsistent)
- # 3. Global fallback
-
- local current_region
- if [[ $ARCH == amd64 ]]; then
- current_region=$(grep -oP '(?<=http://)[^.]+(?=\.ec2\.archive\.ubuntu\.com)' "$sources_file" | head -1 || echo "")
-
- if [[ -n $current_region ]]; then
- mirrors+=("$current_region.ec2.archive.ubuntu.com")
- fi
-
- mirrors+=("archive.ubuntu.com")
- else
- current_region=$(grep -oP '(?<=http://)[^.]+(?=\.clouds\.ports\.ubuntu\.com)' "$sources_file" | head -1 || echo "")
-
- # Singapore country mirror for ap-southeast-1
- if [[ $current_region == "ap-southeast-1" ]]; then
- mirrors+=("sg.ports.ubuntu.com")
- fi
-
- if [[ -n $current_region ]]; then
- mirrors+=("$current_region.clouds.ports.ubuntu.com")
- fi
- mirrors+=("ports.ubuntu.com")
- fi
-
- echo "${mirrors[@]}"
+function setup_apt {
+ export DEBIAN_FRONTEND=noninteractive
}
-# Mirror fallback function for resilient apt-get update
-function apt_update_with_fallback {
- local sources_file=/etc/apt/sources.list
- local -a mirror_list
- readarray mirror_list < <(get_mirror_list)
- local attempt=1
- local max_attempts=${#mirror_list[@]}
-
- for mirror in "${mirror_list[@]}"; do
- echo "========================================="
- echo "Attempting apt-get update with mirror: $mirror"
- echo "Attempt $attempt of $max_attempts"
- echo "========================================="
-
- switch_mirror "$mirror"
-
- # Attempt update with timeout (5 minutes)
- if timeout 300 apt-get "${APT_OPTIONS[@]}" update 2>&1; then
- echo "========================================="
- echo "✓ Successfully updated apt cache using mirror: $mirror"
- echo "========================================="
- return 0
- else
- local ret=$?
- echo "========================================="
- echo "✗ Failed to update using mirror: $mirror"
- echo "Exit code: $ret"
- echo "========================================="
-
- # Clean partial downloads
- apt-get clean
- rm -rf /var/lib/apt/lists/*
-
- # Exponential backoff before next attempt
- if [[ $attempt -lt $max_attempts ]]; then
- local sleep_time=$((attempt * 5))
- echo "Waiting $sleep_time seconds before trying next mirror..."
- sleep $sleep_time
- fi
- fi
-
- attempt=$((attempt + 1))
- done
-
- echo "========================================="
- echo "ERROR: All mirror tiers failed after $max_attempts attempts"
- echo "========================================="
- return 1
+function cleanup_apt {
+ apt-get clean
+ apt-get autoremove --purge --yes
}
-# Wrapper for apt-get install with mirror fallback on 404 errors
-function apt_install_with_fallback {
- local -a mirror_list
- readarray mirror_list < <(get_mirror_list)
- local attempt=1
- local max_attempts=${#mirror_list[@]}
-
- for mirror in "${mirror_list[@]}"; do
- echo "========================================="
- echo "Attempting apt-get install with mirror: $mirror"
- echo "Attempt $attempt of $max_attempts"
- echo "========================================="
-
- switch_mirror "$mirror"
-
- # Re-run apt-get update to get package lists from new mirror
- if ! timeout 300 apt-get "${APT_OPTIONS[@]}" update 2>&1; then
- echo "Warning: apt-get update failed for mirror $mirror, trying next..."
- attempt=$((attempt + 1))
- continue
- fi
-
- # Run apt-get install directly (no output capture to avoid buffering/timeout issues)
- if apt-get "$@"; then
- echo "========================================="
- echo "✓ Successfully installed packages using mirror: ${mirror}"
- echo "========================================="
- return 0
- else
- local ret=$?
- # On failure, check if it's a mirror issue worth retrying
- echo "========================================="
- echo "✗ apt-get failed with exit code: $ret"
- echo "========================================="
- fi
-
- # Clean apt cache before potential retry
- apt-get clean
-
- if ((attempt < max_attempts)); then
- local sleep_time=$((attempt * 5))
- echo "Waiting $sleep_time seconds before trying next mirror..."
- sleep $sleep_time
- fi
-
- attempt=$((attempt + 1))
- done
-
- echo "========================================="
- echo "ERROR: All mirror tiers failed for apt-get install after $max_attempts attempts"
- echo "========================================="
- return 1
+function update_and_upgrade_apt {
+ apt-get update --yes
+ apt-get dist-upgrade --yes
}
-function update_install_packages {
- # Update APT with new sources (using fallback mechanism)
- cat /etc/apt/sources.list
- if ! apt_update_with_fallback; then
- echo "FATAL: Failed to update package lists with any mirror tier"
- exit 1
- fi
- apt-get "${APT_OPTIONS[@]}" --yes dist-upgrade
-
- # Do not configure grub during package install
+function install_initial_packages {
+ local packages=(
+ e2fsprogs
+ initramfs-tools
+ )
+ # Install various packages needed for a booting system (with mirror fallback)
if [[ $ARCH == amd64 ]]; then
+ # Do not configure grub during package install
echo 'grub-pc grub-pc/install_devices_empty select true' | debconf-set-selections
echo 'grub-pc grub-pc/install_devices select' | debconf-set-selections
- # Install various packages needed for a booting system (with mirror fallback)
- if ! apt_install_with_fallback install -y linux-aws grub-pc e2fsprogs; then
- echo "FATAL: Failed to install boot packages"
- exit 1
- fi
+ packages+=(
+ linux-aws
+ grub-pc
+ )
else
- if ! apt_install_with_fallback install -y e2fsprogs; then
- echo "FATAL: Failed to install e2fsprogs"
- exit 1
- fi
+ packages+=(
+ dosfstools
+ )
fi
- # Install standard packages (with mirror fallback)
- # Note: ec2-hibinit-agent, ec2-instance-connect, hibagent moved to stage 2
- # because their post-install scripts try to access EC2 metadata service
- # which doesn't work in a chroot and causes long hangs
- if ! apt_install_with_fallback install -y \
- bzip2 \
- sudo \
- wget \
- cloud-init \
- acpid \
- ncurses-term \
- ssh-import-id; then
+
+ packages+=(
+ acpid
+ apparmor
+ apparmor-utils
+ auditd
+ bzip2
+ cloud-init
+ ec2-hibinit-agent
+ ec2-instance-connect
+ hibagent
+ ncurses-term
+ ssh-import-id
+ sudo
+ wget
+ )
+ if ! apt-get install --yes "${packages[@]}"; then
echo "FATAL: Failed to install standard packages"
exit 1
fi
- # apt upgrade
- apt-get upgrade -y
-
- # Install OpenSSH and other packages
- add-apt-repository --yes universe
- if ! apt_update_with_fallback; then
- echo "FATAL: Failed to update package lists after adding universe repository"
- exit 1
- fi
- if ! apt_install_with_fallback install -y --no-install-recommends \
- openssh-server \
- git \
- ufw \
- cron \
- logrotate \
- fail2ban \
- locales \
- at \
- less \
- python3-systemd; then
- echo "FATAL: Failed to install universe packages"
- exit 1
+ packages=(
+ at
+ cron
+ fail2ban
+ git
+ less
+ locales
+ logrotate
+ openssh-server
+ python3-systemd
+ ufw
+ )
+ if [[ $ARCH == arm64 ]]; then
+ packages+=(
+ cloud-guest-utils
+ fdisk
+ grub-efi-arm64
+ efibootmgr
+ # linux-aws is installed separately between amd64 and arm64 because of --no-install-recommends is needed for arm64
+ # otherwise we bring in grub-efi-arm64-signed which fails to boot
+ linux-aws
+ )
fi
- if [[ $ARCH == arm64 ]]; then
- if ! apt_install_with_fallback "${APT_OPTIONS[@]}" --yes install linux-aws initramfs-tools dosfstools; then
- echo "FATAL: Failed to install arm64 boot packages"
- exit 1
- fi
- else
- if ! apt_install_with_fallback "${APT_OPTIONS[@]}" --yes install initramfs-tools; then
- echo "FATAL: Failed to install amd64 boot packages"
- exit 1
- fi
+ if ! apt-get install --yes --no-install-recommends "${packages[@]}"; then
+ echo "FATAL: Failed to install extra packages"
+ exit 1
fi
}
@@ -268,11 +118,6 @@ function setup_postgesql_env {
}
function setup_apparmor {
- if ! apt_install_with_fallback install -y apparmor apparmor-utils auditd; then
- echo "FATAL: Failed to install apparmor packages"
- exit 1
- fi
-
# Copy apparmor profiles
cp -rv /tmp/apparmor_profiles/* /etc/apparmor.d/
}
@@ -290,10 +135,6 @@ function setup_grub {
EOF
if [[ $ARCH == arm64 ]]; then
- if ! apt_install_with_fallback "${APT_OPTIONS[@]}" --yes install cloud-guest-utils fdisk grub-efi-arm64 efibootmgr; then
- echo "FATAL: Failed to install grub packages for arm64"
- exit 1
- fi
rm -rf /etc/grub.d/30_os-prober
sleep 1
fi
@@ -364,28 +205,18 @@ function disable_services {
chmod +x /usr/sbin/policy-rc.d
}
-# Clear apt caches
-function cleanup_cache {
- apt-get clean
-}
-
# Remove policy-rc.d so services start normally on boot
function enable_services {
rm -f /usr/sbin/policy-rc.d
}
-export DEBIAN_FRONTEND=noninteractive
-APT_OPTIONS=(
- -oAPT::Install-Recommends=false
- -oAPT::Install-Suggests=false
- -oAcquire::Languages=none
-)
-
ARCH=$(dpkg --print-architecture)
: "${ARCH:?Failed to detect architecture}"
disable_services
-update_install_packages
+setup_apt
+update_and_upgrade_apt
+install_initial_packages
setup_locale
setup_postgesql_env
setup_grub
@@ -396,5 +227,5 @@ set_default_target
setup_eth0_interface
disable_sshd_passwd_auth
disable_fsck
-cleanup_cache
+cleanup_apt
enable_services
diff --git a/ebssurrogate/scripts/cleanup-qemu.sh b/ebssurrogate/scripts/cleanup-qemu.sh
index e79cc445..9c79a981 100755
--- a/ebssurrogate/scripts/cleanup-qemu.sh
+++ b/ebssurrogate/scripts/cleanup-qemu.sh
@@ -14,64 +14,104 @@ if [[ ! -d /tmp ]]; then
fi
chmod 1777 /tmp
-if [ -n "$(command -v yum)" ]; then
- yum update -y
- yum clean all
-elif [ -n "$(command -v apt-get)" ]; then
- # Cleanup more packages
- apt-get -y remove --purge \
- automake \
- autoconf \
- autotools-dev \
- cmake-data \
- cpp-9 \
- cpp-10 \
- gcc-9 \
- gcc-10 \
- git \
- git-man \
- ansible \
- libicu-dev \
- libcgal-dev \
- libgcc-9-dev \
- ansible \
- snapd
-
- if [[ $(uname -m) == aarch64 ]]; then
- apt-get -y remove --purge libgcc-8-dev
- fi
-
- # add-apt-repository --yes --remove ppa:ansible/ansible
-
- apt-mark manual libevent-2.1-7t64
-
- apt-get remove -y --purge ansible-core apport appstream bash-completion bcache-tools bind9-dnsutils bind9-host bind9-libs bolt btrfs-progs byobu command-not-found console-setup distro-info eject fonts-ubuntu-console friendly-recovery ftp fwupd gawk gdisk keyboard-configuration libvolume-key1 libssl-dev lvm2 lxd-agent-loader man-db mdadm modemmanager mtd-utils nano netcat-openbsd nfs-common ntfs-3g parted pastebinit screen strace thin-provisioning-tools tmux usb-modeswitch vim vim-runtime wget whiptail xfsprogs
-
- apt remove -y --purge libc6-dev linux-libc-dev libevent-dev libpcre3-dev libsystemd-dev packagekit multipath-tools unattended-upgrades plymouth gnupg open-vm-tools xauth lxd-installer publicsuffix libclang-cpp18 python3-twisted python-babel-localedata libicu74 python3-pygments fonts-dejavu* python3-botocore
-
- apt-get remove -y --purge linux-headers*
-
- # remove old kernels
- # CURRENT_KERNEL="$(uname -r | sed 's/-generic//')"
- # INSTALLED_KERNELS=$(dpkg -l | awk '{print $2}' | grep -Eo 'linux-(image|headers|modules|tools)-[0-9]+' | sed -E 's/linux-(image|modules|tools)-//' | sort -Vu)
- # REMOVE_KERNELS=$(echo "$INSTALLED_KERNELS" | grep -v -e "$CURRENT_KERNEL")
- # for VER in $REMOVE_KERNELS; do
- # for PREFIX in linux-image linux-modules linux-tools; do
- # for PKG in $(dpkg -l | awk '{print $2}' | grep "^$PREFIX-$VER"); do
- # apt-get purge -y "$PKG"
- # done
- # done
- # done
- # update-grub
-
- apt-get -y autoremove
- apt-get -y autoclean
-
- apt-get -y update
- apt-get -y upgrade
-
+# Cleanup more packages
+packages=(
+ ansible
+ ansible-core
+ apport
+ appstream
+ autoconf
+ automake
+ autotools-dev
+ bash-completion
+ bcache-tools
+ bind9-dnsutils
+ bind9-host
+ bind9-libs
+ bolt
+ btrfs-progs
+ byobu
+ cmake-data
+ command-not-found
+ console-setup
+ cpp-10
+ cpp-9
+ distro-info
+ eject
+ 'fonts-dejavu*'
+ fonts-ubuntu-console
+ friendly-recovery
+ ftp
+ fwupd
+ gawk
+ gcc-10
+ gcc-9
+ gdisk
+ git
+ git-man
+ gnupg
+ keyboard-configuration
+ libc6-dev
+ libcgal-dev
+ libclang-cpp18
+ libevent-dev
+ libgcc-9-dev
+ libicu-dev
+ libicu74
+ libpcre3-dev
+ libssl-dev
+ libsystemd-dev
+ libvolume-key1
+ 'linux-headers*'
+ linux-libc-dev
+ lvm2
+ lxd-agent-loader
+ lxd-installer
+ man-db
+ mdadm
+ modemmanager
+ mtd-utils
+ multipath-tools
+ nano
+ netcat-openbsd
+ nfs-common
+ ntfs-3g
+ open-vm-tools
+ packagekit
+ parted
+ pastebinit
+ plymouth
+ publicsuffix
+ python-babel-localedata
+ python3-botocore
+ python3-pygments
+ python3-twisted
+ screen
+ snapd
+ strace
+ thin-provisioning-tools
+ tmux
+ unattended-upgrades
+ usb-modeswitch
+ vim
+ vim-runtime
+ wget
+ whiptail
+ xauth
+ xfsprogs
+)
+
+if [[ $(uname -m) == aarch64 ]]; then
+ packages+=(libgcc-8-dev)
fi
+apt-mark manual libevent-2.1-7t64
+apt-get --yes remove --purge "${packages[@]}"
+apt-get --yes autoremove
+apt-get --yes autoclean
+apt-get --yes update
+apt-get --yes upgrade
+
systemctl set-default multi-user.target
systemctl disable getty@tty1.service
systemctl mask getty@tty1.service
diff --git a/ebssurrogate/scripts/cleanup.sh b/ebssurrogate/scripts/cleanup.sh
index 8eab3688..f9a103fb 100755
--- a/ebssurrogate/scripts/cleanup.sh
+++ b/ebssurrogate/scripts/cleanup.sh
@@ -14,35 +14,31 @@ if [[ ! -d /tmp ]]; then
fi
chmod 1777 /tmp
-if [ -n "$(command -v yum)" ]; then
- yum update -y
- yum clean all
-elif [ -n "$(command -v apt-get)" ]; then
- # Cleanup more packages
- apt-get -y remove --purge \
- automake \
- autoconf \
- autotools-dev \
- cmake-data \
- cpp-9 \
- cpp-10 \
- gcc-9 \
- gcc-10 \
- git \
- git-man \
- ansible \
- libicu-dev \
- libcgal-dev \
- libgcc-9-dev \
- ansible
+# Cleanup more packages
+packages=(
+ ansible
+ ansible
+ autoconf
+ automake
+ autotools-dev
+ cmake-data
+ cpp-10
+ cpp-9
+ gcc-10
+ gcc-9
+ git
+ git-man
+ libcgal-dev
+ libgcc-9-dev
+ libicu-dev
+)
- # add-apt-repository --yes --remove ppa:ansible/ansible
+apt-get --yes remove --purge "${packages[@]}"
+apt-get --yes autoremove
+apt-get --yes autoclean
+apt-get --yes update
+apt-get --yes upgrade
- apt-get -y update
- apt-get -y upgrade
- apt-get -y autoremove
- apt-get -y autoclean
-fi
rm -rf /tmp/* /var/tmp/*
history -c
cat /dev/null >/root/.bash_history
diff --git a/ebssurrogate/scripts/nix-provision.sh b/ebssurrogate/scripts/nix-provision.sh
index 8e33ec63..41a81ea1 100755
--- a/ebssurrogate/scripts/nix-provision.sh
+++ b/ebssurrogate/scripts/nix-provision.sh
@@ -7,24 +7,23 @@ set -o xtrace
exec 1>&2
-function install_packages {
- # Setup Ansible on host VM
- apt-get update && apt-get install -y software-properties-common
+function setup_apt {
+ export DEBIAN_FRONTEND=noninteractive
+}
- # Install EC2-specific packages that were deferred from stage 1
- # These packages have post-install scripts that need EC2 metadata service access
- # which only works on a real running EC2 instance (not in chroot)
- apt-get install -y ec2-hibinit-agent ec2-instance-connect hibagent
+function cleanup_apt {
+ apt-get clean
+ apt-get autoremove --purge --yes
+ rm -rf /var/lib/apt/lists/*
+}
- # Manually add GPG key with explicit keyserver
- apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 93C4A3FD7BB9C367
+function update_and_upgrade_apt {
+ apt-get update --yes
+ apt-get upgrade --yes
+}
- # Add repository and install
- # TODO (darora): temporarily disabling while Launchpad is under ddos attack and very frequently timing out
- # sudo add-apt-repository --yes ppa:ansible/ansible
- # sudo apt-get update
+function install_packages {
apt-get install -y ansible
-
ansible-galaxy collection install community.general
}
@@ -68,8 +67,7 @@ function execute_stage2_playbook {
}
function cleanup_packages {
- apt-get -y remove --purge ansible
- # sudo add-apt-repository --yes --remove ppa:ansible/ansible
+ apt-get remove --purge --yes ansible
}
function cleanup_nix {
@@ -77,15 +75,26 @@ function cleanup_nix {
nix-store --optimise -v
}
+function report_packages {
+ # shellcheck disable=SC2016
+ dpkg-query -W -f='${Package}\t${Version}\t${Architecture}\n' | LC_COLLATE=C.UTF-8 sort
+ find /nix/store -maxdepth 1 | LC_COLLATE=C.UTF-8 sort -t- -k2
+}
+
function report_disk_usage {
read -r dub _ < <(du -sx -B1 /)
read -r duh _ < <(du -sx -h /)
printf '::notice::disk_usage bytes=%s human=%s\n' "$dub" "$duh" | tee -a /tmp/ansible.log
}
+setup_apt
+update_and_upgrade_apt
install_packages
install_nix
execute_stage2_playbook
cleanup_packages
cleanup_nix
+update_and_upgrade_apt
+cleanup_apt
+report_packages
report_disk_usage
diff --git a/ebssurrogate/scripts/qemu-bootstrap-nix.sh b/ebssurrogate/scripts/qemu-bootstrap-nix.sh
index bf1005e0..6a9d4df1 100755
--- a/ebssurrogate/scripts/qemu-bootstrap-nix.sh
+++ b/ebssurrogate/scripts/qemu-bootstrap-nix.sh
@@ -8,6 +8,51 @@ set -o xtrace
# stage1 things #
#################
+function setup_apt {
+ export DEBIAN_FRONTEND=noninteractive
+
+ # This function assumes deb822 formatted sources are in use, which is the case in both qemu and aws images
+ # In aws cloud-init creates a sources file with regional mirrors for "default" Suites but keeps ubuntu for security suite
+ # So we grab the first (only) URI from security and append it to non-security's URI
+ # This ends up giving us fastest mirror for installs and falls back to ubuntu if there's an issue
+ #
+ # Note: Ubuntu amd64 mirrors have different hostnames for security vs non but aarch64 are the same, hence the amd64 specific line
+
+ # ensure deb822 format sources are in use
+ tail -n+1 /etc/apt/sources.list /etc/apt/sources.list.d/ubuntu.sources >&2
+
+ local sources defmirror ubumirror
+ sources=$(grep -e '^URIs\s*:' -e '^Suites\s*:' /etc/apt/sources.list.d/ubuntu.sources)
+ defmirror=$(grep -B1 "$CODENAME-updates" <<<"$sources" | awk '/URIs/ {print $2}')
+ ubumirror=$(grep -B1 "$CODENAME-security" <<<"$sources" | awk '/URIs/ {print $2}')
+ if [[ $ARCH == amd64 ]]; then
+ # amd64 hosts use security.ubuntu.com for security but archive.ubuntu.com for everything else
+ ubumirror=${ubumirror/security/archive}
+ fi
+
+ if [[ $ubumirror == "$defmirror" ]]; then
+ # Only using one mirror so nothing to add as fallback, not running in AWS maybe?
+ return
+ fi
+
+ if grep -q "^URIs:.*$defmirror.*$ubumirror" /etc/apt/sources.list.d/ubuntu.sources; then
+ echo "Ubuntu upstream is already a fallback, this is unexpected and needs source changes" >&2
+ exit 1
+ fi
+ sed -i "s|$defmirror|& $ubumirror|" /etc/apt/sources.list.d/ubuntu.sources
+}
+
+function cleanup_apt {
+ apt-get clean
+ apt-get autoremove --purge --yes
+ rm -rf /var/lib/apt/lists/*
+}
+
+function update_and_upgrade_apt {
+ apt-get update --yes
+ apt-get upgrade --yes
+}
+
function waitfor_boot_finished {
# Wait for cloudinit on the surrogate to complete before making progress
while [[ ! -f /var/lib/cloud/instance/boot-finished ]]; do
@@ -17,23 +62,21 @@ function waitfor_boot_finished {
}
function install_packages {
- apt-get update
- apt-get install -y \
- arptables \
- e2fsprogs \
- ebtables \
- gpg \
- iptables \
- less \
- locales \
- logrotate \
- nfs-common \
- software-properties-common \
- ufw \
- ;
- # TODO (darora): temporarily disabling while Launchpad is under ddos attack and very frequently timing out
- # add-apt-repository --yes --update ppa:ansible/ansible &&
- apt-get install ansible -y
+ packages=(
+ ansible
+ arptables
+ e2fsprogs
+ ebtables
+ gpg
+ iptables
+ less
+ locales
+ logrotate
+ nfs-common
+ software-properties-common
+ ufw
+ )
+ apt-get install --yes "${packages[@]}"
ansible-galaxy collection install community.general
}
@@ -122,8 +165,7 @@ function execute_stage2_playbook {
function clean_legacy_things {
# removes things that are bundled for legacy reasons, but we can start without for our newer artifacts
apt-mark auto zlib1g* # TODO (darora): need to make sure that there aren't other things that still need this
- apt-get -y purge kong
- apt-get autoremove -y
+ apt-get purge --yes kong
}
function clean_system {
@@ -163,7 +205,6 @@ function clean_system {
mkdir /var/log/audit
# unwanted files
- rm -rf /var/lib/apt/lists/*
rm -rf /root/.cache
rm -rf /root/.vpython*
rm -rf /root/go
@@ -191,13 +232,31 @@ function clean_nix {
nix-store --optimise -v
}
+function report_packages {
+ # shellcheck disable=SC2016
+ dpkg-query -W -f='${Package}\t${Version}\t${Architecture}\n' | LC_COLLATE=C.UTF-8 sort
+ find /nix/store -maxdepth 1 | LC_COLLATE=C.UTF-8 sort -t- -k2
+}
+
+function report_disk_usage {
+ read -r dub _ < <(du -sx -B1 /)
+ read -r duh _ < <(du -sx -h /)
+ printf '::notice::disk_usage bytes=%s human=%s\n' "$dub" "$duh" | tee -a /tmp/ansible.log
+}
+
#################
# stage1 things #
#################
-export DEBIAN_FRONTEND=noninteractive
+ARCH=$(dpkg --print-architecture)
+: "${ARCH:?Failed to detect architecture}"
+# shellcheck source=/dev/null
+CODENAME=$(source /etc/os-release && echo "$VERSION_CODENAME")
+: "${CODENAME:?Failed to detect OS codename}"
waitfor_boot_finished
+setup_apt
+update_and_upgrade_apt
install_packages
setup_postgesql_env
setup_locale
@@ -212,3 +271,6 @@ execute_stage2_playbook
clean_legacy_things
clean_system
clean_nix
+cleanup_apt
+report_packages
+report_disk_usage
diff --git a/ebssurrogate/scripts/surrogate-bootstrap-nix.sh b/ebssurrogate/scripts/surrogate-bootstrap-nix.sh
index a24072b3..81fae310 100755
--- a/ebssurrogate/scripts/surrogate-bootstrap-nix.sh
+++ b/ebssurrogate/scripts/surrogate-bootstrap-nix.sh
@@ -12,87 +12,43 @@ set -o xtrace
exec 1>&2
-# Mirror fallback function for resilient apt-get update
-function apt_update_with_fallback {
- local sources_file=/etc/apt/sources.list
- local max_attempts=2
- local attempt=1
-
- # Get EC2 region if not already set
- if [[ -z $REGION ]]; then
- REGION=$(curl --silent --fail http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -E 's|[a-z]+$||g' || echo "")
- fi
+function setup_apt {
+ export DEBIAN_FRONTEND=noninteractive
+
+ # This function assumes deb822 formatted sources are in use, which is the case in both qemu and aws images
+ # In aws cloud-init creates a sources file with regional mirrors for "default" Suites but keeps ubuntu for security suite
+ # So we grab the first (only) URI from security and append it to non-security's URI
+ # This ends up giving us fastest mirror for installs and falls back to ubuntu if there's an issue
+ #
+ # Note: Ubuntu amd64 mirrors have different hostnames for security vs non but aarch64 are the same, hence the amd64 specific line
- # Define mirror tiers (in priority order)
- local -a mirror_tiers=()
+ # ensure deb822 format sources are in use
+ tail -n+1 /etc/apt/sources.list /etc/apt/sources.list.d/ubuntu.sources >&2
+
+ local sources defmirror ubumirror
+ sources=$(grep -e '^URIs\s*:' -e '^Suites\s*:' /etc/apt/sources.list.d/ubuntu.sources)
+ defmirror=$(grep -B1 "$CODENAME-updates" <<<"$sources" | awk '/URIs/ {print $2}')
+ ubumirror=$(grep -B1 "$CODENAME-security" <<<"$sources" | awk '/URIs/ {print $2}')
if [[ $ARCH == amd64 ]]; then
- if [[ -n $REGION ]]; then
- mirror_tiers+=("$REGION.ec2.archive.ubuntu.com")
- fi
- mirror_tiers+=("archive.ubuntu.com")
- else
- if [[ -n $REGION ]]; then
- mirror_tiers+=("$REGION.clouds.ports.ubuntu.com")
- fi
- mirror_tiers+=("ports.ubuntu.com")
+ # amd64 hosts use security.ubuntu.com for security but archive.ubuntu.com for everything else
+ ubumirror=${ubumirror/security/archive}
fi
- # If we couldn't get REGION, skip tier 1
- if [[ -z $REGION ]]; then
- echo "Warning: Could not determine EC2 region, skipping regional mirror"
- mirror_tiers=("${mirror_tiers[@]:1}") # Remove first element
+ if [[ $ubumirror == "$defmirror" ]]; then
+ # Only using one mirror so nothing to add as fallback, not running in AWS maybe?
+ return
fi
- for mirror in "${mirror_tiers[@]}"; do
- echo "========================================="
- echo "Attempting apt-get update with mirror: $mirror"
- echo "Attempt $attempt of $max_attempts"
- echo "========================================="
-
- # Update sources.list to use current mirror
- if [[ $ARCH == amd64 ]]; then
- sed -i "s|http://[^/]*/ubuntu/|http://$mirror/ubuntu/|g" "$sources_file"
- else
- sed -i "s|http://[^/]*/ubuntu-ports/|http://$mirror/ubuntu-ports/|g" "$sources_file"
- sed -i "s|http://ports.ubuntu.com/ubuntu-ports|http://$mirror/ubuntu-ports|g" "$sources_file"
- fi
-
- # Show what we're using
- echo "Current sources.list configuration:"
- grep -E '^deb ' "$sources_file" | head -3
-
- # Attempt update with timeout (5 minutes)
- if timeout 300 apt-get update 2>&1; then
- echo "========================================="
- echo "✓ Successfully updated apt cache using mirror: $mirror"
- echo "========================================="
- return 0
- else
- local exit_code=$?
- echo "========================================="
- echo "✗ Failed to update using mirror: $mirror"
- echo "Exit code: $exit_code"
- echo "========================================="
-
- # Clean partial downloads
- apt-get clean
- rm -rf /var/lib/apt/lists/*
-
- # Exponential backoff before next attempt
- if ((attempt < max_attempts)); then
- local sleep_time=$((attempt * 5))
- echo "Waiting $sleep_time seconds before trying next mirror..."
- sleep $sleep_time
- fi
- fi
-
- attempt=$((attempt + 1))
- done
+ if grep -q "^URIs:.*$defmirror.*$ubumirror" /etc/apt/sources.list.d/ubuntu.sources; then
+ echo "Ubuntu upstream is already a fallback, this is unexpected and needs source changes" >&2
+ exit 1
+ fi
+ sed -i "s|$defmirror|& $ubumirror|" /etc/apt/sources.list.d/ubuntu.sources
+}
- echo "========================================="
- echo "ERROR: All mirror tiers failed after $max_attempts attempts"
- echo "========================================="
- return 1
+function update_and_upgrade_apt {
+ apt-get update --yes
+ apt-get upgrade --yes
}
function waitfor_boot_finished {
@@ -103,29 +59,15 @@ function waitfor_boot_finished {
}
function install_packages {
- # Setup Ansible on host VM
- if ! apt_update_with_fallback; then
- echo "FATAL: Failed to update package lists on host VM"
- exit 1
- fi
-
- apt-get install software-properties-common -y
- # TODO (darora): temporarily disabling while Launchpad is under ddos attack and very frequently timing out
- # add-apt-repository --yes --update ppa:ansible/ansible
-
- if ! apt_update_with_fallback; then
- echo "FATAL: Failed to update package lists after adding Ansible PPA"
- exit 1
- fi
-
- apt-get install ansible -y
- ansible-galaxy collection install community.general
-
- apt-get install -y \
- gdisk \
- e2fsprogs \
- debootstrap \
+ packages=(
+ ansible
+ debootstrap
+ e2fsprogs
+ gdisk
nvme-cli
+ )
+ apt-get install --yes "${packages[@]}"
+ ansible-galaxy collection install community.general
}
# Partition the new root EBS volume
@@ -226,9 +168,6 @@ function create_fstab {
}
function setup_chroot_environment {
- local UBUNTU_VERSION
- UBUNTU_VERSION=$(lsb_release -cs) # 'noble' for Ubuntu 24.04
-
# sometimes debootstrap will get stuck on a download for a long time
# the default read timeout in wget is 900s, which can cause a ~15min increase in build time
# this forces the process to fail-fast and retry
@@ -238,19 +177,14 @@ function setup_chroot_environment {
tries = 5
EOF
- # Update ec2-region
- local REGION
- REGION=$(curl --silent --fail http://169.254.169.254/latest/meta-data/placement/availability-zone | sed -E 's|[a-z]+$||g')
-
- # Bootstrap Ubuntu into /mnt using the regional mirror (avoids global mirror stalls)
- if [[ $ARCH == amd64 ]]; then
- debootstrap --arch "$ARCH" --variant=minbase "$UBUNTU_VERSION" /mnt "http://$REGION.ec2.archive.ubuntu.com/ubuntu"
- else
- debootstrap --arch "$ARCH" --variant=minbase "$UBUNTU_VERSION" /mnt "http://$REGION.clouds.ports.ubuntu.com/ubuntu-ports"
- fi
+ # Use the preferred mirror (if multiple), which is the first URI/preferred
+ local mirror
+ mirror=$(grep -B1 "$CODENAME-updates" /etc/apt/sources.list.d/ubuntu.sources | awk '/URIs/ {print $2}')
+ debootstrap --arch "$ARCH" --variant=minbase "$CODENAME" /mnt "$mirror"
- sed -i "s/REGION/$REGION/g" /tmp/sources.list
- cp /tmp/sources.list /mnt/etc/apt/sources.list
+ # Copy our files in since they are updated with all the mirrors!
+ cp -a /etc/apt/sources.list /mnt/etc/apt/sources.list
+ cp -a /etc/apt/sources.list.d/ubuntu.sources /mnt/etc/apt/sources.list.d/ubuntu.sources
create_fstab
@@ -379,6 +313,11 @@ function clean_system {
rm -rf /mnt/usr/share/doc
}
+function report_packages {
+ # shellcheck disable=SC2016
+ chroot /mnt dpkg-query -W -f='${Package}\t${Version}\t${Architecture}\n' | LC_COLLATE=C.UTF-8 sort
+}
+
# Unmount bind mounts
function umount_reset_mappings {
umount -l /mnt/dev
@@ -399,12 +338,15 @@ function umount_reset_mappings {
done
}
-export DEBIAN_FRONTEND=noninteractive
-
ARCH=$(dpkg --print-architecture)
: "${ARCH:?Failed to detect architecture}"
+# shellcheck source=/dev/null
+CODENAME=$(source /etc/os-release && echo "$VERSION_CODENAME")
+: "${CODENAME:?Failed to detect OS codename}"
waitfor_boot_finished
+setup_apt
+update_and_upgrade_apt
install_packages
device_partition_mappings
format_and_mount_rootfs
@@ -413,4 +355,5 @@ setup_chroot_environment
execute_playbook
update_systemd_services
clean_system
+report_packages
umount_reset_mappings