Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tS3vp-00Dgax-3k for pgsql-bugs@arkaria.postgresql.org; Mon, 30 Dec 2024 00:46:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1tS3vn-008kCo-23 for pgsql-bugs@arkaria.postgresql.org; Mon, 30 Dec 2024 00:46:50 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tS3lc-008iDm-80 for pgsql-bugs@lists.postgresql.org; Mon, 30 Dec 2024 00:36:20 +0000 Received: from 02d.relay.hey.com ([204.62.114.231]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1tS3lY-002GNu-Sy for pgsql-bugs@lists.postgresql.org; Mon, 30 Dec 2024 00:36:19 +0000 Received: from hey.com (bigip-vip.rw-ash-int.37signals.com [10.20.0.24]) by 02.relay.hey.com (Postfix) with ESMTP id A899C81591 for ; Mon, 30 Dec 2024 00:36:14 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=hey.com; s=heymail; t=1735518975; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=cuiJmSNHlPSH9vPnO0JgedDSClYNI/ek+wRX2mWX/3Y=; b=U2YbBjuPInkLvNNpsc+82daux+DL+PQtDeoQ0hOdAjj2TOrU+vAjgX+w1gaBWgSVybZEe9 dVsmzP3JZYlcLK7eOWiT57eGowyzz9SLPvzbZFyiK81Q+0ZNyigfJ5X2Z37ns6u3gFqg0e H74yGsi0TbaGwAWgKYSCcRtMzm9rlsuEZMR9tItVBNAmoLgH9U7ibt6nyx3VNV/lOjuyCm 4B8Ah6YdSt/aPCc4fStuUDlcctmNJFTEw6I+vPFEPDze42SW+2CRlvtM6SnFOYKRPmUYZ1 W1hZv6KPXlVgWTNhSJWeBUiG76HxQjk+ePDMl6aXMOM5rd3gHor7lviEWylrrA== Date: Sun, 29 Dec 2024 19:36:13 -0500 From: Michael Misiewicz To: pgsql-bugs@lists.postgresql.org Message-ID: <63b5aaafb0f89a714be6415d33844de21d682770@hey.com> Subject: pg_upgrade cannot create btrfs clones on linux kernel 6.8.0 Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="--==_mimepart_6771eafedfa51_4f1f468515556"; charset=UTF-8 Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk ----==_mimepart_6771eafedfa51_4f1f468515556 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Hello all,=C2=A0 I am attempting to upgrade a postgres cluster from 16 to 17. I'm running ubuntu 24.04 on x86_64, with postgres installed from the postgres apt repositories.=C2=A0 I'm invoking pg_upgrade like so: /usr/lib/postgresql/17/bin/pg_upgrade --clone -d /var/lib/postgresql/16/main -b /usr/lib/postgresql/16/bin -B /usr/lib/postgresql/17/bin -D /var/lib/postgresql/17/main -o '-c config_file=3D/etc/postgresql/16/main/postgresql.conf' -O '-c config_file=3D/etc/postgresql/17/main/postgresql.conf' and it fails with: could not clone file between old and new data directories: Invalid argument Failure, exiting I ran the following strace command to find out where the problem is happening: sudo -u postgres strace -f -e trace=3Dcopy_file_range,clone,ioctl,%file -= s 2000 -v /usr/lib/postgresql/17/bin/pg_upgrade --clone -d /var/lib/postgresql/16/main -b /usr/lib/postgresql/16/bin -B /usr/lib/postgresql/17/bin -D /var/lib/postgresql/17/main -o '-c config_file=3D/etc/postgresql/16/main/postgresql.conf' -O '-c config_file=3D/etc/postgresql/17/main/postgresql.conf' 2>/tmp/pg_strace.log And grepping through the log I found what looks like the problem here: -- [pid 1153372] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) =3D 8 [pid 1153372] openat(AT_FDCWD, "/lib/x86_64- linux-gnu/libpq.so.5", O_RDONLY|O_CLOEXEC) =3D 8 [pid 1153261] unlink("/var/lib/postgresql/17/main/PG_VERSION.clonetest") =3D 0 [pid 1153261] openat(AT_FDCWD, "/var/lib/postgresql/16/main/PG_VERSION", O_RDONLY) =3D 4 [pid 1153261] openat(AT_FDCWD, "/var/lib/postgresql/17/main/PG_VERSION.clonetest", O_RDWR|O_CREAT|O_EXCL, 0600) =3D 5 [pid 1153261] ioctl(5, BTRFS_IOC_CLONE= or FICLONE, 4) =3D -1 EINVAL (Invalid argument) Which I believe can be traced back to this commit 3a769d8239afdc003c91a56d2d8d5adfadacda5d .=C2=A0 I have validated that my filesystem supports reflinks: $ dd of=3Dtempfile if=3D/dev/random bs=3D1M count=3D24 24+0 records in 24= +0 records out 25165824 bytes (25 MB, 24 MiB) copied, 0.0806355 s, 312 MB/s $ cp --reflink=3Dalways tempfile tempfile.reflink $ btrfs filesystem du -= s tempfile tempfile.reflink Total Exclusive Set shared Filename 24.00MiB 0.00B 24.00MiB tempfile 24.00MiB 0.00B 24.00MiB tempfile.reflink And there's nothing interesting in `dmesg`.=C2=A0 I ran this same command today on a macOS system (using apfs) and it worked great. I have no idea how to fix this problem and I'm curious if anyone has any pointers.=C2=A0 Thanks, Michael ----==_mimepart_6771eafedfa51_4f1f468515556 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hello all,=C2=A0

I am attempting to upgrade a postgres clus= ter from 16 to 17. I'm running ubuntu 24.04 on x86_64, with postgres = installed from the postgres apt repositories.=C2=A0

I'm invoki= ng pg_upgrade like so:
/usr/lib/postgresql/17/bin/pg_upgrade --=
clone -d /var/lib/postgresql/16/main -b /usr/lib/postgresql/16/bin -B  /u=
sr/lib/postgresql/17/bin -D /var/lib/postgresql/17/main -o '-c config=
_file=3D/etc/postgresql/16/main/postgresql.conf' -O '-c config_fi=
le=3D/etc/postgresql/17/main/postgresql.conf'

and it fa= ils with:
could not clone file between old and new data directo=
ries: Invalid argument
Failure, exiting

I ran the following strace command to find= out where the problem is happening:
sudo -u postgres strace -f=
 -e trace=3Dcopy_file_range,clone,ioctl,%file -s 2000 -v /usr/lib/postgre=
sql/17/bin/pg_upgrade --clone -d /var/lib/postgresql/16/main -b /usr/lib/=
postgresql/16/bin -B  /usr/lib/postgresql/17/bin -D /var/lib/postgresql/1=
7/main -o '-c config_file=3D/etc/postgresql/16/main/postgresql.conf&#=
39; -O '-c config_file=3D/etc/postgresql/17/main/postgresql.conf'=
 2>/tmp/pg_strace.log

And grepping through the log I fou= nd what looks like the problem here:
--
[pid 1153372] openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_C=
LOEXEC) =3D 8
[pid 1153372] openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpq.so.5&quo=
t;, O_RDONLY|O_CLOEXEC) =3D 8
[pid 1153261] unlink("/var/lib/postgresql/17/main/PG_VERSION.clonete=
st") =3D 0
[pid 1153261] openat(AT_FDCWD, "/var/lib/postgresql/16/main/PG_VERSI=
ON", O_RDONLY) =3D 4
[pid 1153261] openat(AT_FDCWD, "/var/lib/postgresql/17/main/PG_VERSI=
ON.clonetest", O_RDWR|O_CREAT|O_EXCL, 0600) =3D 5
[pid 1153261] ioctl(5, BTRFS_IOC_CLONE or FICLONE, 4) =3D -1 EINVAL (Inva=
lid argument)

Which I believe can be traced back to this co= mmit 3a769d8239afdc003c91= a56d2d8d5adfadacda5d.=C2=A0

I have validated that my filesyste= m supports reflinks:
$  dd of=3Dtempfile if=3D/dev/random bs=3D=
1M count=3D24
24+0 records in
24+0 records out
25165824 bytes (25 MB, 24 MiB) copied, 0.0806355 s, 312 MB/s
$  cp --reflink=3Dalways tempfile tempfile.reflink
$  btrfs filesystem du -s tempfile tempfile.reflink
     Total   Exclusive  Set shared  Filename
  24.00MiB       0.00B    24.00MiB  tempfile
  24.00MiB       0.00B    24.00MiB  tempfile.reflink

And th= ere's nothing interesting in `dmesg`.=C2=A0

I ran this same co= mmand today on a macOS system (using apfs) and it worked great. I have no= idea how to fix this problem and I'm curious if anyone has any point= ers.=C2=A0

Thanks,
Michael
----==_mimepart_6771eafedfa51_4f1f468515556--