Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qOH3z-007LDs-9W for pgsql-general@arkaria.postgresql.org; Tue, 25 Jul 2023 12:22:51 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1qOH3x-00GEbU-MS for pgsql-general@arkaria.postgresql.org; Tue, 25 Jul 2023 12:22:49 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qOH3x-00GEbK-1E for pgsql-general@lists.postgresql.org; Tue, 25 Jul 2023 12:22:49 +0000 Received: from tamriel.snowman.net ([2001:470:e38f::11]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qOH3t-000G3E-Ty for pgsql-general@postgresql.org; Tue, 25 Jul 2023 12:22:48 +0000 Received: by tamriel.snowman.net (Postfix, from userid 1000) id 2FBE45F7B7; Tue, 25 Jul 2023 08:22:39 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=snowman.net; s=dkim; t=1690287759; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=XltkKxPvVksgJWnO63y2jfOFiXA38f2CWbMF1cAICYs=; b=TFOihDGu2+84ZkAWvII/Io06FB8iO20AWxLIpaW6ZAkdkl48WiVqDj1ZMF5hZ9y1OsVoQ/ hFI1cD7VCgbxjT4DxGufLTkLNcD76QUOAgspnHuwCGZSL6zLLGzh0eUsJwNiCXA3Rirvqj ruaOJ/7C4BQr4eD1Cm5ZPJBalpfDFSA= Date: Tue, 25 Jul 2023 08:22:39 -0400 From: Stephen Frost To: Abhishek Bhola Cc: pgsql-general Subject: Re: Difference in the tablespace folders on primary and secondary nodes Message-ID: References: MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="9yDdhyZspMkRgMoL" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/2.1.4 (2021-12-11) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --9yDdhyZspMkRgMoL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Greetings, * Abhishek Bhola (abhishek.bhola@japannext.co.jp) wrote: > I recently set up a Postgres15 master-slave (Primary-secondary) cluster on > 2 nodes. At the time of starting, I ensured that data files are exactly t= he > same on both the nodes. The size of the DB is 1.5TB. > The directory structure on both nodes looks as follows: > ``` > - /PROD/datadg/ > | > |-> /PROD/datadg/tablespace > | |-> /PROD/datadg/tablespace/tablespace1 > | |-> /PROD/datadg/tablespace/tablespace2 > | > |-> /PROD/datadg/data > | |-> /PROD/datadg/data/pg_tblspc > | | | -> /PROD/datadg/data/pg_tblspc/16432 -> > /PROD/datadg/tablespace/tablespace1 > | | | -> /PROD/datadg/data/pg_tblspc/16433 -> > /PROD/datadg/tablespace/tablespace2 >=20 > ``` > Almost a week later now, I see almost a 2GB size difference in the > tablespace folders on the 2 nodes. I also see some file count difference = on > both the nodes. Not really enough to go on here. > `autovacuum` is on on both the nodes and there aren't any `idle in > transaction` queries on the slave node. Also there hasn't been any > disruption on the streaming replication. I did not get any error like `WAL > segment already removed` or so on the slave node. `pg_stat_replication` on > the master node also doesn't show anything out of the ordinary and the > `sent_lsn`, `write_lsn` and `flush_lsn` are regularly updated. I can not > see a difference in counts of most tables either, haven't verified for all > of them. > So my **first question** is: >=20 > * Why is there a difference in the files in the tablespace folder? I can > understand the difference in the modification timestamps, but some files > are just missing on the slave node. Unlogged tables would certainly be a pretty easy explanation of size differences between the two. There's also temporary files that might be created on one system but not the other for in-progress queries. These aren't the only possibilities but just a couple of likely candidates. > Now if I were to run `vacuumdb` on the master node, there are chances that > the slave node will break and give an error like this > ``` > PANIC,XX000,"WAL contains references to invalid pages",,,,,"WAL redo at > 875E/21A70BD0 for > Heap2/VISIBLE: cutoff xid 60350476 flags 0x01; blkref #0: rel > 16405/16419/533716933, fork 2, blk 26; blkref #1: rel > 16405/16419/533716933, blk 853758",,,,"","startup > ``` Did you actually run vacuumdb and actually see this? Are you able to reproduce it? > In the case when slave node breaks, these are the steps I usually do to > bring the slave node back: > 1) Start `pg_backup_start('backup')` on the master node > 2) rsync the files from master to slave by running the following on the > slave node: > ``` > rsync -av --delete master_node:/PROD/datadg/data/ /PROD/datadg/data > --exclude 'pg_log' --exclude 'pg_replslot' > ``` > 3. Stop `pg_backup_stop()` on master node This really isn't good enough as rsync will use timestamp/size by default and this also doesn't grab and restore the absolutely required backup_label file that's returned from pg_backup_stop(). Basically, this is not a valid way to perform a backup/restore of PG. Please go read the documentation for how to use the low-level API ... or, even better, use an existing well maintained backup/restore tool for this. > 4. Start the slave node again and it usually works, even though the > tablespace files might not still be the same. If you didn't grab the backup_label from pg_backup_stop() and drop it into place on the replica, then you've almost certainly got a corrupt system. > **Second question**: >=20 > * What is the best way to bring the slave node back? Is the `rsync` betwe= en > tablespaces required? And if yes, what is the best method to do it for ve= ry > large databases, something maybe as big as 30TB or more. I don't want to > `rsync` all the files even if the timestamp on them is different. So is a > command like this safe to do? Or should an option like `--checksum` be > used? > ``` > rsync -av --delete master_node:/PROD/datadg/tablespace/ > /PROD/datadg/tablespace --size-only > ``` Using --size-only is an absolutely horrid idea and will miss lots of changes. What you're doing isn't working because you're just creating corrupt replicas by not following the process correctly for performing a backup and restore of PG. This stuff isn't easy. Use a tool for it, especially for larger systems. > **Third question:** >=20 > * Is it advised to run `vacuumdb` before or after bringing the slave node > back again? No, this isn't necessary. Thanks, Stephen --9yDdhyZspMkRgMoL Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCgAdFiEEwf6gbxKhD863zrx/7WyKOINHZFUFAmS/vowACgkQ7WyKOINH ZFXI9xAApubC+5X1Msugx5PXSRCkn4VuXBCCc8r68j1GwLd/dDl3oLHzZWrjW4Ns J8ri/NKjuJaM1/+9jbe3OzHxJ9jYYnNroKu5c/J249xAEAuxyquwIvcWYUvUarQP IhY9nJtNrYzIFxQz7xIEh9K5Icn8aJSKbzt7AXAL3DjrFxWtgszD+3AC70iJqbkz U42QpSuuv49CZw3ghXbwR5HixDhrYAD1w3HT/Zw1pAvEmPPUETmmVBxjUGorrs/y 4Jbulh4hHGjVzo6Ld100e+7kcr8YO4WGdE15ZfVyxJhchCx23Cw7CzI+r82sgqvJ lob0XnWOUafdw4MYgBKfhk8nNVgzTLxY+sjBoIn6rW9X0C/vIJYWuPwQAXCHdRg1 cd0gNjOIp42yWhAUdLPds8mW7nuV5o6mBeFGajUGB9YD1gAColFRHPhkJSzvb5Of B5qIvh2OSS8N/VcDvff7TWAeVaQxs8jXfCa9DYSo0V4exibZxFavsSqmDd10pz/O 1kJe/IPV3Rq1KpwpB2zvmfo8lcehnkorCMQccMVEb5mlR73KeKF8jYNImzov8rKG KBkoCFPCHkGF5LvCYiTcgHumOHqTST2V3BXdt3hvs0Wa2Iw/wvLSlWMW98Jpctdf f/2FVhj8qFfPeAOyOjqBagaIYt8IbuSdWia+n/So4ic5wOckrFQ= =3SxM -----END PGP SIGNATURE----- --9yDdhyZspMkRgMoL--