Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uCckN-00HaGc-O6 for pgsql-general@arkaria.postgresql.org; Wed, 07 May 2025 11:15:32 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1uCckM-00Dfgf-7z for pgsql-general@arkaria.postgresql.org; Wed, 07 May 2025 11:15:30 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uCckL-00DfgX-Kx for pgsql-general@lists.postgresql.org; Wed, 07 May 2025 11:15:29 +0000 Received: from mail-yb1-xb30.google.com ([2607:f8b0:4864:20::b30]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1uCckI-000ZzD-2Y for pgsql-general@lists.postgresql.org; Wed, 07 May 2025 11:15:28 +0000 Received: by mail-yb1-xb30.google.com with SMTP id 3f1490d57ef6-e78e13d7af8so462251276.3 for ; Wed, 07 May 2025 04:15:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1746616526; x=1747221326; darn=lists.postgresql.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=+Ab/P7c9MkUPtmj+sgvTnvU5p776SF7N6btZrqERJZY=; b=Pb2vo2oRqWBoz1OAIZfQvyKIKXwHgIAzBMhubJQx7tPTgnZHHtbUScil4HTdZyJl2L dqELkgmBody6nhzsQlgj2nK0OMYQgpj/Qw08iDUvrGmHC5BhpFDseykVm9Y/BkN3EqbK p84O8mIpB5D7vDjX+kr18brdbWifKT5DWs7t177CzypDCh80IJIi+nTr5JF3B9JEKRsD jk2QE0QoP0M+0Uke8zUTGiX9bsFyES2OwVhiktZmZ2VNiaHghf7bg84/CcgZR6XzHs2d DZ9I/RzcIFHU2KJQ/tbq0bDEkhVdBJH7Pe0U9ofnYTKX8TUiskYQnszwC9xtFaLy/wKM c0dA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1746616526; x=1747221326; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=+Ab/P7c9MkUPtmj+sgvTnvU5p776SF7N6btZrqERJZY=; b=KLlr7fPg55xqXCZHOe9m6nJAjxB5EOUTP1N2mS5qi1A2rfT3EwWrgB3pNOQcJHlvee hK9iSacCdJADs5F3GsNbp1ph0IhmMZUw56Rqf4aNLdJVkcyCgk7BpgwgxCHug8Wgn+cs 52hoK+GTS9QuSTvaboUbYf1/kVs5k8Vbltdoas6TdvkD0fXw5sI28nQj5d7UpdECpyjz bvYUBfNUizGEWuZV8n9O3+sq2W7sQucHItdIwMZr1Lk6rPW234gvmk5IUTrd8eTHJXXe LhU8O15J/Voyis6tP4CUMNzwrptX20grwyZZBOnIHk5HPtCY29bEVtbTuK27HQmJnmEW 38zA== X-Gm-Message-State: AOJu0YwuA/Hy6zCOM/Jhylqo+72R0+JWLGEB1JwaWbkq1K4W0K0E3eTF KlGxHFDB2GFBPnYDhNU3UpGu2v8fA91nO57K2OaZkOqRu/qgqd1PHkx4UdjijTvCXF/VxqkIC1M jah+exKuikBKnRiidTqIgU5nqzdOgdO/a X-Gm-Gg: ASbGncvfQYshg8q82AHeA24m8/3aMi4PQDraTFWPVc050xtNxy/geMIVLCInEWBkoV0 j8RVQVVQM+iAwAKJvE0vcJwMvnAhiGMkm/rLyIEX7PPN0FrP8HxJFCf9PkDKj7VSJUJWx6/a781 YeLGrZu8yShsJf6HJbka0qHwI= X-Google-Smtp-Source: AGHT+IHgvzkM0f7O4BHhtuBmrqB2vf/cJhw6XmYBF6PBfCD235vOsULOWSoEd6WNPVkeogdWXLAGnB4+balZqcz5MQc= X-Received: by 2002:a05:6902:2493:b0:e72:d424:8e11 with SMTP id 3f1490d57ef6-e78814436d7mr3336425276.31.1746616526218; Wed, 07 May 2025 04:15:26 -0700 (PDT) MIME-Version: 1.0 From: KK CHN Date: Wed, 7 May 2025 16:47:29 +0530 X-Gm-Features: ATxdqUHULLDaU-UhNYWDEjV3Bjo4kOiYKYgKexKyPNdND-7aD31NCo35A-t0Zqs Message-ID: Subject: Pgbackrest failure for INCR and DIFF but not FULL backup To: pgsql-general@lists.postgresql.org Content-Type: multipart/alternative; boundary="0000000000009aade6063489d98d" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000009aade6063489d98d Content-Type: text/plain; charset="UTF-8" Hi folks, I am facing a strange issue, Pgbackrest backup fails for DIFF or INCR backups but not Full backup, with the * error WAL file cannot be archived before 60000 ms timeout.* The pgbackrest " *stanza check* " command *sometimes succeeds, but sometimes fails.* I don't know why * PG is unable to* *copy WAL files from pg_wal to /data/myarchive_di*r* in real time*. I always observed a delay of a few minutes for a wal file from pg_wal to appear in /data/my_archive_dir. I'hv observed in the postgresql.conf (checkpoint_timeout = 5 m, max_wal_size = 16 GB, wal_keep_size=15GB, min_wal_size=80MB etc.) and I found pg_wal dir size is 16 GB on disk (du -h -d 1 ) and /data/archive = 1.2 T /dev/mapper/rhel_bcga68-data 5.0T 1.8T 3.3T 35% /data Can we suspect the 5 M is the reason for the WAL archiving delay ? Backup to a remote RepoServer for INCR or DIFF backup always fails, I found a full backup always succeeds!!! What is the ideal value needed to be set for "*checkpoint_timeout*" ? *Or this doesn't have any impact on pgbackrest failure ?* *archive_command = 'pgbackrest --stanza=My_Repo archive-push %p && cp %p /data/archive/%f' * From postgresql logs I am seeing this .. ERROR: [082]: unable to push WAL file '000000010000026300000002' to the archive asynchronously after 60 second(s) HINT: check '/var/log/pgbackrest/My_Repo-archive-push-async.log' for errors. INFO: archive-push command end: aborted with exception [082] 2025-05-02 12:15:17 IST LOG: archive command failed with exit code 82 2025-05-02 12:15:17 IST DETAIL: The failed archive command was: pgbackrest --stanza=My_Repo archive-push pg_wal/000000010000026300000002 && cp pg_wal/000000010000026300000002 /data/archive/000000010000026300000002 INFO: archive-push command begin 2.52.1: [pg_wal/000000010000026300000002] --archive-async --compress-type=zst --exec-id=2848559-384cf49c --log-level-console=info --log-level-file=debug --log-level-stderr=info --pg1-path= /var/lib/postgres/16/data --pg-version-force=16 --process-max=6 --repo1-host=10.x.y.202 --repo1-host-user=pgbackrest --spool-path=/var/spool/pgbackrest --stanza=My_Repo top output on DB cluster: top - 12:37:00 up 66 days, 17:24, 2 users, load average: 4.04, 4.72, 4.56 Tasks: 902 total, 4 running, 897 sleeping, 0 stopped, 1 zombie %Cpu(s): 7.4 us, 1.7 sy, 0.0 ni, 89.9 id, 0.4 wa, 0.2 hi, 0.4 si, 0.0 st MiB Mem : 31837.6 total, 706.1 free, 15243.0 used, 24741.0 buff/cache MiB Swap: 8060.0 total, 6634.0 free, 1426.0 used. 16608.9 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 2839363 postgre+ 20 0 8965608 7.2g 7.1g S 70.2 23.0 2:02.61 postgres 2864108 postgre+ 20 0 8967848 7.1g 7.1g S 64.9 22.8 0:30.04 postgres 2865547 postgre+ 20 0 8965432 7.1g 7.1g S 39.1 22.8 0:32.30 postgres 2865752 postgre+ 20 0 8964352 6.9g 6.9g S 16.6 22.3 0:32.94 postgres Model name: Intel(R) Xeon(R) Gold 6430 BIOS Model name: Intel(R) Xeon(R) Gold 6430 CPU family: 6 Model: 143 Thread(s) per core: 1 Core(s) per socket: 16 These are vCPUs (16 nos) , OS RHEL 9, postgres 16, pgbackrest 2.52.1 *Any hints most welcome to find the root cause / troubleshoot the pgbackrest failures for DIFF/ INCR backups.* Thank you Krishane *For More Inputs: * For more info : you can see the full backup success here. but a diff backup fails. full backup: 20250505-070204F timestamp start/stop: 2025-05-05 07:02:04+05:30 / 2025-05-05 22:11:23+05:30 wal start/stop: 000000010000026F00000066 / 000000010000027300000045 database size: 503.1GB, database backup size: 503.1GB repo1: backup size: 79GB When I try diff backup it always fails. [root@Repo ~]# tail -f /var/log/pgbackrest/My_Repo-backup.log stack trace: command/archive/find.c:walSegmentFind:191:(this: {WalSegmentFind}, walSegment: {"000000010000027B0000006C"}) command/backup/backup.c:backupArchiveCheckCopy:(backupData: {BackupData}, manifest: {Manifest}) command/backup/backup.c:cmdBackup:(void) main.c:main:(debug log level required for parameters) -------------------------------------------------------------------- 2025-05-07 15:47:49.760 P00 INFO: backup command end: aborted with exception [082] 2025-05-07 15:47:49.760 P00 DEBUG: command/exit::exitSafe: => 82 2025-05-07 15:47:49.860 P00 DEBUG: main::main: => 82 ^C [ root@Repo ~ ~]# date Wednesday 07 May 2025 04:06:37 PM IST *The postgres log says* =316781-61d82f85 --log-level-console=info --log-level-file=debug --log-level-stderr=info --pg1-path= /var/lib/postgres/16/data --pg-version-force=16 --process-max=3 --repo1-host=10.x.y.202 --repo1-host-user=pgbackrest --spool-path=/var/spool/pgbackrest --stanza=My_Repo INFO: pushed WAL file '000000010000027B00000082' to the archive asynchronously INFO: archive-push command end: completed successfully (37506ms) INFO: archive-push command begin 2.52.1: [pg_wal/000000010000027B00000083] --archive-async --compress-type=zst --exec-id=317334-27a30b57 --log-level-console=info --log-level-file=debug --log-level-stderr=info --pg1-path= /var/lib/postgres/16/data --pg-version-force=16 --process-max=3 --repo1-host=10.x.y.202 --repo1-host-user=pgbackrest --spool-path=/var/spool/pgbackrest --stanza=My_Repo ERROR: [082]: unable to push WAL file '000000010000027B00000083' to the archive asynchronously after 60 second(s) HINT: check '/var/log/pgbackrest/My_Repo-archive-push-async.log' for errors. INFO: archive-push command end: aborted with exception [082] 2025-05-07 16:22:56 IST LOG: archive command failed with exit code 82 [ root@Repo ~ ~]# --0000000000009aade6063489d98d Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hi folks,

I am facing a stra= nge issue, Pgbackrest backup fails for DIFF or INCR backups but not Full ba= ckup,=C2=A0 with the=C2=A0 error WAL file cannot be archived before 6000= 0 ms timeout.

The pgbackrest=C2=A0 " s= tanza check " command=C2=A0 sometimes succeeds, but sometimes f= ails.

I don't know why=C2=A0 =C2=A0 PG = is unable to=C2=A0 =C2=A0copy=C2=A0 WAL files from=C2=A0 =C2=A0pg_wa= l to=C2=A0 /data/myarchive_dir in real time. I always=C2=A0 obse= rved a delay of a few minutes for a wal file from pg_wal to appear in=C2=A0= /data/my_archive_dir.=C2=A0

=C2=A0I'hv observ= ed=C2=A0 in the=C2=A0 postgresql.conf=C2=A0 =C2=A0(checkpoint_timeout =3D 5= m,=C2=A0 max_wal_size =3D 16 GB, wal_keep_size=3D15GB, min_wal_size=3D80MB= etc.)=C2=A0

and I found=C2=A0 pg_wal dir size is = 16 GB on disk (du -h -d 1 )=C2=A0 and /data/archive=C2=A0 =3D 1.2 T
/dev/mapper/rhel_bcga68-data =C2=A05.0T =C2=A01.8T =C2=A03.3T =C2=A035% = /data


Can we suspect the 5 M is the= reason=C2=A0for the WAL archiving delay=C2=A0 ?=C2=A0

=
Backup=C2=A0 to a remote RepoServer for INCR or DIFF=C2=A0 backup alwa= ys fails, I found a full backup=C2=A0always succeeds!!!=C2=A0
What is the ideal value needed=C2=A0to be set for=C2=A0 "<= b>checkpoint_timeout"=C2=A0 =C2=A0?=C2=A0=C2=A0
Or th= is doesn't have any impact on=C2=A0 pgbackrest failure ?
=

archive_command =3D 'pgbackrest --stan= za=3DMy_Repo archive-push %p && cp %p /data/archive/%f'=C2=A0


From postgresql logs=C2=A0 I am = seeing this ..=C2=A0

ERROR: [082]: unable to push WAL f= ile '000000010000026300000002' to the archive asynchronously after = 60 second(s)
=C2=A0 =C2=A0 =C2=A0 =C2=A0HINT: check '/var/log/pgback= rest/My_Repo-archive-push-async.log' for errors.
INFO: archive-push = command end: aborted with exception [082]
2025-05-02 12:15:17 IST LOG: = =C2=A0archive command failed with exit code 82
2025-05-02 12:15:17 IST D= ETAIL: =C2=A0The failed archive command was: pgbackrest --stanza=3DMy_Repo = archive-push pg_wal/000000010000026300000002 && cp pg_wal/000000010= 000026300000002 /data/archive/000000010000026300000002
INFO: archive-pus= h command begin 2.52.1: [pg_wal/000000010000026300000002] --archive-async -= -compress-type=3Dzst --exec-id=3D2848559-384cf49c --log-level-console=3Dinf= o --log-level-file=3Ddebug --log-level-stderr=3Dinfo --pg1-path=3D /var/lib= /postgres/16/data=C2=A0=C2=A0=C2=A0--pg-version-force=3D16 --process-max=3D= 6 --repo1-host=3D10.x.y.202 --repo1-host-user=3Dpgbackrest --spool-path=3D/= var/spool/pgbackrest --stanza=3DMy_Repo

top=C2=A0 output=C2=A0 =C2= =A0on DB cluster:

top - 12:37:00 up 66 days, 17:24, =C2= =A02 users, =C2=A0load average: 4.04, 4.72, 4.56

T= asks: 902 total, =C2=A0 4 running, 897 sleeping, =C2=A0 0 stopped, =C2=A0 1= zombie
%Cpu(s): =C2=A07.4 us, =C2=A01.7 sy, =C2=A00.0 ni, 89.9 i= d, =C2=A00.4 wa, =C2=A00.2 hi, =C2=A00.4 si, =C2=A00.0 st
MiB Mem= : =C2=A031837.6 total, =C2=A0 =C2=A0706.1 free, =C2=A015243.0 used, =C2=A0= 24741.0 buff/cache
MiB Swap: =C2=A0 8060.0 total, =C2=A0 6634.0 f= ree, =C2=A0 1426.0 used. =C2=A016608.9 avail Mem

=C2= =A0 =C2=A0 PID USER =C2=A0 =C2=A0 =C2=A0PR =C2=A0NI =C2=A0 =C2=A0VIRT =C2= =A0 =C2=A0RES =C2=A0 =C2=A0SHR S =C2=A0%CPU =C2=A0%MEM =C2=A0 =C2=A0 TIME+ = COMMAND
2839363 postgre+ =C2=A020 =C2=A0 0 8965608 =C2=A0 7.2g =C2=A0 7.= 1g S =C2=A070.2 =C2=A023.0 =C2=A0 2:02.61 postgres
2864108 postgre+ =C2= =A020 =C2=A0 0 8967848 =C2=A0 7.1g =C2=A0 7.1g S =C2=A064.9 =C2=A022.8 =C2= =A0 0:30.04 postgres
2865547 postgre+ =C2=A020 =C2=A0 0 8965432 =C2=A0 7= .1g =C2=A0 7.1g S =C2=A039.1 =C2=A022.8 =C2=A0 0:32.30 postgres
2865752 = postgre+ =C2=A020 =C2=A0 0 8964352 =C2=A0 6.9g =C2=A0 6.9g S =C2=A016.6 =C2= =A022.3 =C2=A0 0:32.94 postgres



Model name: =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0Intel(R) Xeon(R) Gold 6430
=C2=A0 =C2=A0 BIOS= Model name: =C2=A0 =C2=A0 Intel(R) Xeon(R) Gold 6430
=C2=A0 =C2=A0 CPU = family: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A06
=C2=A0 =C2=A0 Model: =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 143
=C2=A0 =C2=A0 Thread(s) pe= r core: =C2=A01
=C2=A0 =C2=A0 Core(s) per socket: =C2=A016

These = are vCPUs=C2=A0 =C2=A0 (16 nos) , OS RHEL 9,=C2=A0 postgres 16, pgbackrest = 2.52.1=C2=A0

Any hints=C2=A0 most welcome to find the root cause = / troubleshoot the=C2=A0=C2=A0pgbackrest failures for DIFF/ INCR backups.

Thank you
Krishane
=





=
For More Inputs:=C2=A0 =C2=A0
For more info=C2=A0 =C2= =A0:=C2=A0 you can see the full backup success here. but a diff backup fail= s.=C2=A0

=C2=A0 =C2=A0 =C2=A0 =C2=A0 full backup: 20250505-07= 0204F
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 timestamp start/stop: 20= 25-05-05 07:02:04+05:30 / 2025-05-05 22:11:23+05:30
=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 wal start/stop: 000000010000026F00000066 / 0000000100= 00027300000045
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 database size: = 503.1GB, database backup size: 503.1GB
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 repo1: backup size: 79GB


When I try diff backup=C2=A0 it always fails.=C2=A0

[root@Re= po ~]# tail -f /var/log/pgbackrest/My_Repo-backup.log

=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 stack trace:
=C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 command/archive/find.c:walSegmentFind:191:(= this: {WalSegmentFind}, walSegment: {"000000010000027B0000006C"})=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 command/backup/back= up.c:backupArchiveCheckCopy:(backupData: {BackupData}, manifest: {Manifest}= )
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 command/backup/back= up.c:cmdBackup:(void)
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 main.c:main:(debug log level required for parameters)
=C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ------------------------------------= --------------------------------
2025-05-07 15:47:49.760 P00 =C2=A0 INFO= : backup command end: aborted with exception [082]
2025-05-07 15:47:49.7= 60 P00 =C2=A0DEBUG: =C2=A0 =C2=A0 command/exit::exitSafe: =3D> 82
202= 5-05-07 15:47:49.860 P00 =C2=A0DEBUG: =C2=A0 =C2=A0 main::main: =3D> 82<= br>^C
[ root@Repo ~=C2=A0=C2=A0=C2=A0~]# date
Wednesday 07 May 2025 04:06:37 PM = IST

The postgres=C2=A0 log says
<= br>
=3D316781-61d82f85 --log-level-console=3Dinfo --log-level-fil= e=3Ddebug --log-level-stderr=3Dinfo --pg1-path=3D /var/lib/postgres/16/data=C2=A0=C2=A0=C2=A0--pg-version-force=3D16 --proces= s-max=3D3 --repo1-host=3D10.x.y.202 --repo1-host-user=3Dpgbackrest --spool-= path=3D/var/spool/pgbackrest --stanza=3DMy_Repo
INFO: pushed WAL file &#= 39;000000010000027B00000082' to the archive asynchronously
INFO: arc= hive-push command end: completed successfully (37506ms)
INFO: archive-pu= sh command begin 2.52.1: [pg_wal/000000010000027B00000083] --archive-async = --compress-type=3Dzst --exec-id=3D317334-27a30b57 --log-level-console=3Dinf= o --log-level-file=3Ddebug --log-level-stderr=3Dinfo --pg1-path=3D /var/lib/postgres/16/data=C2=A0=C2=A0=C2=A0--pg-version-force=3D16 --proces= s-max=3D3 --repo1-host=3D10.x.y.202 --repo1-host-user=3Dpgbackrest --spool-= path=3D/var/spool/pgbackrest --stanza=3DMy_Repo
ERROR: [082]: unable to = push WAL file '000000010000027B00000083' to the archive asynchronou= sly after 60 second(s)
=C2=A0 =C2=A0 =C2=A0 =C2=A0HINT: check '/var/= log/pgbackrest/My_Repo-archive-push-async.log' for errors.
INFO: arc= hive-push command end: aborted with exception [082]
2025-05-07 16:22:56 = IST LOG: =C2=A0archive command failed with exit code 82

[ root@Repo ~=C2=A0=C2=A0=C2=A0~]#
--0000000000009aade6063489d98d--