Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ptUhk-0003cu-GB for pgsql-admin@arkaria.postgresql.org; Mon, 01 May 2023 14:40:40 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1ptUhi-00046q-38 for pgsql-admin@arkaria.postgresql.org; Mon, 01 May 2023 14:40:38 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1ptUhh-00046g-Li for pgsql-admin@lists.postgresql.org; Mon, 01 May 2023 14:40:37 +0000 Received: from mail-yb1-xb2d.google.com ([2607:f8b0:4864:20::b2d]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1ptUhe-006AlI-Oy for pgsql-admin@lists.postgresql.org; Mon, 01 May 2023 14:40:37 +0000 Received: by mail-yb1-xb2d.google.com with SMTP id 3f1490d57ef6-b9a7e639656so4507743276.0 for ; Mon, 01 May 2023 07:40:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1682952033; x=1685544033; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=hvvjnttNZi+ti5pmHJ94M0JFeFdmeiN9zJsGdJxlmkc=; b=ibr6TtFRLOFH3XH552eAJfik5Xek3xeGzr99GCduaZ1ipAgv5RhFr7oyFKaiazCPIs rRKWoIVEjTuwdnO4MD4bLsqUG7GW5xqEl9pu+IR0b7klM8ggJSYA4qQDC0SgwGp1NDkW jaBKB1KNm8gcy2iwcj7kOOMXDf9UT460YU3e5088/QhdHwk7RvhoyVFTSDYgjS/t3UKF 6CoStFdbfOA4Y93GtTNH+HmlFGMFrV9mkTULNtnetIbsNPKIE8YqRUInc2ao3IWw4tF5 Di9pheDrdrMrZclZsgg42NvLenWQHZPEDpj3vS/t7L9yb0cFDJU+yF/8NqPoZrkw8qAb 4Wsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682952033; x=1685544033; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=hvvjnttNZi+ti5pmHJ94M0JFeFdmeiN9zJsGdJxlmkc=; b=cAvSdWiCCSsYEhh2xfRyv+F48+Wgf3HSqPtG8OWg7GNyoqv9ED1MCi72S4L3gbCo3l fNj9s119zMtfgo2vdIkY7Uo86u5vpUjY4tpPaQmSRhPRVLIOVF+XXrBGprkAawX4rzpy vv5emg4w2mBq1q91//TEWuuGUVZG8Af4jw98d3ve1bJ/WGwq5RfGY/1x7mfh1flAbUSs RwMArQHulQRymknqeJvHPAJf+may5frfV89LZWgqQNAu85xJoABOv+A25gblZ+EkJP8J cl97MGKRZ6oRPMkFRo40hoCf5+JzJ/eT9BDcRUxeQoRTzu8PlcrFK/yMI5nLTsV8hvva jWng== X-Gm-Message-State: AC+VfDyjyFN1NOTecZI9UQ8FAVWmC1M61wT4t5OAWkCPww54h4olGc/D pKnKnYlcQIMjNDfikgM4OYQCkK8wkGs= X-Google-Smtp-Source: ACHHUZ4UpQg9CNyDDXXI3qTOvCCPbaDSvY8hzebRvXEzaAjEdvE8te2Kyt4DvnjorPBcZ234wUoMhQ== X-Received: by 2002:a25:5302:0:b0:b8f:22e7:e524 with SMTP id h2-20020a255302000000b00b8f22e7e524mr10702377ybb.40.1682952032766; Mon, 01 May 2023 07:40:32 -0700 (PDT) Received: from ?IPV6:2600:1700:bd2c:b810:edd6:19bb:e3ec:a4c2? ([2600:1700:bd2c:b810:edd6:19bb:e3ec:a4c2]) by smtp.googlemail.com with ESMTPSA id s62-20020a255e41000000b00b92579d3d7csm2570341ybb.52.2023.05.01.07.40.32 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 01 May 2023 07:40:32 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------ZbWU0ICx2KDoQt0gFNa7e8Z0" Message-ID: Date: Mon, 1 May 2023 09:40:32 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.10.0 Subject: Re: pg_dump why no indicator of completion Content-Language: en-US To: pgsql-admin@lists.postgresql.org References: <294e2735-73e5-3935-17e4-a00a35635a31@gmail.com> From: Ron In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------ZbWU0ICx2KDoQt0gFNa7e8Z0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit I backup some databases every night using pg_dump.  From a bash script initiated by cron, the way God intended them to be run.  And I check the return code, like every good production shell script should. I back up other databases using pgBackRest (also from a bash script initiated by cron, the way God intended them to be run).  And I check the return code, like every good production shell script should (even though it prints a message saying "Finished"). Because who in their right mind manually checks log files every morning, when a shell script will gladly email you if something went wrong? *Make the computer work for you!!!* On 5/1/23 09:31, richard coleman wrote: > Ron, > > That seems to be a very odd stance.  PostgreSQL and it's various utilities > are and have been used in very large commercial operations for decades. > pg_dump successfully backs up databases from 1MB to 10TB and beyond.  The > only issue I am having is for some inexplicable reason the devs haven't > included a status message upon completion. > > If you don't think that pg_dump is fit for purpose, then what do you > propose using in its stead? > > thanks, > rik. > > On Mon, May 1, 2023 at 10:07 AM Ron wrote: > > That's like complaining that a Toyota Camry isn't fit for hauling 10 > cubic yards of sand. > > > On 5/1/23 08:55, richard coleman wrote: >> Ron, >> >> Are you writing that pg_dump is unfit for purpose and that I should >> be using a commercial backup solution instead? >> >> rik. >> >> On Mon, May 1, 2023 at 9:43 AM Ron wrote: >> >> >> >> On 5/1/23 07:56, richard coleman wrote: >>> Ron, >>> So what you are saying is that I have to write a shell script >>> around pg_dump to catch the exit code of pg_dump since the >>> developers of pg_dump didn't have pg_dump simply write an exit >>> message? >>> >>> Since multiple pg_dump commands are often run at the same >>> command prompt, and they can take hours, if not days to run, >> >> Running a days-long pg_dump at any time except special occasions >> isn't very wise.  Especially from a command prompt. >> >> Binary backup/restore programs are /much/ faster (if for no other >> reason than they let you do incremental and differential backups). >> >>> and there are a myriad of other commands the will be run in the >>> interim, any exit code generated by pg_dump with be lost in the >>> flotsam and jetsam of the multitude of exit codes created by >>> every other command run between the calling of pg_dump and it's >>> eventual termination. It could have completed successfully, >>> crashed, been killed, etc. >>> >>> Or am I mistaken? >>> >>> thanks, >>> rik. >>> >>> On Sun, Apr 30, 2023 at 11:28 PM Ron >>> wrote: >>> >>> On 4/30/23 19:18, richard coleman wrote: >>>> Hi all, >>>> >>>> I've been working with pg_dump and one thing that's always >>>> struck me as strange is that there never seems to be an >>>> indication in the log that the pg_dump process completed >>>> successfully. >>>> >>>> For example, I've been running a bunch of backups with >>>> pg_dump in directory mode with multiple jobs. When I come >>>> back several hours later there are no pg_dump processes >>>> running.  The log is filled with: >>>> >>>> dumping contents of foo >>>> finished item 123456 of TABLE DATA bar >>>> etc. >>>> >>>> >>>> Other than trying to restore the multi terabyte database >>>> somewhere else is there any way to know that it actually >>>> finished successfully? >>>> >>>> Why doesn't pg_dump add a line in the output like: >>>> >>>> pg_backup finished >>>> >>>> when it's completed successfully? >>>> >>>> This seems like a terrible oversight. >>>> >>>> Am I missing something obvious? >>> >>> "They" expect you to do the Unix Thing and check $?. Thus, >>> that's what I do; any non-zero value generates an email with >>> a scary subject line, the specifics of which are based on >>> the exact code value.  I'm not near my work computer; >>> otherwise I'd share them with you. >>> >>> -- >>> Born in Arizona, moved to Babylonia. >>> >> >> -- >> Born in Arizona, moved to Babylonia. >> > > -- > Born in Arizona, moved to Babylonia. > -- Born in Arizona, moved to Babylonia. --------------ZbWU0ICx2KDoQt0gFNa7e8Z0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit I backup some databases every night using pg_dump.  From a bash script initiated by cron, the way God intended them to be run.  And I check the return code, like every good production shell script should.

I back up other databases using pgBackRest (also from a bash script initiated by cron, the way God intended them to be run).  And I check the return code, like every good production shell script should (even though it prints a message saying "Finished").

Because who in their right mind manually checks log files every morning, when a shell script will gladly email you if something went wrong?  Make the computer work for you!!!

On 5/1/23 09:31, richard coleman wrote:
Ron, 

That seems to be a very odd stance.  PostgreSQL and it's various utilities are and have been used in very large commercial operations for decades.
pg_dump successfully backs up databases from 1MB to 10TB and beyond.  The only issue I am having is for some inexplicable reason the devs haven't included a status message upon completion.

If you don't think that pg_dump is fit for purpose, then what do you propose using in its stead?

thanks, 
rik.

On Mon, May 1, 2023 at 10:07 AM Ron <ronljohnsonjr@gmail.com> wrote:
That's like complaining that a Toyota Camry isn't fit for hauling 10 cubic yards of sand.


On 5/1/23 08:55, richard coleman wrote:
Ron, 

Are you writing that pg_dump is unfit for purpose and that I should be using a commercial backup solution instead?

rik.

On Mon, May 1, 2023 at 9:43 AM Ron <ronljohnsonjr@gmail.com> wrote:


On 5/1/23 07:56, richard coleman wrote:
Ron, 
So what you are saying is that I have to write a shell script around pg_dump to catch the exit code of pg_dump since the developers of pg_dump didn't have pg_dump simply write an exit message?

Since multiple pg_dump commands are often run at the same command prompt, and they can take hours, if not days to run,

Running a days-long pg_dump at any time except special occasions isn't very wise.  Especially from a command prompt.

Binary backup/restore programs are much faster (if for no other reason than they let you do incremental and differential backups).

and there are a myriad of other commands the will be run in the interim, any exit code generated by pg_dump with be lost in the flotsam and jetsam of the multitude of exit codes created by every other command run between the calling of pg_dump and it's eventual termination. It could have completed successfully, crashed, been killed, etc.

Or am I mistaken?

thanks, 
rik. 

On Sun, Apr 30, 2023 at 11:28 PM Ron <ronljohnsonjr@gmail.com> wrote:
On 4/30/23 19:18, richard coleman wrote:
Hi all, 

I've been working with pg_dump and one thing that's always struck me as strange is that there never seems to be an indication in the log that the pg_dump process completed successfully.

For example, I've been running a bunch of backups with pg_dump in directory mode with multiple jobs.  When I come back several hours later there are no pg_dump processes running.  The log is filled with:
dumping contents of foo
finished item 123456 of TABLE DATA bar
etc.

Other than trying to restore the multi terabyte database somewhere else is there any way to know that it actually finished successfully?

Why doesn't pg_dump add a line in the output like:
pg_backup finished
when it's completed successfully? 

This seems like a terrible oversight.

Am I missing something obvious?

"They" expect you to do the Unix Thing and check $?. Thus, that's what I do; any non-zero value generates an email with a scary subject line, the specifics of which are based on the exact code value.  I'm not near my work computer; otherwise I'd share them with you.

--
Born in Arizona, moved to Babylonia.

--
Born in Arizona, moved to Babylonia.

--
Born in Arizona, moved to Babylonia.

--
Born in Arizona, moved to Babylonia.
--------------ZbWU0ICx2KDoQt0gFNa7e8Z0--