Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qnfWY-00Ban3-75 for pgsql-general@arkaria.postgresql.org; Tue, 03 Oct 2023 13:33:18 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1qnfWW-005ecw-L7 for pgsql-general@arkaria.postgresql.org; Tue, 03 Oct 2023 13:33:16 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qnfWW-005ec3-9v for pgsql-general@lists.postgresql.org; Tue, 03 Oct 2023 13:33:16 +0000 Received: from mail-yw1-x112f.google.com ([2607:f8b0:4864:20::112f]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1qnfWT-007yvC-O5 for pgsql-general@lists.postgresql.org; Tue, 03 Oct 2023 13:33:15 +0000 Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-59f57ad6126so10666087b3.3 for ; Tue, 03 Oct 2023 06:33:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1696339993; x=1696944793; darn=lists.postgresql.org; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=onU7IbOb8bbc4zygteW7UGKXaQQ+n6W2EdBd9wcEeJs=; b=FYFh9jtfKgWeTr7qDLrYbJuckA/ct8vbZKNtWtZDsUmYUZDRwU4atBHoDMbodEXlxI unBjSG89a3n4CzHKSEwjorZlY0fmbNWE2xgz2ZbpGYr6kbDkTEwDS9q68ROdfV6kzm/K dWqO9fAdXql6Iu3w6+ndOl+AtslsNEZLY3VH9PLQDjbT6vry9FXJAWqfNtxz6FIL7QE2 qbgHRpLZioDXEwiccPunh9OBAQi3krNLO2d59tTB9FSeBnwlasCwjLZVoA7Uua5bQal6 ezIHcsXunzG38h4Md9OxcEPAs/oz8b80TO0mhl+4tYMOIyxkDhY6BOZQg3wA/XaHQuW3 D7bA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1696339993; x=1696944793; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=onU7IbOb8bbc4zygteW7UGKXaQQ+n6W2EdBd9wcEeJs=; b=jMRuap00E3Z0HVZTEAedRbd8hpUz2y2gdhf2KaGIPaBi9HFgDnu2UJCMeqztAKnLpz eJbKqINQ5lqa8Uxhzt+e1kyxxtqPo0ZI+6B1TKNoInilzUnaBvA4dGgbEFomD89oV1vQ 9fAbDEuH+n0MliOK8zrnjOx6AOX1qcBZAxIM+dd/ta0sqtaFDQe8Ba4B0iaWc3/a6NLo 91Oxh3pLfzGOsfrlueXX9cQ+TJEQt83uZlkBdADHNrDCwP3UaZwBKVWCwSIn2oTYeMH2 wPPYaBbf4NNzpwhGKvMTrVmV1q5nA1cO3Jeg76HeHEv+boL9MqLwvjpdmvxmoy1qGBcc oVUg== X-Gm-Message-State: AOJu0YxvGubR/oAH+xZ3xqYMbvwWke0HZOIG87ETxbsW471059SnQ8de FAJXl9uAUkV7cZGw/jidN1W2AFxoYz0= X-Google-Smtp-Source: AGHT+IHhzMnVnD8OQy1KMePK660yh9+Fi0wrTC6W5alakFnYBOmgqmRpOAz8wY1O8lqrGJ5vMVxxvQ== X-Received: by 2002:a0d:e3c3:0:b0:583:c917:7ff0 with SMTP id m186-20020a0de3c3000000b00583c9177ff0mr11685000ywe.51.1696339992722; Tue, 03 Oct 2023 06:33:12 -0700 (PDT) Received: from ?IPV6:2600:1700:bd2c:b810:a9ff:8e0:a7ce:5aa0? ([2600:1700:bd2c:b810:a9ff:8e0:a7ce:5aa0]) by smtp.googlemail.com with ESMTPSA id i132-20020a816d8a000000b005a237c85821sm379403ywc.110.2023.10.03.06.33.12 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 03 Oct 2023 06:33:12 -0700 (PDT) Content-Type: multipart/alternative; boundary="------------196WIZwsITPf1WZ79TbmZ1mF" Message-ID: <07295fb2-aaef-8d3c-18ef-ffd8cdb1de11@gmail.com> Date: Tue, 3 Oct 2023 08:33:12 -0500 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.11.0 Subject: Re: Problems starting slave Content-Language: en-US To: pgsql-general@lists.postgresql.org References: <1263466497.931257.1696257526649.ref@mail.yahoo.com> <1263466497.931257.1696257526649@mail.yahoo.com> From: Ron In-Reply-To: <1263466497.931257.1696257526649@mail.yahoo.com> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------196WIZwsITPf1WZ79TbmZ1mF Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit On 10/2/23 09:38, Douglas Reed wrote: > Hi guys > > The servers are virtual running on Nutanix > > We are running Pg version 12 (12.10) > > On Linux km-data1.rs.fsbtech.com 5.4.191-1.el7.elrepo.x86_64 #1 SMP Tue > Apr 26 12:14:16 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux > > 48G/16 x CPU (Master and slave) > > Timeline > > System had a number of issues due to kafka and slots, would not always > shut down correctly. Several incidents where it had to be forced. > Found corruption on a number of indexes and tables. Decided to recover > from a backup (barman). Due to missing wal file restored data up > to about three hours prior to expectation but it's working. > > Attempts to build a slave; > > On the slave at first we got error messages from pg_basebackup stating > that the target directory was not empty; > >     pg_basebackup: error: directory "/var/lib/pgsql/12/data" exists but is > not empty > > Although the directory /var/lib/pgsql/12/data was empty (using rm -r > ....). Finally deleted the data directory and re-created ensuring that > perms were same. Restore started successfully and completed with error=0. > What was the full pg_basebackup command, including all options? -- Born in Arizona, moved to Babylonia. --------------196WIZwsITPf1WZ79TbmZ1mF Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit On 10/2/23 09:38, Douglas Reed wrote:
Hi guys

The servers are virtual running on Nutanix

We are running Pg version 12 (12.10)

On Linux km-data1.rs.fsbtech.com 5.4.191-1.el7.elrepo.x86_64 #1 SMP Tue Apr 26 12:14:16 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

48G/16 x CPU (Master and slave)

Timeline

System had a number of issues due to kafka and slots, would not always shut down correctly. Several incidents where it had to be forced.
Found corruption on a number of indexes and tables. Decided to recover from a backup (barman). Due to missing wal file restored data up
to about three hours prior to expectation but it's working.

Attempts to build a slave;

On the slave at first we got error messages from pg_basebackup stating that the target directory was not empty;

    pg_basebackup: error: directory "/var/lib/pgsql/12/data" exists but is not empty

Although the directory /var/lib/pgsql/12/data was empty (using rm -r ....). Finally deleted the data directory and re-created ensuring that
perms were same. Restore started successfully and completed with error=0.


What was the full pg_basebackup command, including all options?

--
Born in Arizona, moved to Babylonia.
--------------196WIZwsITPf1WZ79TbmZ1mF--