Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r0NY5-005JQm-Tv for pgsql-admin@arkaria.postgresql.org; Tue, 07 Nov 2023 14:59:26 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r0NY4-008wju-Hn for pgsql-admin@arkaria.postgresql.org; Tue, 07 Nov 2023 14:59:24 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r0NY4-008wjm-2R for pgsql-admin@lists.postgresql.org; Tue, 07 Nov 2023 14:59:24 +0000 Received: from mail-yb1-xb35.google.com ([2607:f8b0:4864:20::b35]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1r0NY1-004PmC-Jq for pgsql-admin@lists.postgresql.org; Tue, 07 Nov 2023 14:59:22 +0000 Received: by mail-yb1-xb35.google.com with SMTP id 3f1490d57ef6-d9ac9573274so6055017276.0 for ; Tue, 07 Nov 2023 06:59:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1699369160; x=1699973960; darn=lists.postgresql.org; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:from:to:cc:subject:date:message-id :reply-to; bh=4HGNCNuwVn4zznTaGF0NmmIyUNxcJ7hqZus7HjyvDLo=; b=jJxrbm/2gtlw7D8QUb6gjLKsdGQbAGjnF08QDB3JWvNqh/TQphWayJRZr/K3FTB96l 2lngjNaY68Okm6hAslC95Z6D22XV7bYRPNKAYOs+9lTEBbPVEPAkLE4xJnjKbb/HHFUi Cpf5ZPrea1jB56p3qAQ3YQt+6wBMiYf7L3hkMlatm7Og9JrOeaAROZZf4yBi/Bu+aWiC TbwZq27R5PgvjjKJAHh9O0OuanQOVAKM6d7FY2KVHcKAaPwpxHSMuIshzm+7AHyAa3gk r+rg2E32GDIKo52we32VFf5fOrmGcIIvhkR2lwFlgZrrbZUrEYnXPYCE8RE/1P0Iv/+T 9EQw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1699369160; x=1699973960; h=in-reply-to:from:references:to:content-language:subject:user-agent :mime-version:date:message-id:x-gm-message-state:from:to:cc:subject :date:message-id:reply-to; bh=4HGNCNuwVn4zznTaGF0NmmIyUNxcJ7hqZus7HjyvDLo=; b=VQbPk2YXWQi0CV9wUBJnSbFWYZwHDh4O4PsTjVqCKlEjJbC1mfbWPlsg0ooxccUd2E yG5owBazP24UkxF7v/TOyJeBw3gS9m1TGOuE9u7pzhqad0hw/YwO/t8tMGhStk66JVI2 3EL6Wj/rZsAg3TwerelrYB7Msm5sZtk4kcWuKeWwcZFB7UJQCx01bmJvzKXrN/heccfB cza+7XcrnaFtXtPKd83l8LJBM+6Et9fU9B6I4UUj5deneBrMWgK1zjWnCiGS1Iu4jmhI zFj5K4Un51poMItNKSwha4T+uxgwM+nRVfmvKYKKA020xEkHeyXsVm4c57H+0lowZee6 a1ag== X-Gm-Message-State: AOJu0Yy3INrJVotEhNmi+/23/ILlr5+wjcJE5I30li2oLdT3WcjEOeaD XW/xp8poNUTK8vpOJOqQ1fZlwkCkgsI= X-Google-Smtp-Source: AGHT+IHHWp6flOX5hR9eeQIvMo7/ddgOCkITknycysqABkOWDtvVh2Un2Vss7194QzWZH31gPAbgyw== X-Received: by 2002:a25:8292:0:b0:d9a:d91f:3888 with SMTP id r18-20020a258292000000b00d9ad91f3888mr32575498ybk.13.1699369160320; Tue, 07 Nov 2023 06:59:20 -0800 (PST) Received: from ?IPV6:2600:1700:bd2c:b810::44? ([2600:1700:bd2c:b810::44]) by smtp.googlemail.com with ESMTPSA id v13-20020a25fc0d000000b00d9caf6f7539sm5275510ybd.1.2023.11.07.06.59.19 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 07 Nov 2023 06:59:20 -0800 (PST) Content-Type: multipart/alternative; boundary="------------SHHnbLZSWyeIlJYbA0pl5QGR" Message-ID: <678bb3fe-eead-454e-8c8c-5e0fb49100af@gmail.com> Date: Tue, 7 Nov 2023 08:59:19 -0600 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: Frequent failover Content-Language: en-US To: Pgsql-admin References: <9cabf7bc-e98f-449c-82ce-446727ff4814@gmail.com> From: Ron In-Reply-To: List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------SHHnbLZSWyeIlJYbA0pl5QGR Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Crank up the log level in both Postgresql and Patroni. On 11/7/23 07:31, Rajesh Kumar wrote: > I saw only the follow in postgres logs. > > 57P01 terminating due to administrative command > 57P03 database shutting down > ..then "received fast shutdown request",,,,postmaster > > > > On Tue, 7 Nov 2023, 18:39 Ron, wrote: > > Have you scoured the Patroni and Postgresql logs? > > Have you increased the logging level? > > On 11/6/23 23:44, Rajesh Kumar wrote: >> Is there a way to narrow down the cause of failover? Yesterday itself >> failover happened 7 times. >> >> On Mon, 6 Nov, 2023, 1:34 PM kaido vaikla, >> wrote: >> >> Hi >> First of all, read a logs. >> $PGDATA/log >> #journalctl -u patroni >> I had once failover flapping, because of master heavy load, >> caused by apliccation change. So, find a pattern. >> >> If physical replication, after failover you don't need run a >> vacuum manually if autovacuum is enabled. >> >> br >> Kaido >> >> On Mon, 6 Nov 2023 at 04:53, Rajesh Kumar >> wrote: >> >> Hi, >> >> When I issue command patronictl history, I can see frequent >> failover. Like atleast weekly once. >> >> Every time this happened we have to do vacuum analyse for all >> dba. >> >> So, I need assistance in how to avoid frequent failovers and >> what is the solution for vacuum part we r doing.. >> > > -- > Born in Arizona, moved to Babylonia. > -- Born in Arizona, moved to Babylonia. --------------SHHnbLZSWyeIlJYbA0pl5QGR Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 7bit Crank up the log level in both Postgresql and Patroni.

On 11/7/23 07:31, Rajesh Kumar wrote:
I saw only the follow in postgres logs.

57P01 terminating due to administrative command
57P03 database shutting down
..then "received fast shutdown request",,,,postmaster



On Tue, 7 Nov 2023, 18:39 Ron, <ronljohnsonjr@gmail.com> wrote:
Have you scoured the Patroni and Postgresql logs?

Have you increased the logging level?

On 11/6/23 23:44, Rajesh Kumar wrote:
Is there a way to narrow down the cause of failover? Yesterday itself failover happened 7 times.

On Mon, 6 Nov, 2023, 1:34 PM kaido vaikla, <kaido.vaikla@gmail.com> wrote:
Hi
First of all, read a logs.
$PGDATA/log
#journalctl -u patroni
I had once failover flapping, because of master heavy load, caused by apliccation change. So, find a pattern.

If physical replication, after failover you don't need run a vacuum manually if autovacuum is enabled.

br
Kaido

On Mon, 6 Nov 2023 at 04:53, Rajesh Kumar <rajeshkumar.dba09@gmail.com> wrote:
Hi,

When I issue command patronictl history, I can see frequent failover. Like atleast weekly once.

Every time this happened we have to do vacuum analyse for all dba.

So, I need assistance in how to avoid frequent failovers and what is the solution for vacuum part we r doing..

--
Born in Arizona, moved to Babylonia.

--
Born in Arizona, moved to Babylonia.
--------------SHHnbLZSWyeIlJYbA0pl5QGR--