public inbox for [email protected]
help / color / mirror / Atom feedFrom: Pavan Kumar <[email protected]>
To: Fernando Hevia <[email protected]>
Cc: Tayyab Fayyaz <[email protected]>
Cc: Chris Lee <[email protected]>
Cc: Imran Khan <[email protected]>
Cc: pgsql-admin <[email protected]>
Subject: Re: repmgr cannot bring up the standby database after switchover manaully
Date: Fri, 3 Oct 2025 08:19:50 -0500
Message-ID: <CA+M0sHFfKRd1VTp8PZz0fD9CB=SXXsvc7HAY6Py4A+aUWSnQFQ@mail.gmail.com> (raw)
In-Reply-To: <CAGYT1XTZOMQ88zfN7T76RJTam47XyX-U1V_CqTTVhk2f5LVObA@mail.gmail.com>
References: <CAP0GqH7hYsFGmLg3Ge-_Lf_VEeES8yeHpOhSvn071=XRnmVxTw@mail.gmail.com>
<CAFVRaQ3No2rEYjkuKsxNNMCP16_J_kjDm_pv0+CYwTLXsjE8OQ@mail.gmail.com>
<CAC4eXDgNZM12t8OgKYhjbu_7NFcsu0qG5aTpdXmbN=fmmp-5Tw@mail.gmail.com>
<CAP0GqH7oRXztf1Ti13usH9CEgBmMJ4mPMsOn+qacD8Adi+Dxew@mail.gmail.com>
<CAGYT1XRLUstysEBwWKNVVWrUJCfahCFgmzEkFwpjQmABOaMocg@mail.gmail.com>
<CAFVRaQ0zz-81jReYtOEo2pHXD0GzZ5sr6=mL7nOQ_EBZkRh8MA@mail.gmail.com>
<CAGYT1XTZOMQ88zfN7T76RJTam47XyX-U1V_CqTTVhk2f5LVObA@mail.gmail.com>
Hello Chris,
I hope you configured required parameters in PostgreSQL. I do noticed the
same issue when your primary is Idle (no activity).
Before doing switchover please perform checkpoint on primary and run
switchover command.
review repmgr -f repmgr.conf cluster events , this will provide more
information on what happened during switchover.
Note: Make sure repmgr daemon are running and not in pause mode before
switchover .
On Wed, Oct 1, 2025 at 3:03 PM Fernando Hevia <[email protected]> wrote:
>
>
>> In my recent experience, there was no issue starting the old primary—it
>> came up normally. However, it resulted in a split-brain situation where the
>> old primary continued to accept both read and write operations while still
>> assuming the other two nodes were replicas.
>
>
> Hi Tayyab,
>
> A split-brain is definitely an unexpected behavior. After issuing a
> failover or switchover command, always check the exit code to ensure it was
> successful. If not, you should find in the command output or in the
> postgresql logs an indication of what went wrong.
>
> Seems that either the previous primary couldn't be shutdown or repmgr
> failed somehow to change it to a standby. Repmgr sets the node's role by
> creating the standby.signal file in the data directory. Upon startup, if
> Postgres finds the signal file, it will assume the standby role (providing
> the postgresql.conf file has the correct configuration too). I can only
> theorize here, but maybe repmgr failed to write the signal file in $PGDATA
> either due to lack of permissions or a network failure.
>
> The exact output would help in figuring out what went wrong.
>
> Regards,
> Fernando
>
>
>
>
>
> El mié, 1 oct 2025 a la(s) 4:04 p.m., Tayyab Fayyaz (
> [email protected]) escribió:
>
>> Hello Fernando,
>>
>> In my recent experience, there was no issue starting the old primary—it
>> came up normally. However, it resulted in a split-brain situation where the
>> old primary continued to accept both read and write operations while still
>> assuming the other two nodes were replicas.
>>
>> This issue occurred with the following environment:
>>
>> -
>>
>> *OS version:* RHEL 8.10
>> -
>>
>> *Postgres DB version:* 14.9
>> -
>>
>> *repmgr version:* 5.5.0
>>
>> Tayyab
>>
>> On Wed, Oct 1, 2025 at 11:52 AM Fernando Hevia <[email protected]> wrote:
>>
>>>
>>> I have 2 postgresql servers. One is the primary and another one is the
>>>> standby. I am trying to setup repmgr to do the switchover manually.
>>>> Passwordless ssh have been setup for postgres ID on both servers.
>>>>
>>>> I use this command "repmgr standby switchover --log-level=DEBUG
>>>> --verbose". The standy database is able to promote to be the primary. For
>>>> the previous primary database, it was shutdown. It was not able to bring up
>>>> as standby by repmgr.
>>>
>>>
>>> In a switchover the primary server is shutdown and restarted as a
>>> standby server after the newly promoted primary (former secondary) node has
>>> been started.
>>> If the primary did not start, there must have been an issue since this
>>> is not the standard behavior for a switchover command.
>>>
>>> Have you checked the Postgres log file for the previous primary? You
>>> should find the startup failure cause in the log.
>>>
>>> Regards,
>>> Fernando
>>>
>>>
>>>
>>> El mié, 1 oct 2025 a la(s) 7:30 a.m., Chris Lee ([email protected])
>>> escribió:
>>>
>>>> Hi Tayyab,
>>>>
>>>> Thanks for your information . I also want to find out whether that is
>>>> the default behavior, or I am not configuring repmgr correctly.
>>>>
>>>> Regards,
>>>> Chris
>>>>
>>>> On Wed, 1 Oct 2025, 18:12 Imran Khan, <[email protected]> wrote:
>>>>
>>>>> Hi Tayyab,
>>>>>
>>>>> Is this a default behavior? We have 4 nodes cluster but never had
>>>>> issue in switchovers.
>>>>>
>>>>> Thanks,
>>>>> Imran
>>>>>
>>>>> On Wed, Oct 1, 2025, 1:10 PM Tayyab Fayyaz <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Hello Chris,
>>>>>>
>>>>>> I faced this issue it will not add automatically as standby you have
>>>>>> to add it manually.
>>>>>>
>>>>>> But I wrote a script which perform to add old primary as standby once
>>>>>> it's back online.
>>>>>>
>>>>>> Tayyab
>>>>>>
>>>>>>
>>>>>> On Wed, 1 Oct 2025, 3:02 pm Chris Lee, <[email protected]> wrote:
>>>>>>
>>>>>>> Hi all,
>>>>>>>
>>>>>>> I have 2 postgresql servers. One is the primary and another one is
>>>>>>> the standby. I am trying to setup repmgr to do the switchover manually.
>>>>>>> Passwordless ssh have been setup for postgres ID on both servers.
>>>>>>>
>>>>>>> I use this command "repmgr standby switchover --log-level=DEBUG
>>>>>>> --verbose". The standy database is able to promote to be the primary. For
>>>>>>> the previous primary database, it was shutdown. It was not able to bring up
>>>>>>> as standby by repmgr.
>>>>>>>
>>>>>>> Does anyone encounter this issue before? Thanks a lot for any
>>>>>>> suggestions.
>>>>>>>
>>>>>>> Here is my OS and DB versions:
>>>>>>>
>>>>>>> OS version: CentOS Stream release 8
>>>>>>> Postgres DB version: 15.12
>>>>>>> rempmgr version: 5.5.0
>>>>>>>
>>>>>>> Here is the repmgr conf files:
>>>>>>> >>>>>
>>>>>>> node_id=1 # Use 2 on standby
>>>>>>> node_name='primary'
>>>>>>> conninfo='host=centos804 user=repmgr dbname=repmgr password=xxx
>>>>>>> connect_timeout=15'
>>>>>>> use_primary_conninfo_password=true
>>>>>>> data_directory='/var/lib/pgsql/15/data' # Adjust for your setup
>>>>>>> pg_bindir='/usr/pgsql-15/bin'
>>>>>>> service_start_command = 'sudo systemctl start postgresql-15'
>>>>>>> service_stop_command = 'sudo systemctl stop postgresql-15'
>>>>>>> <<<<<
>>>>>>>
>>>>>>> >>>>>
>>>>>>> node_id=2 # Use 2 on standby
>>>>>>> node_name='standby'
>>>>>>> conninfo='host=centos803 user=repmgr dbname=repmgr password=xxx
>>>>>>> connect_timeout=15'
>>>>>>> use_primary_conninfo_password=true
>>>>>>> data_directory='/var/lib/pgsql/15/data' # Adjust for your setup
>>>>>>> pg_bindir='/usr/pgsql-15/bin'
>>>>>>> service_start_command = 'sudo systemctl start postgresql-15'
>>>>>>> service_stop_command = 'sudo systemctl stop postgresql-15'
>>>>>>> <<<<<
>>>>>>>
>>>>>>> Regards,
>>>>>>> Chris
>>>>>>>
>>>>>>
--
*Regards,#! Pavan Kumar----------------------------------------------*-
*Sr. Database Administrator..!*
*NEXT GENERATION PROFESSIONALS, LLC*
*Cell # 267-799-3182 # pavan.dba27 (Gtalk) *
*India # 9000459083*
*Take Risks; if you win, you will be very happy. If you lose you will be
Wise *
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: repmgr cannot bring up the standby database after switchover manaully
In-Reply-To: <CA+M0sHFfKRd1VTp8PZz0fD9CB=SXXsvc7HAY6Py4A+aUWSnQFQ@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox