public inbox for [email protected]
help / color / mirror / Atom feedFrom: Tayyab Fayyaz <[email protected]>
To: Pavan Kumar <[email protected]>
Cc: Fernando Hevia <[email protected]>
Cc: Chris Lee <[email protected]>
Cc: Imran Khan <[email protected]>
Cc: pgsql-admin <[email protected]>
Subject: Re: repmgr cannot bring up the standby database after switchover manaully
Date: Fri, 3 Oct 2025 18:51:40 +0500
Message-ID: <CAFVRaQ3wLJQj6dpT7zm7fDsUBMJHwVeQ3GQJJB_fUXL43EkdSg@mail.gmail.com> (raw)
In-Reply-To: <CA+M0sHFfKRd1VTp8PZz0fD9CB=SXXsvc7HAY6Py4A+aUWSnQFQ@mail.gmail.com>
References: <CAP0GqH7hYsFGmLg3Ge-_Lf_VEeES8yeHpOhSvn071=XRnmVxTw@mail.gmail.com>
<CAFVRaQ3No2rEYjkuKsxNNMCP16_J_kjDm_pv0+CYwTLXsjE8OQ@mail.gmail.com>
<CAC4eXDgNZM12t8OgKYhjbu_7NFcsu0qG5aTpdXmbN=fmmp-5Tw@mail.gmail.com>
<CAP0GqH7oRXztf1Ti13usH9CEgBmMJ4mPMsOn+qacD8Adi+Dxew@mail.gmail.com>
<CAGYT1XRLUstysEBwWKNVVWrUJCfahCFgmzEkFwpjQmABOaMocg@mail.gmail.com>
<CAFVRaQ0zz-81jReYtOEo2pHXD0GzZ5sr6=mL7nOQ_EBZkRh8MA@mail.gmail.com>
<CAGYT1XTZOMQ88zfN7T76RJTam47XyX-U1V_CqTTVhk2f5LVObA@mail.gmail.com>
<CA+M0sHFfKRd1VTp8PZz0fD9CB=SXXsvc7HAY6Py4A+aUWSnQFQ@mail.gmail.com>
Hello Pavan,
Please share required parameters for PostgreSQL, I will compare with my
existing configuration.
As I understand, automatically re-adding the old primary as a standby is
not an out-of-the-box feature and needs to be handled manually. Is that
correct?
Tayyab
On Fri, 3 Oct 2025, 6:20 pm Pavan Kumar, <[email protected]> wrote:
> Hello Chris,
>
> I hope you configured required parameters in PostgreSQL. I do noticed the
> same issue when your primary is Idle (no activity).
> Before doing switchover please perform checkpoint on primary and run
> switchover command.
> review repmgr -f repmgr.conf cluster events , this will provide more
> information on what happened during switchover.
>
> Note: Make sure repmgr daemon are running and not in pause mode before
> switchover .
>
>
>
>
> On Wed, Oct 1, 2025 at 3:03 PM Fernando Hevia <[email protected]> wrote:
>
>>
>>
>>> In my recent experience, there was no issue starting the old primary—it
>>> came up normally. However, it resulted in a split-brain situation where the
>>> old primary continued to accept both read and write operations while still
>>> assuming the other two nodes were replicas.
>>
>>
>> Hi Tayyab,
>>
>> A split-brain is definitely an unexpected behavior. After issuing a
>> failover or switchover command, always check the exit code to ensure it was
>> successful. If not, you should find in the command output or in the
>> postgresql logs an indication of what went wrong.
>>
>> Seems that either the previous primary couldn't be shutdown or repmgr
>> failed somehow to change it to a standby. Repmgr sets the node's role by
>> creating the standby.signal file in the data directory. Upon startup, if
>> Postgres finds the signal file, it will assume the standby role (providing
>> the postgresql.conf file has the correct configuration too). I can only
>> theorize here, but maybe repmgr failed to write the signal file in $PGDATA
>> either due to lack of permissions or a network failure.
>>
>> The exact output would help in figuring out what went wrong.
>>
>> Regards,
>> Fernando
>>
>>
>>
>>
>>
>> El mié, 1 oct 2025 a la(s) 4:04 p.m., Tayyab Fayyaz (
>> [email protected]) escribió:
>>
>>> Hello Fernando,
>>>
>>> In my recent experience, there was no issue starting the old primary—it
>>> came up normally. However, it resulted in a split-brain situation where the
>>> old primary continued to accept both read and write operations while still
>>> assuming the other two nodes were replicas.
>>>
>>> This issue occurred with the following environment:
>>>
>>> -
>>>
>>> *OS version:* RHEL 8.10
>>> -
>>>
>>> *Postgres DB version:* 14.9
>>> -
>>>
>>> *repmgr version:* 5.5.0
>>>
>>> Tayyab
>>>
>>> On Wed, Oct 1, 2025 at 11:52 AM Fernando Hevia <[email protected]> wrote:
>>>
>>>>
>>>> I have 2 postgresql servers. One is the primary and another one is the
>>>>> standby. I am trying to setup repmgr to do the switchover manually.
>>>>> Passwordless ssh have been setup for postgres ID on both servers.
>>>>>
>>>>> I use this command "repmgr standby switchover --log-level=DEBUG
>>>>> --verbose". The standy database is able to promote to be the primary. For
>>>>> the previous primary database, it was shutdown. It was not able to bring up
>>>>> as standby by repmgr.
>>>>
>>>>
>>>> In a switchover the primary server is shutdown and restarted as a
>>>> standby server after the newly promoted primary (former secondary) node has
>>>> been started.
>>>> If the primary did not start, there must have been an issue since this
>>>> is not the standard behavior for a switchover command.
>>>>
>>>> Have you checked the Postgres log file for the previous primary? You
>>>> should find the startup failure cause in the log.
>>>>
>>>> Regards,
>>>> Fernando
>>>>
>>>>
>>>>
>>>> El mié, 1 oct 2025 a la(s) 7:30 a.m., Chris Lee ([email protected])
>>>> escribió:
>>>>
>>>>> Hi Tayyab,
>>>>>
>>>>> Thanks for your information . I also want to find out whether that is
>>>>> the default behavior, or I am not configuring repmgr correctly.
>>>>>
>>>>> Regards,
>>>>> Chris
>>>>>
>>>>> On Wed, 1 Oct 2025, 18:12 Imran Khan, <[email protected]> wrote:
>>>>>
>>>>>> Hi Tayyab,
>>>>>>
>>>>>> Is this a default behavior? We have 4 nodes cluster but never had
>>>>>> issue in switchovers.
>>>>>>
>>>>>> Thanks,
>>>>>> Imran
>>>>>>
>>>>>> On Wed, Oct 1, 2025, 1:10 PM Tayyab Fayyaz <[email protected]>
>>>>>> wrote:
>>>>>>
>>>>>>> Hello Chris,
>>>>>>>
>>>>>>> I faced this issue it will not add automatically as standby you have
>>>>>>> to add it manually.
>>>>>>>
>>>>>>> But I wrote a script which perform to add old primary as standby
>>>>>>> once it's back online.
>>>>>>>
>>>>>>> Tayyab
>>>>>>>
>>>>>>>
>>>>>>> On Wed, 1 Oct 2025, 3:02 pm Chris Lee, <[email protected]> wrote:
>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> I have 2 postgresql servers. One is the primary and another one is
>>>>>>>> the standby. I am trying to setup repmgr to do the switchover manually.
>>>>>>>> Passwordless ssh have been setup for postgres ID on both servers.
>>>>>>>>
>>>>>>>> I use this command "repmgr standby switchover --log-level=DEBUG
>>>>>>>> --verbose". The standy database is able to promote to be the primary. For
>>>>>>>> the previous primary database, it was shutdown. It was not able to bring up
>>>>>>>> as standby by repmgr.
>>>>>>>>
>>>>>>>> Does anyone encounter this issue before? Thanks a lot for any
>>>>>>>> suggestions.
>>>>>>>>
>>>>>>>> Here is my OS and DB versions:
>>>>>>>>
>>>>>>>> OS version: CentOS Stream release 8
>>>>>>>> Postgres DB version: 15.12
>>>>>>>> rempmgr version: 5.5.0
>>>>>>>>
>>>>>>>> Here is the repmgr conf files:
>>>>>>>> >>>>>
>>>>>>>> node_id=1 # Use 2 on standby
>>>>>>>> node_name='primary'
>>>>>>>> conninfo='host=centos804 user=repmgr dbname=repmgr password=xxx
>>>>>>>> connect_timeout=15'
>>>>>>>> use_primary_conninfo_password=true
>>>>>>>> data_directory='/var/lib/pgsql/15/data' # Adjust for your setup
>>>>>>>> pg_bindir='/usr/pgsql-15/bin'
>>>>>>>> service_start_command = 'sudo systemctl start postgresql-15'
>>>>>>>> service_stop_command = 'sudo systemctl stop postgresql-15'
>>>>>>>> <<<<<
>>>>>>>>
>>>>>>>> >>>>>
>>>>>>>> node_id=2 # Use 2 on standby
>>>>>>>> node_name='standby'
>>>>>>>> conninfo='host=centos803 user=repmgr dbname=repmgr password=xxx
>>>>>>>> connect_timeout=15'
>>>>>>>> use_primary_conninfo_password=true
>>>>>>>> data_directory='/var/lib/pgsql/15/data' # Adjust for your setup
>>>>>>>> pg_bindir='/usr/pgsql-15/bin'
>>>>>>>> service_start_command = 'sudo systemctl start postgresql-15'
>>>>>>>> service_stop_command = 'sudo systemctl stop postgresql-15'
>>>>>>>> <<<<<
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Chris
>>>>>>>>
>>>>>>>
>
> --
>
>
>
> *Regards,#! Pavan Kumar----------------------------------------------*-
> *Sr. Database Administrator..!*
> *NEXT GENERATION PROFESSIONALS, LLC*
> *Cell # 267-799-3182 # pavan.dba27 (Gtalk) *
> *India # 9000459083*
>
> *Take Risks; if you win, you will be very happy. If you lose you will be
> Wise *
>
>
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected], [email protected]
Subject: Re: repmgr cannot bring up the standby database after switchover manaully
In-Reply-To: <CAFVRaQ3wLJQj6dpT7zm7fDsUBMJHwVeQ3GQJJB_fUXL43EkdSg@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox