pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedWould you ever recommend Shared Disk Failover for HA?
7+ messages / 5 participants
[nested] [flat]
* Would you ever recommend Shared Disk Failover for HA?
@ 2024-02-22 19:34 norbert poellmann <np@ibu.de>
2024-02-22 19:49 ` Re: Would you ever recommend Shared Disk Failover for HA? Ron Johnson <ronljohnsonjr@gmail.com>
2024-02-23 07:02 ` Re: Would you ever recommend Shared Disk Failover for HA? Laurenz Albe <laurenz.albe@cybertec.at>
2024-02-26 19:24 ` Re: Would you ever recommend Shared Disk Failover for HA? Stephen Frost <sfrost@snowman.net>
0 siblings, 3 replies; 7+ messages in thread
From: norbert poellmann @ 2024-02-22 19:34 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Admins,
https://www.postgresql.org/docs/current/different-replication-solutions.html
is listing a shared disk solution for HA.
It also mentions, "that the standby server should never access the shared storage while the primary server is running."
In a datacenter, where we have postgresql servers running on vmware VMs, the shared disk configuration sounds like an appealing solution: simple configuration, single server at a given time, simple fail-over, fully non-or-nothing write mechanics, no hazzle with replication_slots during/after failover, etc...
But "Attempts to use PostgreSQL in multi-master shared storage configurations will result in extremely severe data corruption" (https://wiki.postgresql.org/wiki/Shared_Storage).
So it seems to me, getting the comfort of a single server solution, which, in a failover, gets replaced by another single server, is paid by getting the low risk of high damage.
I know of the provisions of fencing, STONITH, etc. - but in practise, what is a robust solution?
For example: How can I STONITH a node while having network problems?
Whithout reaching the host, I cannot shoot it, nor shut it.
I also cannot wait for it get visible in the network again: another client might have interfered and commited a transaction upon the retired master server, faster than I can do some fencing/stonith or whatever.
Would you share your opinions or practical business experiences on this topic?
Thanks a lot
cheers
Norbert Poellmann
--
Norbert Poellmann EDV-Beratung email : np@ibu.de
Severinstrasse 5 telefon: 089 38469995
81541 Muenchen, Germany telefon: 0179 2133436
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Would you ever recommend Shared Disk Failover for HA?
2024-02-22 19:34 Would you ever recommend Shared Disk Failover for HA? norbert poellmann <np@ibu.de>
@ 2024-02-22 19:49 ` Ron Johnson <ronljohnsonjr@gmail.com>
2 siblings, 0 replies; 7+ messages in thread
From: Ron Johnson @ 2024-02-22 19:49 UTC (permalink / raw)
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Thu, Feb 22, 2024 at 2:35 PM norbert poellmann <np@ibu.de> wrote:
> Admins,
>
>
> https://www.postgresql.org/docs/current/different-replication-solutions.html
> is listing a shared disk solution for HA.
>
> It also mentions, "that the standby server should never access the shared
> storage while the primary server is running."
>
> In a datacenter, where we have postgresql servers running on vmware VMs,
> the shared disk configuration sounds like an appealing solution: simple
> configuration, single server at a given time, simple fail-over, fully
> non-or-nothing write mechanics, no hazzle with replication_slots
> during/after failover, etc...
>
> But "Attempts to use PostgreSQL in multi-master shared storage
> configurations will result in extremely severe data corruption" (
> https://wiki.postgresql.org/wiki/Shared_Storage).
>
Our DB servers are also VMware VMs, with the disks managed by VMware, too.
If a blade dies, the VM automatically restarts on a different blade.
(Heck, ESX might automagically migrate it with no downtime. I've never
*known* this to happen, but I don't have access to the VMware console; they
just stay up for months, getting migrated around as necessary for load
management.)
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Would you ever recommend Shared Disk Failover for HA?
2024-02-22 19:34 Would you ever recommend Shared Disk Failover for HA? norbert poellmann <np@ibu.de>
@ 2024-02-23 07:02 ` Laurenz Albe <laurenz.albe@cybertec.at>
2024-02-23 07:21 ` Re: Would you ever recommend Shared Disk Failover for HA? Ron Johnson <ronljohnsonjr@gmail.com>
2024-02-23 07:38 ` Re: Would you ever recommend Shared Disk Failover for HA? vignesh kumar <vigneshkumar.venugopal@outlook.com>
2 siblings, 2 replies; 7+ messages in thread
From: Laurenz Albe @ 2024-02-23 07:02 UTC (permalink / raw)
To: norbert poellmann <np@ibu.de>; pgsql-admin@lists.postgresql.org
On Thu, 2024-02-22 at 20:34 +0100, norbert poellmann wrote:
> https://www.postgresql.org/docs/current/different-replication-solutions.html
> is listing a shared disk solution for HA.
>
> It also mentions, "that the standby server should never access the shared storage
> while the primary server is running."
>
> In a datacenter, where we have postgresql servers running on vmware VMs, the
> shared disk configuration sounds like an appealing solution
>
> But [...]
>
> So it seems to me, getting the comfort of a single server solution, which, in a
> failover, gets replaced by another single server, is paid by getting the low risk
> of high damage.
>
> I know of the provisions of fencing, STONITH, etc. - but in practise, what is a robust solution?
>
> For example: How can I STONITH a node while having network problems?
> Whithout reaching the host, I cannot shoot it, nor shut it.
>
> Would you share your opinions or practical business experiences on this topic?
Back in the old days, we had special hardware devices for STONITH.
Anyway, my personal experience with a shared disk setup is a bad one.
Imagine two nodes, redundantly attached to disks mirrored across data
centers with fibrechannel. No single point of failure, right?
Well, one day one of the fibrechannel cables had intermittent failures,
which led to a corrupted file system.
So we ended up with a currupted file system, nicely mirrored across
data centers. We had to restore the 3TB database from backup.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Would you ever recommend Shared Disk Failover for HA?
2024-02-22 19:34 Would you ever recommend Shared Disk Failover for HA? norbert poellmann <np@ibu.de>
2024-02-23 07:02 ` Re: Would you ever recommend Shared Disk Failover for HA? Laurenz Albe <laurenz.albe@cybertec.at>
@ 2024-02-23 07:21 ` Ron Johnson <ronljohnsonjr@gmail.com>
2024-02-23 09:08 ` Re: Would you ever recommend Shared Disk Failover for HA? Laurenz Albe <laurenz.albe@cybertec.at>
1 sibling, 1 reply; 7+ messages in thread
From: Ron Johnson @ 2024-02-23 07:21 UTC (permalink / raw)
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Fri, Feb 23, 2024 at 2:02 AM Laurenz Albe <laurenz.albe@cybertec.at>
wrote:
[snip]
>
> Anyway, my personal experience with a shared disk setup is a bad one.
>
> Imagine two nodes, redundantly attached to disks mirrored across data
> centers with fibrechannel. No single point of failure, right?
> Well, one day one of the fibrechannel cables had intermittent failures,
> which led to a corrupted file system.
> So we ended up with a currupted file system, nicely mirrored across
> data centers. We had to restore the 3TB database from backup.
>
1. Sounds like both nodes were turned on.
2. Couldn't this happen in /any/ SAN with redundant cabling?
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Would you ever recommend Shared Disk Failover for HA?
2024-02-22 19:34 Would you ever recommend Shared Disk Failover for HA? norbert poellmann <np@ibu.de>
2024-02-23 07:02 ` Re: Would you ever recommend Shared Disk Failover for HA? Laurenz Albe <laurenz.albe@cybertec.at>
2024-02-23 07:21 ` Re: Would you ever recommend Shared Disk Failover for HA? Ron Johnson <ronljohnsonjr@gmail.com>
@ 2024-02-23 09:08 ` Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 0 replies; 7+ messages in thread
From: Laurenz Albe @ 2024-02-23 09:08 UTC (permalink / raw)
To: Ron Johnson <ronljohnsonjr@gmail.com>; Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Fri, 2024-02-23 at 02:21 -0500, Ron Johnson wrote:
> On Fri, Feb 23, 2024 at 2:02 AM Laurenz Albe <laurenz.albe@cybertec.at> wrote:
> [snip]
> >
> > Anyway, my personal experience with a shared disk setup is a bad one.
> >
> > Imagine two nodes, redundantly attached to disks mirrored across data
> > centers with fibrechannel. No single point of failure, right?
> > Well, one day one of the fibrechannel cables had intermittent failures,
> > which led to a corrupted file system.
> > So we ended up with a currupted file system, nicely mirrored across
> > data centers. We had to restore the 3TB database from backup.
>
> 1. Sounds like both nodes were turned on.
Possible, but I think that's not relevant.
> 2. Couldn't this happen in /any/ SAN with redundant cabling?
I'd say yes, and the redundant cabling is irrelevant too - I just
wanted to emphasize that hardware redundancy is not enough.
The single point of failure was the file system.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Would you ever recommend Shared Disk Failover for HA?
2024-02-22 19:34 Would you ever recommend Shared Disk Failover for HA? norbert poellmann <np@ibu.de>
2024-02-23 07:02 ` Re: Would you ever recommend Shared Disk Failover for HA? Laurenz Albe <laurenz.albe@cybertec.at>
@ 2024-02-23 07:38 ` vignesh kumar <vigneshkumar.venugopal@outlook.com>
1 sibling, 0 replies; 7+ messages in thread
From: vignesh kumar @ 2024-02-23 07:38 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; norbert poellmann <np@ibu.de>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
With the knowledge and design architecture of Postgres engine. It is not a redundant solution to have a shared storage for a huge database cluster. Here is why?
* When a cluster directory is initiated - It has checksums based on the system and the WAL block size.
* so the database engine shared object files and its related binary system call will make an attempt to write to disk or read to disk.
*
so this has the file lock enabled on the data cluster.
*
now imagine that you have a shared storage.
*
since more than one postmaster process trying to write to it. will lead to file buffer ( conflicts )
*
you will end up with inode corruption
*
Ideally by design it is not possible to have them enabled.
*
that is the reason is designed in such a way to hold the data cluster directory implicit.
*
Now consider the below scenario.
*
Lets assume we have a shared storage space.
*
INODE buffer will try to page write on cluster database directory
*
Say allocated MEMBUFF Address [x000Fxxxxx] - some address in memory.
*
This will be refered before writing to the disk. as it will first map the inode index to flush the data.
*
So when another postmaster process attempts to write to the file system
*
It will over ride the MEMBUFF
*
leading to page corruption / conflict.
This case would be more worse to repair the corrupted data at page level.
* doing stacktrace
* fetch the disk sector which is affected.
* fetch all systemcalls affecting the data page
*
make the page invisible
*
fix / remove the data page
Hence it doesnt have it by default.
Just sharing my thoughts. hope this helps you
Thanks & Regards,
Viggu
________________________________
From: Laurenz Albe <laurenz.albe@cybertec.at>
Sent: Friday, February 23, 2024 12:32 PM
To: norbert poellmann <np@ibu.de>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Subject: Re: Would you ever recommend Shared Disk Failover for HA?
On Thu, 2024-02-22 at 20:34 +0100, norbert poellmann wrote:
> https://www.postgresql.org/docs/current/different-replication-solutions.html
> is listing a shared disk solution for HA.
>
> It also mentions, "that the standby server should never access the shared storage
> while the primary server is running."
>
> In a datacenter, where we have postgresql servers running on vmware VMs, the
> shared disk configuration sounds like an appealing solution
>
> But [...]
>
> So it seems to me, getting the comfort of a single server solution, which, in a
> failover, gets replaced by another single server, is paid by getting the low risk
> of high damage.
>
> I know of the provisions of fencing, STONITH, etc. - but in practise, what is a robust solution?
>
> For example: How can I STONITH a node while having network problems?
> Whithout reaching the host, I cannot shoot it, nor shut it.
>
> Would you share your opinions or practical business experiences on this topic?
Back in the old days, we had special hardware devices for STONITH.
Anyway, my personal experience with a shared disk setup is a bad one.
Imagine two nodes, redundantly attached to disks mirrored across data
centers with fibrechannel. No single point of failure, right?
Well, one day one of the fibrechannel cables had intermittent failures,
which led to a corrupted file system.
So we ended up with a currupted file system, nicely mirrored across
data centers. We had to restore the 3TB database from backup.
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Would you ever recommend Shared Disk Failover for HA?
2024-02-22 19:34 Would you ever recommend Shared Disk Failover for HA? norbert poellmann <np@ibu.de>
@ 2024-02-26 19:24 ` Stephen Frost <sfrost@snowman.net>
2 siblings, 0 replies; 7+ messages in thread
From: Stephen Frost @ 2024-02-26 19:24 UTC (permalink / raw)
To: norbert poellmann <np@ibu.de>; +Cc: pgsql-admin@lists.postgresql.org
Greetings,
* norbert poellmann (np@ibu.de) wrote:
> https://www.postgresql.org/docs/current/different-replication-solutions.html
> is listing a shared disk solution for HA.
Yeah. Frankly, it's bad advice and we should remove it. "Rapid
failover" is a bit laughable compared to replication when you consider
that crash recovery can take a very, very long time (depending on how
much outstanding WAL has been written since the last checkpoint but with
extended checkpoints and single-process WAL replay, crash recovery could
be on the order of hours ...) and promoting an online replica takes only
moments.
Ditto for block-based replication.
Probably should talk about WAL shipping more as "Physical Replication".
At the least, physical replication should really be listed first and
then logical replication, perhaps even in a distinct "included as part
of PostgreSQL" section with everything else pushed down to "some other
things exist that you could try"...
Thanks,
Stephen
Attachments:
[application/pgp-signature] signature.asc (832B, ../../ZdzlafGnXC4fQAww@tamriel.snowman.net/2-signature.asc)
download
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2024-02-26 19:24 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-02-22 19:34 Would you ever recommend Shared Disk Failover for HA? norbert poellmann <np@ibu.de>
2024-02-22 19:49 ` Ron Johnson <ronljohnsonjr@gmail.com>
2024-02-23 07:02 ` Laurenz Albe <laurenz.albe@cybertec.at>
2024-02-23 07:21 ` Ron Johnson <ronljohnsonjr@gmail.com>
2024-02-23 09:08 ` Laurenz Albe <laurenz.albe@cybertec.at>
2024-02-23 07:38 ` vignesh kumar <vigneshkumar.venugopal@outlook.com>
2024-02-26 19:24 ` Stephen Frost <sfrost@snowman.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox