pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedhelp required for Datacenter Migration on Kubernetes Cluster
2+ messages / 2 participants
[nested] [flat]
* help required for Datacenter Migration on Kubernetes Cluster
@ 2024-07-14 18:33 BIPUL KHAN <bipul364@gmail.com>
0 siblings, 1 reply; 2+ messages in thread
From: BIPUL KHAN @ 2024-07-14 18:33 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hello,
We have developed and run a large-scale mobile financial system using a
microservice architecture, which includes 110 databases (big and small).
We use PostgreSQL as the primary database inside Kubernetes via the
CrunchyData operator.
Now I would like to share my problem. Our data storage has grown
significantly since 2020, and we are using Ceph (rook-Ceph). We now have 21
TB of data.
Our current situation is critical as we urgently need to move our data
center. Our source and destination data centers have a 1 GB/s data transfer
rate. We attempted to add a new Ceph node with the same cluster, which
started data rebalancing, but it’s taking a significant amount of time.
This directly and severely impacts our primary transactions, which involve
reading and writing to the database.
What is the best option for moving data from one data center to a new one?
We would appreciate your expertise. Would you happento know how to move it
any other way?
It is mentioned that the two different data centers are different
Kubernetes clusters.
I would appreciate some ideas. Feel free to let me know if you need any
other adjustments!
Let me know if there are any further changes you'd like!
Thanks & kind Regards
*----------------------------------- Md. Bipul Khan *
Mobile : +8801859556355
Skype: bipul364
Email: bipul364@gmail.com
https://www.linkedin.com/in/bipulkhan
^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: help required for Datacenter Migration on Kubernetes Cluster
@ 2024-07-14 19:06 Scott Ribe <scott_ribe@elevated-dev.com>
parent: BIPUL KHAN <bipul364@gmail.com>
0 siblings, 0 replies; 2+ messages in thread
From: Scott Ribe @ 2024-07-14 19:06 UTC (permalink / raw)
To: BIPUL KHAN <bipul364@gmail.com>; +Cc: pgsql-admin@lists.postgresql.org
> On Jul 14, 2024, at 12:33 PM, BIPUL KHAN <bipul364@gmail.com> wrote:
>
> Hello,
> We have developed and run a large-scale mobile financial system using a microservice architecture, which includes 110 databases (big and small). We use PostgreSQL as the primary database inside Kubernetes via the CrunchyData operator.
> Now I would like to share my problem. Our data storage has grown significantly since 2020, and we are using Ceph (rook-Ceph). We now have 21 TB of data.
> Our current situation is critical as we urgently need to move our data center. Our source and destination data centers have a 1 GB/s data transfer rate. We attempted to add a new Ceph node with the same cluster, which started data rebalancing, but it’s taking a significant amount of time. This directly and severely impacts our primary transactions, which involve reading and writing to the database.
> What is the best option for moving data from one data center to a new one? We would appreciate your expertise. Would you happento know how to move it any other way?
> It is mentioned that the two different data centers are different Kubernetes clusters.
> I would appreciate some ideas. Feel free to let me know if you need any other adjustments!
> Let me know if there are any further changes you'd like!
Plan to migrate off Ceph as soon as possible. It is simply not suited for this--yes, I am speaking from experience. The rebalancing will hammer you when you can least afford it. WekaFS, Pure Storage, heck NFS. Or node-local storage, XFS or ZFS. Really, just about anything else. (It might help if you clarify whether "move our data center" really means what I assumed, that you own hardware and are moving it, or whether you're going to another cloud provider.
You're probably going to have to migrate your databases one at a time--even if you temporarily upgraded your bandwidth to 10Gbps you'd have too long of a down time. So, one at a time, set up a standby streaming replica, then when they're all populated and up-to-date, take a very brief downtime (seconds) to switch over. This assumes that 1Gbps can actually handle your stream of normal updates with enough room left over to be transferring the base files. (Current versions of Crunchy support the notion of cross-cluster replication.)
^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2024-07-14 19:06 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2024-07-14 18:33 help required for Datacenter Migration on Kubernetes Cluster BIPUL KHAN <bipul364@gmail.com>
2024-07-14 19:06 ` Scott Ribe <scott_ribe@elevated-dev.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox