Content-Type: text/plain;
	charset=utf-8
Mime-Version: 1.0 (Mac OS X Mail 16.0 \(3774.600.62\))
Subject: Re: help required for Datacenter Migration on Kubernetes Cluster
From: Scott Ribe <scott_ribe@elevated-dev.com>
In-Reply-To: 
 <CAM54HcUJRUJxWF+SxXNrvQEBpemNFuRsfT8noP7UBnQZDrtF+w@mail.gmail.com>
Date: Sun, 14 Jul 2024 13:06:02 -0600
Cc: pgsql-admin@lists.postgresql.org
Content-Transfer-Encoding: quoted-printable
Message-Id: <E88BC06F-F3F1-46A7-847C-3A6CE8CEB887@elevated-dev.com>
References: 
 <CAM54HcUJRUJxWF+SxXNrvQEBpemNFuRsfT8noP7UBnQZDrtF+w@mail.gmail.com>
To: BIPUL KHAN <bipul364@gmail.com>
Archived-At: 
 <https://www.postgresql.org/message-id/E88BC06F-F3F1-46A7-847C-3A6CE8CEB887%40elevated-dev.com>
Precedence: bulk

> On Jul 14, 2024, at 12:33=E2=80=AFPM, BIPUL KHAN <bipul364@gmail.com> =
wrote:
>=20
> Hello,=20
> We have developed and run a large-scale mobile financial system using =
a microservice architecture, which includes 110 databases (big and =
small).   We use PostgreSQL as the primary database inside Kubernetes =
via the CrunchyData operator.
> Now I would like to share my problem. Our data storage has grown =
significantly since 2020, and we are using Ceph (rook-Ceph). We now have =
21 TB of data.
> Our current situation is critical as we urgently need to move our data =
center. Our source and destination data centers have a 1 GB/s data =
transfer rate. We attempted to add a new Ceph node with the same =
cluster, which started data rebalancing, but it=E2=80=99s taking a =
significant amount of time. This directly and severely impacts our =
primary transactions, which involve reading and writing to the database.
> What is the best option for moving data from one data center to a new =
one? We would appreciate your expertise. Would you happento know how to =
move it any other way?
> It is mentioned that the two different data centers are different =
Kubernetes clusters.
> I would appreciate some ideas. Feel free to let me know if you need =
any other adjustments!
>  Let me know if there are any further changes you'd like!

Plan to migrate off Ceph as soon as possible. It is simply not suited =
for this--yes, I am speaking from experience. The rebalancing will =
hammer you when you can least afford it. WekaFS, Pure Storage, heck NFS. =
Or node-local storage, XFS or ZFS. Really, just about anything else. (It =
might help if you clarify whether "move our data center" really means =
what I assumed, that you own hardware and are moving it, or whether =
you're going to another cloud provider.

You're probably going to have to migrate your databases one at a =
time--even if you temporarily upgraded your bandwidth to 10Gbps you'd =
have too long of a down time. So, one at a time, set up a standby =
streaming replica, then when they're all populated and up-to-date, take =
a very brief downtime (seconds) to switch over. This assumes that 1Gbps =
can actually handle your stream of normal updates with enough room left =
over to be transferring the base files. (Current versions of Crunchy =
support the notion of cross-cluster replication.)=