Date: Thu, 16 Oct 2025 12:45:57 +0000 (UTC)
From: Brian Crockard <bcrockard@yahoo.com>
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>,
	mahamood hussain <hussain.ieg@gmail.com>
Message-ID: <1915052734.3603827.1760618757611@mail.yahoo.com>
In-Reply-To: 
 <CAGc_7HnBQZva+cnnm_=_pWSmrS6v217FPW7b8DC43RKXi9jqvQ@mail.gmail.com>
References: 
 <CAGc_7HnBQZva+cnnm_=_pWSmrS6v217FPW7b8DC43RKXi9jqvQ@mail.gmail.com>
Subject: Re: Guidance Requested: Migrating Large-Scale DB2 Databases to
 PostgreSQL
MIME-Version: 1.0
Content-Type: multipart/alternative;
	boundary="----=_Part_3603826_1301813784.1760618757610"
Content-Length: 7302
Archived-At: 
 <https://www.postgresql.org/message-id/1915052734.3603827.1760618757611%40mail.yahoo.com>
Precedence: bulk

------=_Part_3603826_1301813784.1760618757610
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

 You may want to look into an IBM product called InfoSphere Data Replciatio=
n (CDC)
https://www.ibm.com/docs/nl/idr/11.4.0?topic=3Drequirements-supported-sourc=
e-targets
It will replicate data from DB2 to PostgreSQL. You can set it up to activel=
y replciate the data and keep it consistent. Then perform a cutover at some=
 point to the new system running against the new replicated database. You w=
ill then have all of your historical data in the new system. We used this o=
n an extremely active system and it hard very few issues.=C2=A0
    On Wednesday, October 15, 2025 at 05:15:00 PM EDT, mahamood hussain <hu=
ssain.ieg@gmail.com> wrote: =20
=20
=20
Hi Team,

We are in the process of migrating several DB2 databases to PostgreSQL, pri=
marily to reduce the high licensing costs associated with DB2. These databa=
ses support retail applications (e.g., supermarkets and stores), and during=
 peak hours, we anticipate over 100 concurrent connections.

Current Database Profile:
  =20
   -   =20
Approximately 3,000 tables in total

   -   =20
Around 100 tables contain active data

   -   =20
Most tables have low data volume

   -   =20
A few large tables range from 10 GB to 2 TB

   -   =20
The largest table contains approximately 80 billion rows


Migration Approach:
  =20
   -   =20
We are using Ispirer for code conversion (DB2 to PostgreSQL).

   -   =20
For data migration, we are evaluating Fivetran, but noted that it relies on=
 the COPY method for data loading.


Questions & Areas Where We Need Guidance:
  =20
   -   =20
Is Fivetran a suitable option for migrating very large datasets (e.g., tabl=
es with 80+ billion rows)?

   -   =20
Are there any reliable open-source tools for DB2 to PostgreSQL data migrati=
on that we can use internally, without needing to invest in a tool like Fiv=
etran?

   -   =20
Are there more scalable or efficient alternatives for both the initial load=
 and ongoing/incremental sync to PostgreSQL?


Additional Input Requested:
  =20
   -   =20
What are the key best practices (Do=E2=80=99s and Don=E2=80=99ts) to keep i=
n mind during a large-scale DB2 =E2=86=92 PostgreSQL migration?

   -   =20
Are there specific PostgreSQL settings or configurations we should pay atte=
ntion to for optimizing performance, especially for large datasets and DB2-=
style workloads?


We are keen to ensure performance, data integrity, and scalability througho=
ut this migration. Any insights=E2=80=94particularly from those with experi=
ence in similar large-scale PostgreSQL implementations=E2=80=94would be hig=
hly appreciated.

If this is not the right forum for these questions, please do let me know i=
f there is a better place to seek this guidance.


Thanks in advance for your support!
 =20
------=_Part_3603826_1301813784.1760618757610
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<html><head></head><body><div class=3D"ydpee7ab289yahoo-style-wrap" style=
=3D"font-family:Helvetica Neue, Helvetica, Arial, sans-serif;font-size:13px=
;"><div></div>
        <div dir=3D"ltr" data-setdir=3D"false">You may want to look into an=
 IBM product called InfoSphere Data Replciation (CDC)</div><div dir=3D"ltr"=
 data-setdir=3D"false"><br></div><div dir=3D"ltr" data-setdir=3D"false"><a =
href=3D"https://www.ibm.com/docs/nl/idr/11.4.0?topic=3Drequirements-support=
ed-source-targets" rel=3D"nofollow" target=3D"_blank">https://www.ibm.com/d=
ocs/nl/idr/11.4.0?topic=3Drequirements-supported-source-targets</a></div><d=
iv dir=3D"ltr" data-setdir=3D"false"><br></div><div dir=3D"ltr" data-setdir=
=3D"false">It will replicate data from DB2 to PostgreSQL. You can set it up=
 to actively replciate the data and keep it consistent. Then perform a cuto=
ver at some point to the new system running against the new replicated data=
base. You will then have all of your historical data in the new system. We =
used this on an extremely active system and it hard very few issues.&nbsp;<=
/div><div><br></div>
       =20
        <div id=3D"ydpee7ab289yahoo_quoted_1463862638" class=3D"ydpee7ab289=
yahoo_quoted">
            <div style=3D"font-family:'Helvetica Neue', Helvetica, Arial, s=
ans-serif;font-size:13px;">
               =20
                <div>
                    On Wednesday, October 15, 2025 at 05:15:00 PM EDT, maha=
mood hussain &lt;hussain.ieg@gmail.com&gt; wrote:
                </div>
                <div><br></div>
                <div><br></div>
                <div><div id=3D"ydpee7ab289yiv4950937167"><div dir=3D"ltr">=
<div dir=3D"ltr"><div dir=3D"ltr"><p>Hi Team,</p><p>We are in the process o=
f migrating several DB2 databases to PostgreSQL, primarily to reduce the hi=
gh licensing costs associated with DB2. These databases support retail appl=
ications (e.g., supermarkets and stores), and during peak hours, we anticip=
ate over 100 concurrent connections.</p><hr><h3><span style=3D"font-weight:=
normal;">Current Database Profile:</span></h3><ul>
<li>
<p>Approximately 3,000 tables in total</p>
</li>
<li>
<p>Around 100 tables contain active data</p>
</li>
<li>
<p>Most tables have low data volume</p>
</li>
<li>
<p>A few large tables range from 10 GB to 2 TB</p>
</li>
<li>
<p>The largest table contains approximately 80 billion rows</p>
</li>
</ul><hr><h3><span style=3D"font-weight:normal;">Migration Approach:</span>=
</h3><ul>
<li>
<p>We are using Ispirer for code conversion (DB2 to PostgreSQL).</p>
</li>
<li>
<p>For data migration, we are evaluating Fivetran, but noted that it relies=
 on the <code>COPY</code> method for data loading.</p>
</li>
</ul><hr><h3><span style=3D"font-weight:normal;">Questions &amp; Areas Wher=
e We Need Guidance:</span></h3><ol>
<li>
<p>Is Fivetran a suitable option for migrating very large datasets (e.g., t=
ables with 80+ billion rows)?</p>
</li>
<li>
<p>Are there any reliable open-source tools for DB2 to PostgreSQL data migr=
ation that we can use internally, without needing to invest in a tool like =
Fivetran?</p>
</li>
<li>
<p>Are there more scalable or efficient alternatives for both the initial l=
oad and ongoing/incremental sync to PostgreSQL?</p>
</li>
</ol><hr><h3><span style=3D"font-weight:normal;">Additional Input Requested=
:</span></h3><ul>
<li>
<p>What are the key best practices (Do=E2=80=99s and Don=E2=80=99ts) to kee=
p in mind during a large-scale DB2 =E2=86=92 PostgreSQL migration?</p>
</li>
<li>
<p>Are there specific PostgreSQL settings or configurations we should pay a=
ttention to for optimizing performance, especially for large datasets and D=
B2-style workloads?</p>
</li>
</ul><hr><p>We are keen to ensure performance, data integrity, and scalabil=
ity throughout this migration. Any insights=E2=80=94particularly from those=
 with experience in similar large-scale PostgreSQL implementations=E2=80=94=
would be highly appreciated.</p><p>If this is not the right forum for these=
 questions, please do let me know if there is a better place to seek this g=
uidance.</p><p>


</p><p>Thanks in advance for your support!</p></div></div></div>
</div></div>
            </div>
        </div></div></body></html>
------=_Part_3603826_1301813784.1760618757610--