public inbox for [email protected]
help / color / mirror / Atom feedFrom: Muhammet Kurtoğlu <[email protected]>
To: mahamood hussain <[email protected]>
Cc: Brian Crockard <[email protected]>
Cc: Pgsql-admin <[email protected]>
Subject: Re: Guidance Requested: Migrating Large-Scale DB2 Databases to PostgreSQL
Date: Fri, 17 Oct 2025 23:31:50 +0300
Message-ID: <CANM7_8U5NxXUVSbGUzi9yi6YOL_L9R9dmMTVZfMqX=0c73U6OA@mail.gmail.com> (raw)
In-Reply-To: <CAGc_7HmfZ+6dKWuvKvUPDHdJH1ra1Jr=hk-=QUOpwpe-=WGc9w@mail.gmail.com>
References: <CAGc_7HnBQZva+cnnm_=_pWSmrS6v217FPW7b8DC43RKXi9jqvQ@mail.gmail.com>
<[email protected]>
<CAGc_7HmsiEpekifed=xWpx0Cn-c8=GhOfjxai=aEFA-GSK+WDg@mail.gmail.com>
<CAGc_7HmfZ+6dKWuvKvUPDHdJH1ra1Jr=hk-=QUOpwpe-=WGc9w@mail.gmail.com>
Symetricds is trigger based not log based
*Muhammet KURTOĞLU*
T: +90(312) 220 12 20 <%2B90%28374%29%20262%2098%2000>
F: +90(312) 286 00 10 <%2B90%28374%29%20262%2090%2091>
[email protected] <[email protected]>
17 Eki 2025 Cum, saat 23:30 tarihinde mahamood hussain <
[email protected]> şunu yazdı:
> A few other points I’d like to highlight — we use the DB2 LOAD utility for
> our daily data loads into tables. Since this utility bypasses transaction
> logging for performance reasons, any open-source replication tool that
> relies on transaction logs to replicate data into PostgreSQL will not
> capture or replicate these changes. Therefore, I'm looking for tools that
> can specifically address this primary limitation.
>
> On Sat, Oct 18, 2025 at 1:19 AM mahamood hussain <[email protected]>
> wrote:
>
>> Hi Brian, Ron, Kurtoglu,
>>
>> Thank you all for the thoughtful responses. It took a bit of effort to
>> pull this information together, but I'm glad to see the insights coming in
>> — it's given me more confidence that this migration is indeed achievable
>> with the right approach.
>>
>> To clarify our setup:
>> We're currently running DB2 LUW v11.5.9 (Advanced Enterprise Server
>> Edition). One of our key concerns is licensing cost, which is driving our
>> move to PostgreSQL.
>> A few follow-up questions based on your responses:
>>
>> 1.
>>
>> InfoSphere Data Replication (CDC):
>> We understand this is a strong option, but it likely incurs
>> additional licensing costs. We’re exploring Fivetran for data replication
>> between DB2 and PostgreSQL, with the flexibility to fall back to DB2 if
>> performance or stability issues arise in production.
>> -
>>
>> Can CDC be used for such a failback scenario (i.e., from
>> PostgreSQL → DB2) similar to how Fivetran supports bidirectional sync?
>> 2.
>>
>> Open-source Tools:
>> Do we have any robust open-source tools that support both forward
>> and backward replication between DB2 and PostgreSQL? This would help us
>> plan for both migration and rollback scenarios without heavy vendor lock-in.
>> 3.
>>
>> Pitfalls to Avoid During Migration:
>> Since our primary goal is cost reduction, we would ideally avoid
>> needing to fall back to DB2. Based on your experience, what are some common
>> pitfalls or gotchas we should look out for when moving from DB2 to
>> PostgreSQL?
>> 4.
>>
>> PostgreSQL Performance Tuning & Config:
>> If there’s a list of standard PostgreSQL parameters that you'd
>> recommend tweaking right after installation — especially for large datasets
>> — that would be very helpful. We're anticipating around 100+ concurrent
>> connections during peak hours and migrating some very large tables (up to
>> 80B rows).
>> 5.
>>
>> Community vs Enterprise Support:
>> In production, having support with a guaranteed response time (e.g.,
>> under an hour) is crucial.
>> -
>>
>> Does the PostgreSQL community offer any premium or paid support
>> options with SLAs, or would we need to go with providers like EDB for this
>> level of assurance?
>>
>> ------------------------------
>>
>>
>> On Thu, Oct 16, 2025 at 6:15 PM Brian Crockard <[email protected]>
>> wrote:
>>
>>> You may want to look into an IBM product called InfoSphere Data
>>> Replciation (CDC)
>>>
>>>
>>> https://www.ibm.com/docs/nl/idr/11.4.0?topic=requirements-supported-source-targets
>>>
>>> It will replicate data from DB2 to PostgreSQL. You can set it up to
>>> actively replciate the data and keep it consistent. Then perform a cutover
>>> at some point to the new system running against the new replicated
>>> database. You will then have all of your historical data in the new system.
>>> We used this on an extremely active system and it hard very few issues.
>>>
>>> On Wednesday, October 15, 2025 at 05:15:00 PM EDT, mahamood hussain <
>>> [email protected]> wrote:
>>>
>>>
>>> Hi Team,
>>>
>>> We are in the process of migrating several DB2 databases to PostgreSQL,
>>> primarily to reduce the high licensing costs associated with DB2. These
>>> databases support retail applications (e.g., supermarkets and stores), and
>>> during peak hours, we anticipate over 100 concurrent connections.
>>> ------------------------------
>>> Current Database Profile:
>>>
>>> -
>>>
>>> Approximately 3,000 tables in total
>>> -
>>>
>>> Around 100 tables contain active data
>>> -
>>>
>>> Most tables have low data volume
>>> -
>>>
>>> A few large tables range from 10 GB to 2 TB
>>> -
>>>
>>> The largest table contains approximately 80 billion rows
>>>
>>> ------------------------------
>>> Migration Approach:
>>>
>>> -
>>>
>>> We are using Ispirer for code conversion (DB2 to PostgreSQL).
>>> -
>>>
>>> For data migration, we are evaluating Fivetran, but noted that it
>>> relies on the COPY method for data loading.
>>>
>>> ------------------------------
>>> Questions & Areas Where We Need Guidance:
>>>
>>> 1.
>>>
>>> Is Fivetran a suitable option for migrating very large datasets
>>> (e.g., tables with 80+ billion rows)?
>>> 2.
>>>
>>> Are there any reliable open-source tools for DB2 to PostgreSQL data
>>> migration that we can use internally, without needing to invest in a tool
>>> like Fivetran?
>>> 3.
>>>
>>> Are there more scalable or efficient alternatives for both the
>>> initial load and ongoing/incremental sync to PostgreSQL?
>>>
>>> ------------------------------
>>> Additional Input Requested:
>>>
>>> -
>>>
>>> What are the key best practices (Do’s and Don’ts) to keep in mind
>>> during a large-scale DB2 → PostgreSQL migration?
>>> -
>>>
>>> Are there specific PostgreSQL settings or configurations we should
>>> pay attention to for optimizing performance, especially for large datasets
>>> and DB2-style workloads?
>>>
>>> ------------------------------
>>>
>>> We are keen to ensure performance, data integrity, and scalability
>>> throughout this migration. Any insights—particularly from those with
>>> experience in similar large-scale PostgreSQL implementations—would be
>>> highly appreciated.
>>>
>>> If this is not the right forum for these questions, please do let me
>>> know if there is a better place to seek this guidance.
>>>
>>> Thanks in advance for your support!
>>>
>>
reply
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: [email protected]
Cc: [email protected], [email protected], [email protected], [email protected]
Subject: Re: Guidance Requested: Migrating Large-Scale DB2 Databases to PostgreSQL
In-Reply-To: <CANM7_8U5NxXUVSbGUzi9yi6YOL_L9R9dmMTVZfMqX=0c73U6OA@mail.gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox