Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1v9r7R-009f5K-J0 for pgsql-admin@arkaria.postgresql.org; Fri, 17 Oct 2025 20:32:09 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1v9r7Q-005GaT-IT for pgsql-admin@arkaria.postgresql.org; Fri, 17 Oct 2025 20:32:07 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1v9r7P-005GaK-VI for pgsql-admin@lists.postgresql.org; Fri, 17 Oct 2025 20:32:07 +0000 Received: from mail-ed1-x534.google.com ([2a00:1450:4864:20::534]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1v9r7M-002LbN-2E for pgsql-admin@lists.postgresql.org; Fri, 17 Oct 2025 20:32:06 +0000 Received: by mail-ed1-x534.google.com with SMTP id 4fb4d7f45d1cf-6317348fa4fso412454a12.3 for ; Fri, 17 Oct 2025 13:32:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bisoft-com-tr.20230601.gappssmtp.com; s=20230601; t=1760733123; x=1761337923; darn=lists.postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=m/SkRc28d2keEAm20bsnjbsNn/JiNqJGWfSpEtpoOTo=; b=ZVMFQiXWMqO9GTl/dPmF2kSo/fmc93jX2qBUetRW5ADHKe3ZS4y1YI7734udsDPON5 z5de5e/iKeADtHiV6ZyH1cJfL2vy1WHrMqNdF5tRyW3dKOgL+a8RdkHQq2w2jmgqjgUY SKwAVuqq5M6CCybGVn4lo1JHeQsoFsCf3ZbTo0jziD5QVDXdvyGKiQVbr1OfTOFRZwTl q6abOInlPcYl2x5hFl9w84Z0xX2sIB3Tz3SuPxbYAAUIzgErsr+EoeAOEbWJN34Ti/0S US626//qOF9ZLUI+sXhd8UTcQHbvv1ZzwQIojsynR3Iyl28t98rK5XImx5KKn6pVzK7K DnDg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1760733123; x=1761337923; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=m/SkRc28d2keEAm20bsnjbsNn/JiNqJGWfSpEtpoOTo=; b=jQKW7eYbMFLgaYp7l3iEuHjQXSC4xsFYHtHFKk3TilfD1XaVADhH+tZkl++duiOupf 1L0EfJXdOlzrXvEWm9vMbKZdsFHovKfpTBlQCaO/klmvsfJ+hdABzZuQeIX4SLbcpzAH rh51zfBkoXMRM1FLLK2sTrbnmvvd0gDeGEETBjkfmTKgUHXcIKNnPPufCsSfPAYyrbiu wNrBXGGB+q3huAjiMEjViVSgHnC5cU+r++q0vO1p4AaF3K/Kki+71KQeEyAItfVBdcbl MD7HIT/So9+xdSRjXaNW9gLIft9ZV35wnhNhS17/C2+BWuqS/4SUBGByGFJhICGP/j4b pnbA== X-Forwarded-Encrypted: i=1; AJvYcCW7dphblGVW9U9zMXPAC+Q/JFyWmTtDjfxsYabXva24gdx5dWH32MVhyllgp1Q82qB3UOIqoQBwAFvabQ==@lists.postgresql.org X-Gm-Message-State: AOJu0YwMcnn/BVhhihupKVBLPMVVqcCr6Y0F192YcUxy60njiRaAamPa KO6S/gK8RR7MCaEwb2mWqyVyRXF0iGkAeEgRoqRgCkLG3xAVD7gP4+a8pgAa/piaoQrSyyayxeJ nOd92qGxu4Nh6RA/wLXP7zGOGEFwuyyt6F7B9Ma5/Ylali2M46mc/Z5T13bWd X-Gm-Gg: ASbGnctRz98tzhJbYZdpc5mFI9UpbrbB1tNlEjtRh4jgQWXGl0dGOloeE9dzye1zqHC clZnp4SuEAeVgkKT+9dPiiYivMnMtQsJDPzCXfjvjwYZkPPZWoAdNiDMjkaBrqOfGwSKG2QM4G/ avG0tX6N+ash2nMtQJlX74qZmE4gOy+tnlTndRRmYp1aeg217J4Az9y57vyD5LiS0qWQB55B1fd pK8SHPrJug1Zkf/8gtvRuBm0L9MzK9sQqVPFwPtp4S/cj6Jg6Y81ZiJqUoYmkvEPu3xIw== X-Google-Smtp-Source: AGHT+IEiCW7Y5en05RifK43H47UELhoJlBHoc7RG6UjmGNgSTurt95Rg6RJU/X6Gm++BEoSPB5CANIigISreB70oUfo= X-Received: by 2002:a05:6402:510f:b0:63c:1d4a:afcb with SMTP id 4fb4d7f45d1cf-63c1f5928ecmr2479668a12.0.1760733123133; Fri, 17 Oct 2025 13:32:03 -0700 (PDT) MIME-Version: 1.0 References: <1915052734.3603827.1760618757611@mail.yahoo.com> In-Reply-To: From: =?UTF-8?Q?Muhammet_Kurto=C4=9Flu?= Date: Fri, 17 Oct 2025 23:31:50 +0300 X-Gm-Features: AS18NWB8iHUell99NprqTvQE9UKr6ShOg_T__qh_lLAdp0hA5feibPWLMcv4dIA Message-ID: Subject: Re: Guidance Requested: Migrating Large-Scale DB2 Databases to PostgreSQL To: mahamood hussain Cc: Brian Crockard , Pgsql-admin Content-Type: multipart/alternative; boundary="0000000000005952cd064160a02f" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000005952cd064160a02f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Symetricds is trigger based not log based *Muhammet KURTO=C4=9ELU* T: +90(312) 220 12 20 <%2B90%28374%29%20262%2098%2000> F: +90(312) 286 00 10 <%2B90%28374%29%20262%2090%2091> muhammet.kurtoglu@bisoft.com.tr 17 Eki 2025 Cum, saat 23:30 tarihinde mahamood hussain < hussain.ieg@gmail.com> =C5=9Funu yazd=C4=B1: > A few other points I=E2=80=99d like to highlight =E2=80=94 we use the DB2= LOAD utility for > our daily data loads into tables. Since this utility bypasses transaction > logging for performance reasons, any open-source replication tool that > relies on transaction logs to replicate data into PostgreSQL will not > capture or replicate these changes. Therefore, I'm looking for tools that > can specifically address this primary limitation. > > On Sat, Oct 18, 2025 at 1:19=E2=80=AFAM mahamood hussain > wrote: > >> Hi Brian, Ron, Kurtoglu, >> >> Thank you all for the thoughtful responses. It took a bit of effort to >> pull this information together, but I'm glad to see the insights coming = in >> =E2=80=94 it's given me more confidence that this migration is indeed ac= hievable >> with the right approach. >> >> To clarify our setup: >> We're currently running DB2 LUW v11.5.9 (Advanced Enterprise Server >> Edition). One of our key concerns is licensing cost, which is driving ou= r >> move to PostgreSQL. >> A few follow-up questions based on your responses: >> >> 1. >> >> InfoSphere Data Replication (CDC): >> We understand this is a strong option, but it likely incurs >> additional licensing costs. We=E2=80=99re exploring Fivetran for data= replication >> between DB2 and PostgreSQL, with the flexibility to fall back to DB2 = if >> performance or stability issues arise in production. >> - >> >> Can CDC be used for such a failback scenario (i.e., from >> PostgreSQL =E2=86=92 DB2) similar to how Fivetran supports bidirec= tional sync? >> 2. >> >> Open-source Tools: >> Do we have any robust open-source tools that support both forward >> and backward replication between DB2 and PostgreSQL? This would help = us >> plan for both migration and rollback scenarios without heavy vendor l= ock-in. >> 3. >> >> Pitfalls to Avoid During Migration: >> Since our primary goal is cost reduction, we would ideally avoid >> needing to fall back to DB2. Based on your experience, what are some = common >> pitfalls or gotchas we should look out for when moving from DB2 to >> PostgreSQL? >> 4. >> >> PostgreSQL Performance Tuning & Config: >> If there=E2=80=99s a list of standard PostgreSQL parameters that you= 'd >> recommend tweaking right after installation =E2=80=94 especially for = large datasets >> =E2=80=94 that would be very helpful. We're anticipating around 100+ = concurrent >> connections during peak hours and migrating some very large tables (u= p to >> 80B rows). >> 5. >> >> Community vs Enterprise Support: >> In production, having support with a guaranteed response time (e.g., >> under an hour) is crucial. >> - >> >> Does the PostgreSQL community offer any premium or paid support >> options with SLAs, or would we need to go with providers like EDB = for this >> level of assurance? >> >> ------------------------------ >> >> >> On Thu, Oct 16, 2025 at 6:15=E2=80=AFPM Brian Crockard >> wrote: >> >>> You may want to look into an IBM product called InfoSphere Data >>> Replciation (CDC) >>> >>> >>> https://www.ibm.com/docs/nl/idr/11.4.0?topic=3Drequirements-supported-s= ource-targets >>> >>> It will replicate data from DB2 to PostgreSQL. You can set it up to >>> actively replciate the data and keep it consistent. Then perform a cuto= ver >>> at some point to the new system running against the new replicated >>> database. You will then have all of your historical data in the new sys= tem. >>> We used this on an extremely active system and it hard very few issues. >>> >>> On Wednesday, October 15, 2025 at 05:15:00 PM EDT, mahamood hussain < >>> hussain.ieg@gmail.com> wrote: >>> >>> >>> Hi Team, >>> >>> We are in the process of migrating several DB2 databases to PostgreSQL, >>> primarily to reduce the high licensing costs associated with DB2. These >>> databases support retail applications (e.g., supermarkets and stores), = and >>> during peak hours, we anticipate over 100 concurrent connections. >>> ------------------------------ >>> Current Database Profile: >>> >>> - >>> >>> Approximately 3,000 tables in total >>> - >>> >>> Around 100 tables contain active data >>> - >>> >>> Most tables have low data volume >>> - >>> >>> A few large tables range from 10 GB to 2 TB >>> - >>> >>> The largest table contains approximately 80 billion rows >>> >>> ------------------------------ >>> Migration Approach: >>> >>> - >>> >>> We are using Ispirer for code conversion (DB2 to PostgreSQL). >>> - >>> >>> For data migration, we are evaluating Fivetran, but noted that it >>> relies on the COPY method for data loading. >>> >>> ------------------------------ >>> Questions & Areas Where We Need Guidance: >>> >>> 1. >>> >>> Is Fivetran a suitable option for migrating very large datasets >>> (e.g., tables with 80+ billion rows)? >>> 2. >>> >>> Are there any reliable open-source tools for DB2 to PostgreSQL data >>> migration that we can use internally, without needing to invest in a= tool >>> like Fivetran? >>> 3. >>> >>> Are there more scalable or efficient alternatives for both the >>> initial load and ongoing/incremental sync to PostgreSQL? >>> >>> ------------------------------ >>> Additional Input Requested: >>> >>> - >>> >>> What are the key best practices (Do=E2=80=99s and Don=E2=80=99ts) to= keep in mind >>> during a large-scale DB2 =E2=86=92 PostgreSQL migration? >>> - >>> >>> Are there specific PostgreSQL settings or configurations we should >>> pay attention to for optimizing performance, especially for large da= tasets >>> and DB2-style workloads? >>> >>> ------------------------------ >>> >>> We are keen to ensure performance, data integrity, and scalability >>> throughout this migration. Any insights=E2=80=94particularly from those= with >>> experience in similar large-scale PostgreSQL implementations=E2=80=94wo= uld be >>> highly appreciated. >>> >>> If this is not the right forum for these questions, please do let me >>> know if there is a better place to seek this guidance. >>> >>> Thanks in advance for your support! >>> >> --0000000000005952cd064160a02f Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Symetricds is trigger based not log based=C2=A0


Muhammet KURTO=C4=9ELU=

T:=C2=A0=C2=A0+90(312) 220 12 20

F:=C2=A0=C2=A0+90(312) 286 0= 0 10<= /p>

muhammet.kurtoglu@bisoft.com.tr=C2=A0

=C2=A0=C2=A0=C2=A0=C2=A0

=







17 Eki 2025 Cum, saat 23:30 tarihinde mahamood hussain &= lt;hussain.ieg@gmail.com> = =C5=9Funu yazd=C4=B1:
A few other points I=E2=80=99d like to highlight =E2= =80=94 we use the DB2 LOAD utility for our daily data loads into tables. Si= nce this utility bypasses transaction logging for performance reasons, any = open-source replication tool that relies on transaction logs to replicate d= ata into PostgreSQL will not capture or replicate these changes. Therefore,= I'm=C2=A0looking for tools that can specifically address this primary = limitation.

On Sat, Oct 18, 2025 at = 1:19=E2=80=AFAM mahamood hussain <hussain.ieg@gmail.com> wrote:

Hi Brian, Ron, Kurtoglu,

Thank you all for the thoughtful responses. It took a bit of effort to p= ull this information together, but I'm glad to see the insights coming = in =E2=80=94 it's given me more confidence that this migration is indee= d achievable with the right approach.

To clarify our setup:
We're currently running DB2 LUW v11.5.9 (Advanced Enterprise Server Edi= tion). One of our key concerns is licensing cost, which is driving our move= to PostgreSQL.

A few follow-up questions based on y= our responses:

  1. InfoSphere Data Replication (CDC):
    We understand this is a strong option, but it likely incurs additional lice= nsing costs. We=E2=80=99re exploring Fivetran for data replication between = DB2 and PostgreSQL, with the flexibility to fall back to DB2 if performance= or stability issues arise in production.

    • Can CDC be used for such a failback scenario (i.e., from PostgreSQL =E2= =86=92 DB2) similar to how Fivetran supports bidirectional sync?

  2. Open-source Tools:
    Do we have any robust open-source tools =C2=A0that support both forward and= backward replication between DB2 and PostgreSQL? This would help us plan f= or both migration and rollback scenarios without heavy vendor lock-in.

  3. Pitfalls to Avoid During Migration:
    Since our primary goal is cost reduction, we would ideally avoid needing to= fall back to DB2. Based on your experience, what are some common pitfalls = or gotchas we should look out for when moving from DB2 to PostgreSQL?

  4. PostgreSQL Performance Tuning & Config:
    If there=E2=80=99s a list of standard PostgreSQL parameters =C2=A0that you&= #39;d recommend tweaking right after installation =E2=80=94 especially for = large datasets =E2=80=94 that would be very helpful. We're anticipating= around 100+ concurrent connections during peak hours and migrating some ve= ry large tables (up to 80B rows).

  5. Community vs Enterprise Support:
    In production, having support with a guaranteed response time (e.g., under = an hour) is crucial.

    • Does the PostgreSQL community offer any premium or paid support options = with SLAs, or would we need to go with providers like EDB for this level of= assurance?


On Thu, Oct 16, 2025 at 6:15=E2=80=AFPM Brian Crockard <bcrockard@yahoo.com> wrote:
You may want to look into an IBM product called In= foSphere Data Replciation (CDC)



=20
=20


Hi Team,

We are in the process of migrating= several DB2 databases to PostgreSQL, primarily to reduce the high licensin= g costs associated with DB2. These databases support retail applications (e= .g., supermarkets and stores), and during peak hours, we anticipate over 10= 0 concurrent connections.

Cur= rent Database Profile:

  • Approximately 3,000 tables in total

  • Around 100 tables contain active data

  • Most tables have low data volume

  • A few large tables range from 10 GB to 2 TB

  • The largest table contains approximately 80 billion rows

Migration Approach:<= /h3>

  • We are using Ispirer for code conversion (DB2 to PostgreSQL).

  • For data migration, we are evaluating Fivetran, but noted that it relies= on the COPY method for data loading.

Questions & Areas Where= We Need Guidance:

  1. Is Fivetran a suitable option for migrating very large datasets (e.g., t= ables with 80+ billion rows)?

  2. Are there any reliable open-source tools for DB2 to PostgreSQL data migr= ation that we can use internally, without needing to invest in a tool like = Fivetran?

  3. Are there more scalable or efficient alternatives for both the initial l= oad and ongoing/incremental sync to PostgreSQL?

Additional Input Requested:=

  • What are the key best practices (Do=E2=80=99s and Don=E2=80=99ts) to kee= p in mind during a large-scale DB2 =E2=86=92 PostgreSQL migration?

  • Are there specific PostgreSQL settings or configurations we should pay a= ttention to for optimizing performance, especially for large datasets and D= B2-style workloads?

We are keen to ensure performance, data integrity, and scalabil= ity throughout this migration. Any insights=E2=80=94particularly from those= with experience in similar large-scale PostgreSQL implementations=E2=80=94= would be highly appreciated.

If this is not the right forum for these= questions, please do let me know if there is a better place to seek this g= uidance.

Thanks in advance for your support!

--0000000000005952cd064160a02f--