Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xM1s1-00000001ok1-25o5 for pgsql-announce@arkaria.postgresql.org; Wed, 28 Oct 2026 11:31:06 +0000 Received: from localhost ([127.0.0.1]:51256 helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xM1s0-0000000FdZd-372b for pgsql-announce@arkaria.postgresql.org; Wed, 28 Oct 2026 11:31:04 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]:54716) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xM1pG-0000000FUI7-1xpQ for pgsql-announce@lists.postgresql.org; Wed, 28 Oct 2026 11:28:14 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xM1p9-00000001LrU-0sVn for pgsql-announce@lists.postgresql.org; Wed, 28 Oct 2026 11:28:13 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:From:To:Subject: MIME-Version:Content-Type:Sender:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:In-Reply-To:References; bh=Y2VQRGGf8YUwajug1THyWXD9B1pEof13nHGWnskEgxE=; b=SEVgMbsudBD1L7dlWKWAIHMMYG z6Rju0KzbRHKXl9+b/SBXTavZNfI/VqbJwY9VbzN2P28374lwYFPWH+xKyshQkIHEh6DiU3JWi2/G P4Attn8rQ/7ZkbxLzF747GrwYwz2/EwOXbCGRffRm3zEepFmCtvHrpWoZPoE1QNI4eQmTGpBid/kz CKgtmCUvBdDGPNNMupA1INBLMvOi/hcSjpu3jpDky5AmkD8mTyoFRe3GmpO/D1qIJLoa8t7CWETUi 6xXZZ1dYta6VyDHJs74ErJOSQv+YusbFK1LttCNHPrfm29txfrcakIO1LLYVqnDLjgbrv8lX5M4Ni U7gtI+gg==; Received: from wrigleys.postgresql.org ([217.196.149.60]) by mahout.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.98.2) (envelope-from ) id 1xM1p8-00000003u7k-3Phl for pgsql-announce@lists.postgresql.org; Wed, 28 Oct 2026 11:28:07 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.98.2) (envelope-from ) id 1xM1p5-00000004Bgp-34pt for pgsql-announce@lists.postgresql.org; Wed, 28 Oct 2026 11:28:04 +0000 Content-Type: multipart/alternative; boundary="===============8056394985478644376==" MIME-Version: 1.0 Subject: Estuary Expands PostgreSQL CDC with Targeted Backfills and Scalable Discovery To: PostgreSQL Announce From: Estuary via PostgreSQL Announce Reply-To: sourabh@estuary.dev Date: Wed, 28 Oct 2026 11:27:31 +0000 Message-ID: <179318685158.1248.10784883534264512615@wrigleys.postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-pglister-tags: proprietary X-pglister-tagsig: 62cca97814ec4ccf62f4ccb670ea9c9c50fa1cbdc11afd39062983e92827b8ab List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --===============8056394985478644376== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Estuary has released several updates to its PostgreSQL change data capture = (CDC) connector focused on backfill recovery, discovery at scale, and schem= a-change detection. ## Targeted Backfills PostgreSQL capture bindings now support an **Additional Backfill Filter**, = allowing a SQL expression to be applied to backfill queries for an individu= al table. This can reduce the amount of data that needs to be re-read when only a sub= set of a large PostgreSQL table needs to be recovered or re-backfilled. For= example, teams can use an indexed identifier or timestamp to limit the row= s included in a recovery backfill. The filter affects backfills only. Ongoing PostgreSQL CDC continues to capt= ure new changes independently of the filter. Because the SQL expression is applied directly to backfill queries, this is= an advanced option and should be configured carefully. Changing the filter= requires the affected binding to be backfilled again. ## Discovery Controls for Large PostgreSQL Schemas Estuary has also added PostgreSQL discovery filters that allow captures to = control which tables are surfaced during connector discovery. Captures can now: - include only selected PostgreSQL schemas - exclude selected schemas - discover tables using glob-based table patterns - optionally discover tables that are not yet included in the PostgreSQL pu= blication These controls are useful for PostgreSQL environments with large numbers of= schemas or tables where only part of the database should participate in a = CDC pipeline. Table metadata discovery is also processed in smaller groups to improve rel= iability when working with databases containing large numbers of tables. ## Configurable Rediscovery PostgreSQL captures can now configure how frequently Estuary re-runs discov= ery while a capture is active. Rediscovery is used to detect changes such as newly added tables and schema= updates. The default interval is 15 minutes and can be adjusted for enviro= nments that require faster or less frequent discovery. This gives PostgreSQL operators more control over the trade-off between sch= ema-change detection latency and catalog-query activity. ## PostgreSQL CDC and Backfills Estuary's PostgreSQL source connector uses logical replication to capture i= nserts, updates, and deletes from the PostgreSQL write-ahead log (WAL). When a new capture starts, Estuary can first backfill the existing contents= of selected tables and then transition to ongoing CDC. The new backfill an= d discovery controls provide additional options for operating this process = on large PostgreSQL environments and during recovery scenarios. The PostgreSQL CDC connector supports PostgreSQL 10.0 and later across self= -hosted PostgreSQL and managed PostgreSQL services. ## Documentation - [PostgreSQL CDC connector documentation](https://docs.estuary.dev/referen= ce/Connectors/capture-connectors/PostgreSQL/) - [Backfilling data and additional backfill filters](https://docs.estuary.d= ev/reference/backfilling-data/) --===============8056394985478644376== Content-Type: text/html; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Estuary Expands PostgreSQL CDC with Targeted Backfills and Scala= ble Discovery
 

Estuary Expands PostgreSQL CDC with Targeted Backfills and Scalable D= iscovery

Estuary has released several updates to its= PostgreSQL change data capture (CDC) connector focused on backfill recover= y, discovery at scale, and schema-change detection.

Targeted Backfills

PostgreSQL capture bindings now support an = Additional Backfill Filter, allowing a SQL expression to b= e applied to backfill queries for an individual table.

This can reduce the amount of data that nee= ds to be re-read when only a subset of a large PostgreSQL table needs to be= recovered or re-backfilled. For example, teams can use an indexed identifi= er or timestamp to limit the rows included in a recovery backfill.

The filter affects backfills only. Ongoing = PostgreSQL CDC continues to capture new changes independently of the filter= .

Because the SQL expression is applied direc= tly to backfill queries, this is an advanced option and should be configure= d carefully. Changing the filter requires the affected binding to be backfi= lled again.

Discovery Controls for Large Po= stgreSQL Schemas

Estuary has also added PostgreSQL discovery= filters that allow captures to control which tables are surfaced during co= nnector discovery.

Captures can now:

  • include only se= lected PostgreSQL schemas
  • exclude selecte= d schemas
  • discover tables= using glob-based table patterns
  • optionally disc= over tables that are not yet included in the PostgreSQL publication

These controls are useful for PostgreSQL en= vironments with large numbers of schemas or tables where only part of the d= atabase should participate in a CDC pipeline.

Table metadata discovery is also processed = in smaller groups to improve reliability when working with databases contai= ning large numbers of tables.

Configurable Rediscovery

PostgreSQL captures can now configure how f= requently Estuary re-runs discovery while a capture is active.

Rediscovery is used to detect changes such = as newly added tables and schema updates. The default interval is 15 minute= s and can be adjusted for environments that require faster or less frequent= discovery.

This gives PostgreSQL operators more contro= l over the trade-off between schema-change detection latency and catalog-qu= ery activity.

PostgreSQL CDC and Backfills

Estuary's PostgreSQL source connector uses = logical replication to capture inserts, updates, and deletes from the Postg= reSQL write-ahead log (WAL).

When a new capture starts, Estuary can firs= t backfill the existing contents of selected tables and then transition to = ongoing CDC. The new backfill and discovery controls provide additional opt= ions for operating this process on large PostgreSQL environments and during= recovery scenarios.

The PostgreSQL CDC connector supports Postg= reSQL 10.0 and later across self-hosted PostgreSQL and managed PostgreSQL s= ervices.

Documentation

This email was sent to you from Estuary. It was delivered on their behalf by the PostgreSQL project. Any questions about the content of the message shou= ld be sent to Estuary.

You were sent this email as a subscriber of the pgsql-announce mai= linglist, for the content tag Related Proprietary. To unsubscribe from further emails, or change which emails you want to receive, please click th= e personal unsubscribe link that you can find in the headers of this email, or visit https://lists.postgresql.org/unsubscribe/.
 
--===============8056394985478644376==--