pg.ddx.io aws/aws-advanced-jdbc-wrapper GitHub issues and pull requestshelp / color / mirror / Atom feed
[aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology 7+ messages / 2 participants [nested] [flat]
* [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology @ 2026-11-18 08:07 "seoh10 (@seoh10)" <noreply+seoh10@github.com> 0 siblings, 0 replies; 7+ messages in thread From: seoh10 (@seoh10) @ 2026-11-18 08:07 UTC (permalink / raw) To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com> ### Describe the bug After a Blue/Green Deployment (BGD) switchover completes, some application instances permanently stay in the plugin's `POST` phase. They keep polling the deployment status at the fast switchover interval (`bgHighMs`) forever, never detect a subsequent new BGD, and log no warnings or errors. The state exists only in plugin memory and can only be cleared by restarting the application instance (or removing the `bg` plugin). We initially suspected an issue with our Aurora setup, but AWS Premium Support investigated our logs, analyzed the wrapper 4.6.0 source code, reproduced the behavior in their own environment, and confirmed it as a defect in the `bg` plugin (AWS support case, 2026-11). Their analysis and reproduction are summarized under "Additional Information/Context" below. ### Expected Behavior After `Blue/Green Deployment Switchover COMPLETED`, every instance should reset to a clean state (`NOT_CREATED`) and be able to discover a future BGD normally. ### What plugins are used? What other connection properties were set? wrapperPlugins: auroraConnectionTracker,failover2,efm2,bg ### Current Behavior Roughly half of the instances stick in `POST` after each switchover; the rest correctly transition to `NOT_CREATED`. Which instances get stuck varies run to run. Stuck instances: - keep logging `BG status: POST` at the ~100 ms cadence (`bgHighMs`) indefinitely (50+ minutes in our observations), - do not detect deletion of the old BGD or creation of a new BGD, - produce no warnings or errors (query execution is unaffected), - recover only after a restart. Log excerpt (pod-3, representative stuck instance): ``` 09:33:41 BG status: PREPARATION (monitor threads pool-8/pool-9) 09:33:44 BG status: IN_PROGRESS 09:33:48 BG status: POST 09:34:25 logSwitchoverFinalSummary: Blue/Green Deployment Switchover COMPLETED 09:34:31 BG status: NOT_CREATED (new monitor threads pool-25/pool-26 start) 09:34:32 BG status: POST ← re-enters POST and never leaves ... BG status: POST (every ~100ms, indefinitely) ``` ### Reproduction Steps 1. Set up an Aurora MySQL 3.10.4 cluster (writer + reader) and a Blue/Green Deployment targeting the same engine version. 2. Run ~10 separate application instances using the wrapper 4.6.0 with HikariCP and the `bg` plugin. 3. Start the switchover and wait until all instances log `Blue/Green Deployment Switchover COMPLETED`. 4. Observe the instances right after COMPLETED: those whose post-reset monitoring threads start while the final switchover status row is still present in `mysql.rds_topology` on the new writer will re-enter `POST` and stay there; instances whose threads start after the row is removed will correctly go to `NOT_CREATED`. A separate process polling `mysql.rds_topology` on every endpoint once per second makes the ~30 s persistence of the final status row on the new writer visible and correlates it with which instances get stuck. ### Possible Solution We did not modify the source; the following is from the AWS support analysis. The trigger is the race between: (a) the post-COMPLETED reset (old monitor threads stopped, collected state cleared, new monitor threads started a few seconds later), and (b) the final switchover status row lingering in `mysql.rds_topology` on the new writer for ~30 seconds after the switchover completes. If the new monitor threads start within that window, they read the final status row as an active switchover. Because the reset also cleared the plugin's internal "DNS updated" flags, the plugin cannot complete the switchover a second time and reports `POST`. Since the plugin never moves the phase backward (except on deployment rollback) and `bgSwitchoverTimeoutMs` is cleared by the reset (it is only armed when the plugin observes the `PREPARATION` phase), a pod that lands in `POST` right after the reset never recovers. A possible fix direction: when the plugin's state has just been reset (no in-progress switchover of its own), treat a newly observed final-status row as stale/final rather than an active switchover — e.g., validate it against the deployment's actual lifecycle state, or ignore status rows that carry the terminal status while no switchover is tracked internally. ### Additional Information/Context AWS Premium Support reproduced the issue independently (Aurora MySQL 3.10.4, BGD to the same engine version, 10 application instances on wrapper 4.6.0 + HikariCP with bg/failover/efm2 plugins, default Blue/Green settings): - All 10 instances completed the switchover. 7/10 started their post-reset monitor threads while the final status row was still present and stayed in `POST` (still stuck ~50 minutes later, no warnings/errors). The other 3 started after the row was removed and moved to `NOT_CREATED`. - After deleting the old BGD/cluster and creating a new BGD, the 3 healthy instances moved to `CREATED` and logged `Blue/Green target topology recognized`; the 7 stuck instances read the new deployment's status but remained in `POST`. - Restarting one stuck instance cleared the state immediately (moved to `CREATED` and recognized the new deployment). - While stuck, the plugin polls at `bgHighMs` (100 ms) instead of `bgBaselineMs` (60 s). Application queries kept working normally throughout. - AWS also confirmed the same code path with a unit test against the unmodified 4.6.0 source, independent of timing. - Upgrading does not currently help: the same logic exists in 4.6.0 (the latest release) and the 4.5.0 deleted/recreated-deployment detection only applies while the deployment is idle in `AVAILABLE` (the plugin's `CREATED` phase), so it does not cover this case. Our own observation (10-pod dev application, multiple switchover rounds) matches this: the split between stuck/healthy pods aligned exactly with when each pod's new monitor threads started relative to the metadata cleanup, not with the old deployment's deletion. Happy to provide additional log extracts if useful. ### The AWS Advanced JDBC Wrapper version used 4.6.0 ### JDK version used 8 ### Operating System and version Amazon Linux 2 ^ permalink raw [nested|flat] 7+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology @ 2026-11-18 08:07 "seoh10 (@seoh10)" <noreply+seoh10@github.com> 5 siblings, 0 replies; 7+ messages in thread From: seoh10 (@seoh10) @ 2026-11-18 08:07 UTC (permalink / raw) To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com> @seoh10 applied label "bug". ^ permalink raw [nested|flat] 7+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology @ 2026-11-19 16:36 "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 5 siblings, 0 replies; 7+ messages in thread From: sergiyvamz (@sergiyvamz) @ 2026-11-19 16:36 UTC (permalink / raw) To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com> @sergiyvamz applied label "pending release". ^ permalink raw [nested|flat] 7+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology @ 2026-11-19 16:37 "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 5 siblings, 0 replies; 7+ messages in thread From: sergiyvamz (@sergiyvamz) @ 2026-11-19 16:37 UTC (permalink / raw) To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com> Hi @seoh10, Thanks for the detailed report, and please pass our thanks to AWS Support for the analysis and reproduction. This issue is fixed by #2242, which has been merged into `main`. After a completed switchover, a context that never saw the deployment in the `CREATED` phase now ignores a lingering `POST`/`COMPLETED` status row, so it stays in idle monitoring and can detect a new deployment. The switchover timeout is also now started in every phase from `PREPARATION` onwards, so a stuck switchover is always bounded. The fix will be included in the next release. If you'd like to verify it earlier, you can try the latest snapshot build of `main`. We'll close this issue once the release is out, unless you see the problem again. Thank you for your patience! ^ permalink raw [nested|flat] 7+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology @ 2026-11-19 23:42 "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 5 siblings, 0 replies; 7+ messages in thread From: sergiyvamz (@sergiyvamz) @ 2026-11-19 23:42 UTC (permalink / raw) To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com> Hi @seoh10, The fix from #2242 is now released in [4.7.0](https://github.com/aws/aws-advanced-jdbc-wrapper/releases/tag/4.7.0). Since this was the plan in our previous comment, we're closing this issue. If you still see instances stuck in the `POST` phase after upgrading, please reopen it or file a new issue with the logs. Thank you! ^ permalink raw [nested|flat] 7+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology @ 2026-11-19 23:42 "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 5 siblings, 0 replies; 7+ messages in thread From: sergiyvamz (@sergiyvamz) @ 2026-11-19 23:42 UTC (permalink / raw) To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com> @sergiyvamz closed this. ^ permalink raw [nested|flat] 7+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology @ 2026-11-19 23:42 "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 5 siblings, 0 replies; 7+ messages in thread From: sergiyvamz (@sergiyvamz) @ 2026-11-19 23:42 UTC (permalink / raw) To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com> @sergiyvamz removed label "pending release". ^ permalink raw [nested|flat] 7+ messages in thread
end of thread, other threads:[~2026-11-19 23:42 UTC | newest] Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2026-11-18 08:07 [aws/aws-advanced-jdbc-wrapper] issue #2254: [bg plugin] Pods stick in POST phase permanently after a Blue/Green switchover completes, due to a race between post-COMPLETED monitor reset and the lingering final status row in mysql.rds_topology "seoh10 (@seoh10)" <noreply+seoh10@github.com> 2026-11-18 08:07 ` "seoh10 (@seoh10)" <noreply+seoh10@github.com> 2026-11-19 16:36 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 2026-11-19 16:37 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 2026-11-19 23:42 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 2026-11-19 23:42 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com> 2026-11-19 23:42 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox