pg.ddx.io aws/aws-advanced-jdbc-wrapper GitHub issues and pull requests
help / color / mirror / Atom feed[aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
39+ messages / 3 participants
[nested] [flat]
* [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-13 20:13 "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
0 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-13 20:13 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
## Summary
Adds `efm3`, a new Enhanced Failure Monitoring plugin that registers the connection with the host monitor once instead of registering every JDBC call. `efm` and `efm2` are unchanged and `efm2` remains the default.
## Why
In `efm`/`efm2` every network-bound call takes a JVM-wide context-pool lock twice, runs a contended `ConcurrentHashMap.compute` on the host key, and (`efm2`) allocates a `WeakReference`. Monitor memory and CPU scale with calls per second (`calls/s x failureDetectionTime` live contexts). Several reported leaks and races (#675, #966, #893, #2061, #2195) come from the per-call context lifecycle.
## How `efm3` works
- One `MonitoredConnectionSlot` per physical connection, holding the connection weakly and an `AtomicLong inFlightSince`. Per call: `getAndSet(now)` on entry, `getAndSet(0)` on exit. No allocation, no lock, no shared writes.
- One monitor thread per host scans slots (O(connections)), probes only once some call has outlived `failureDetectionTime`, and after `failureDetectionCount` consecutive failed probes claims each due connection with `compareAndSet(start, ABORTED)` before aborting it through `dialect.abortConnection`. The claim/exit pair is linearizable, so a completed call is never reported aborted and no connection is aborted twice.
- The monitor thread updates the heartbeat itself, so `MonitorService` stuck detection works. Slots idle longer than `monitorDisposalTime` are swept; the monitor expires when it has no slots.
- Monitoring connection timeouts come from `monitoring-connectTimeout` / `monitoring-socketTimeout`, defaulting to `failureDetectionInterval` when unset. The application's own timeouts are not inherited.
- Host resolution happens once per connection and never calls `setCurrentConnection` or `setRoutedHostSpec`; identification errors surface as `SQLException`.
- New parameter `abortIdleConnectionsOnFailure` (default `false`) also aborts idle connections to an unhealthy host.
## Measurements
JDK 17, 14 cores, fake target driver (`benchmarks/`). Only the comparison between plugins is meaningful; `executionTime` is a trivial subscribing plugin that prices the pipeline itself.
Many connections to one host, one real plugin chain per thread (`EfmContentionBenchmarks`, ops/ms over all threads, GC time per 5 s window):
| threads | executionTime | efm2 | efm3 | GC time efm2 / efm3 |
|---|---|---|---|---|
| 1 | 166 | 127 | 168 | 545 ms / 23 ms |
| 4 | 493 | 252 | 534 | 1,511 / 83 |
| 8 | 741 | 258 | 685 | 1,468 / 135 |
| 14 | 698 | 268 | 737 | 1,811 / 145 |
| 32 | 125* | 246 | 694 | 1,752 / 177 |
efm2 stops scaling at 4 threads; efm3 tracks the pipeline floor. (*`executionTime` writes a `static long` from every thread, its own contention.)
Steady state, 64 connections x 1,000 calls/s to one host for 90 s (`EfmSoakHarness`): efm2 holds ~1.95M context references and 124-126 MB heap after GC with 2 monitoring threads; efm3 holds 64 slots and 5-6 MB with 1 thread. With a connection replaced every 5,000 calls and half of them leaked without `releaseResources`, efm3's registered slots oscillated 128-156 and returned to 64 at shutdown (sweep after `monitorDisposalTime`).
Single thread (`RealPluginChainBenchmarks`): efm2 6,733 +- 1,657 ns/op, efm3 4,732 +- 306 ns/op, pipeline floor 4,940 +- 1,479.
## Changes
- `plugin/efm/v3`: `HostMonitoringConnectionPluginV3` (+ factory), `HostMonitorServiceV3Impl`, `HostMonitorV3Impl`, `MonitoredConnectionSlot`.
- `ConnectionPluginChainBuilder`: code `efm3`, abbreviation `e3`, weight 1550.
- Docs: new "Host Monitoring Plugin v3" section, plugin table row, telemetry metrics, compatibility matrix.
- Tests: 45 unit tests for the slot protocol, monitor loop, service and plugin; `EFM3Test` container test mirroring `EFM2Test`; `PerformanceTest` gains `efm3` and `failover2,efm3` detection-time rounds.
- Benchmarks: `EfmContentionBenchmarks`, `EfmSoakHarness`, shared `RealPluginChain` support, `efm3` chain in `RealPluginChainBenchmarks`. `BenchmarkServices` now honours the context service a plugin installs, so `efm`/`efm2` are measured with the production pooled service rather than the simple one.
## Verification
`checkstyleMain/Test`, `spotbugsMain/Test`, `compileJava -PenableCheckerFramework`, `:benchmarks:jmhJar`, and `test --tests "software.amazon.jdbc.plugin.efm.*"` (137 tests) pass locally. Detection timing on a real cluster (`PerformanceTest`) still needs the Aurora environment.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 19:57 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 19:57 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on benchmarks/README.md:103)
```suggestion
It prints every report interval, the heap after a forced GC, the number of monitoring threads, and the
```
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 20:39 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 20:39 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on docs/using-the-jdbc-driver/using-plugins/UsingTheHostMonitoringPlugin.md:169)
```suggestion
The plugin is available since version 4.7.0.
```
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 20:40 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 20:40 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on docs/using-the-jdbc-driver/using-plugins/UsingTheHostMonitoringPlugin.md:115)
nit: might be valuable to mention efm3 on these types of lines too
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 20:55 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 20:55 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on docs/using-the-jdbc-driver/using-plugins/UsingTheHostMonitoringPlugin.md:189)
```suggestion
| `abortIdleConnectionsOnFailure` | 4.7.0 | Boolean | No | When the monitor decides a host is unhealthy, also abort connections to that host that have no call in flight. When `false`, only calls that have outlived `failureDetectionTime` are aborted. | `false` |
```
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 20:56 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 20:56 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on docs/using-the-jdbc-driver/CompatibilityCrossPlugins.md:10)
missing the changes needed for the other compatibility docs
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 20:57 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 20:57 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on docs/using-the-jdbc-driver/UsingTheJdbcDriver.md:333)
```suggestion
| [Host Monitoring Connection Plugin v3](./using-plugins/UsingTheHostMonitoringPlugin.md#host-monitoring-plugin-v3) | `efm3` | Aurora, RDS Multi-AZ DB Cluster | Enables enhanced host connection failure monitoring with a per-connection design: the connection is registered with the host monitor once, so the per-call overhead is constant and monitor memory scales with the number of connections rather than with query throughput. Uses the same configuration parameters as `efm` and `efm2`. | None | 4.7.0 |
```
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 21:02 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 21:02 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java:427)
Should this catch be broadened to Exception?
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 21:10 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 21:10 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java:64)
Do these both need to be added to the Properties table in the README?
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 21:13 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 21:13 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java:178)
I think there is quite a bit of duplication here between the different HostMonitors - curious if at the very least `MONITORING_PROPERTY_PREFIX` should be defined in one place?
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 21:45 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 21:45 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java:407)
it looks like this aborts the connection if it has been in flight for more then `this.failureDetectionTimeNanos`, should it be `this.failureDetectionTimeNanos + this.failureDetectionCount * this.failureDetectionInvervalMs` instead?
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 21:49 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 21:49 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java:315)
`consecutiveFailures` is only reset by a successful probe. When probing stops because no call is due any more, the partial count carries over into the next probing episode, which might start hours later.
Example with `failureDetectionCount=3`: a long query passes `failureDetectionTime`, 2 probes fail (a short network blip, or the monitoring connection dropped by an idle timeout), and then the query finishes, so probing stops with `consecutiveFailures=2`. Much later another long query becomes due. Its first probe fails once, the count reaches 3, the host is marked unhealthy, and a healthy query is aborted after a single failed probe.
Resetting the count when there's nothing due keeps "consecutive" scoped to one episode:
```suggestion
if (!this.hostUnhealthy.get() && !this.hasDueSlot(now)) {
// Nothing has been in flight long enough to justify a probe. Scan again shortly; a probe is not needed until the
// grace period of some call has elapsed, so a coarse tick does not delay detection. Earlier failed probes no
// longer form a consecutive run.
this.consecutiveFailures.set(0);
this.sleepNanos(this.scanTickNanos);
return;
}
```
A unit test would help here: two failed probes, no due slot for one iteration, then one failed probe. The host should not be marked unhealthy.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:00 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:00 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitoringConnectionPluginV3.java:211)
When `identifyConnection` throws, the exception is rethrown and no slot is cached, so every later network-bound call runs identification again. If the failure is persistent, for example missing permissions for the host-id query or an engine or proxy that doesn't support it, every monitored call on that connection fails, so the connection can't be used at all. The javadoc on `getSlot` also says it "falls back to the endpoint itself when the host cannot be identified". The null case (e.g. RDS Proxy) does fall back, but the exception case doesn't.
Falling back to the endpoint may not be ideal either, since the probe follows DNS and, for reader or custom endpoints, can check a different instance than the one the connection is on. Could the plugin instead log a warning, remember that this connection can't be monitored, and pass its calls through unmonitored until the connection changes? That keeps the connection usable, stops identification from being retried on every call, and the monitor never acts on the wrong host's health. If identification is meant to be required, failing during `connect` would be clearer than failing on the first query.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:04 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:04 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java)
`sweepSlots` unregisters any slot that has been idle longer than `monitorDisposalTime`. As a result, `abortIdleConnectionsOnFailure` skips connections that have been idle for more than 10 minutes by default, and long-idle pooled connections are probably the main thing that option is for.
Example: `abortIdleConnectionsOnFailure=true` with the default `monitorDisposalTime=600000`. Pooled connections that have been idle for 15 minutes are no longer registered. When the host fails, only the recently used connections are aborted. The rest stay in the pool and fail on their next checkout. The docs say idle connections to the host are aborted, with no time limit.
Should idle slots skip eviction when `abortIdleConnections` is set, and rely on the weak reference being cleared to remove them?
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:07 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:07 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitoringConnectionPluginV3.java)
A slot that was claimed `ABORTED` while idle stays `ABORTED` until its next call. That call then marks the host `NOT_AVAILABLE`, no matter how long ago the abort happened or whether the host has recovered since.
Example with `abortIdleConnectionsOnFailure=true`: a short blip marks the host unhealthy, and idle connections are claimed and aborted. The host recovers within seconds. Minutes or hours later the app uses one of those pooled connections, `enter()` returns true, and the plugin marks a healthy host unavailable. That steers host selection, read/write splitting, and failover away from it.
Could this check whether the monitor still considers the host unhealthy (`monitor.isHostUnhealthy()`), or record when the abort happened and ignore old ones? The call fails on the dead connection either way. This only affects whether a stale availability signal is published.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:09 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:09 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java)
`ABORT_EXECUTOR` is one static, unbounded cached thread pool shared by every host, and nothing shuts it down in `Driver.releaseResources()`.
- Thread burst: `connection.abort(executor)` hands the close work to the executor. During a writer outage with 2,000 in-flight connections, `abortConnections` submits 2,000 aborts in one pass, and the cached pool can start up to 2,000 threads at once. efm2 used one abort thread per monitor.
- Lifecycle: a static executor that `Driver.releaseResources()` doesn't shut down keeps its threads alive after shutdown (up to the cached pool's keep-alive)
Should this be a bounded pool (for example a fixed pool, or a cached pool with a cap), with a shutdown hooked into `releaseResources()`?
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:16 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:16 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java:126)
When the cached monitor turns out to be stopped, this method calls `stopAndRemove(key)` unconditionally. That can stop and evict a new monitor that another thread has just installed under the same key.
Example: threads A and B both get the stopped monitor M1. A's `M1.register` fails, A calls `stopAndRemove(key)`, `runIfAbsent` creates M2, and A's slot registers with M2. Then B's `M1.register` fails, B calls `stopAndRemove(key)`, which stops and removes M2, and B creates M3. A's slot now points at the stopped M2, so A's in-flight call runs unmonitored and a hung query isn't aborted.
Should the entry be removed only if the cached monitor is still the stopped instance that was observed? `MonitorService` doesn't have a conditional remove yet, so this might need a small `remove(class, key, expected)` addition.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:17 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:17 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/ConnectionPluginChainBuilder.java:91)
`efm3` is registered here, but `ConnectionPluginManager.pluginNameByClass` (around line 93-94) only has entries for `HostMonitoringConnectionPluginV1` (`plugin:efm`) and `HostMonitoringConnectionPluginV2` (`plugin:efm2`). Could you add `put(HostMonitoringConnectionPluginV3.class, "plugin:efm3");` there?
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:21 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:21 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java)
nit: the monitor initializer lambda reads instance fields, so it captures `this`, and `MonitorServiceImpl.MonitorItem` keeps that supplier for as long as the monitor lives. That keeps the first connection's `HostMonitorServiceV3Impl`, its services container, its `PluginService`, and that connection reachable after the connection is dropped, which works against the weak references in `MonitoredConnectionSlot`. efm2 has the same pattern, but since efm3 is meant to be leak-resistant, copying the fields into locals would avoid it:
```suggestion
final Properties props = this.properties;
final int detectionTime = this.failureDetectionTimeMillis;
final int detectionInterval = this.failureDetectionIntervalMillis;
final int detectionCount = this.failureDetectionCount;
final long disposalTime = this.monitorDisposalTimeMillis;
final boolean abortIdle = this.abortIdleConnections;
final TelemetryCounter counter = this.abortedConnectionsCounter;
return this.coreMonitorService.runIfAbsent(
HostMonitorV3Impl.class,
key,
this.servicesContainer,
this.pluginService.getProperties(),
(monitorServicesContainer) -> new HostMonitorV3Impl(
monitorServicesContainer,
hostSpec,
props,
detectionTime,
detectionInterval,
detectionCount,
disposalTime,
abortIdle,
counter));
```
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:22 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 22:22 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java)
nit: `monitorDisposalTime` is now defined three times (V1, V2, and here) with the same name, default, and description. `HostMonitorV3Impl.buildMonitoringProperties` and `checkConnectionStatus` also largely repeat efm2's `monitoring-` prefix handling and probe logic. Reusing the existing property (or moving it to `HostMonitoringConnectionBasePlugin`) and extracting the shared property and probe code into `efm.base` would keep a future fix in one plugin from missing the others.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on docs/using-the-jdbc-driver/using-plugins/UsingTheHostMonitoringPlugin.md)
Done. The shared notes (detection deadline, using several EFM plugins together) now cover efm3.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on docs/using-the-jdbc-driver/CompatibilityCrossPlugins.md:10)
Added efm3 next to efm2 in CompatibilityDatabaseTypes.md and CompatibilityEndpoints.md; its compatibility is the same as efm2's.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java)
Yes, done. Before, a runtime exception from the probe was caught by the outer loop and never counted, so the host could not be declared unhealthy while the driver kept throwing. It now counts as a failed probe and is logged at FINEST. The abort call catches Exception as well.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java)
Added abortIdleConnectionsOnFailure. monitorDisposalTime was already listed; its reference now points to the shared definition in HostMonitoringConnectionBasePlugin.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java)
monitorDisposalTime now lives in HostMonitoringConnectionBasePlugin (the V1/V2 service constants stay as aliases so existing references compile), and the monitoring- prefix handling moved to efm.base.HostMonitoringProperties, used by all three monitors. I kept the probes separate: efm and efm2 compute the isValid timeout differently, and efm also submits a per-node counter, so sharing that code would change efm/efm2 behaviour. Happy to do it as a follow-up.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java)
monitorDisposalTime now lives in HostMonitoringConnectionBasePlugin (the V1/V2 service constants stay as aliases so existing references compile), and the monitoring- prefix handling moved to efm.base.HostMonitoringProperties, used by all three monitors. I kept the probes separate: efm and efm2 compute the isValid timeout differently, and efm also submits a per-node counter, so sharing that code would change efm/efm2 behaviour. Happy to do it as a follow-up.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java:407)
The probe count is already in the verdict: the host is only marked unhealthy after failureDetectionCount consecutive failed probes, and probing starts once some call has passed failureDetectionTime. So the first call is aborted at roughly failureDetectionTime + (count - 1) x interval. This check only decides which calls are eligible once the host is known to be down. A call that passes the grace period while the host is still failing is aborted then, which is what efm2 does with contexts that become active while the node is unhealthy. Adding count x interval per call would let calls to a host we already know is down run for another full detection cycle.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java:315)
Good catch, thanks. Fixed as suggested, with the test you described (consecutiveFailures_doNotCarryOverIntoALaterProbingEpisode).
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitoringConnectionPluginV3.java:211)
You're right, a persistent failure made the connection unusable. efm and efm2 avoid that by caching the endpoint before identifying, and efm3 now does the same: the call that hits the error fails, and afterwards the connection is monitored through its endpoint without retrying identification. The javadoc describes this. I kept the efm/efm2 behaviour rather than passing calls through unmonitored so efm3 stays a drop-in replacement. The endpoint-follows-DNS concern applies to all three plugins, so I'd rather handle it separately.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java)
Agreed. With abortIdleConnectionsOnFailure=true, idle slots are no longer evicted by monitorDisposalTime; they go away only when the connection is closed or garbage collected. Documented and tested.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:52 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitoringConnectionPluginV3.java)
Fixed. The availability signal on the next call is now sent only if the monitor still considers the host unhealthy; the call still fails on the aborted connection. Added a test for the recovered case.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:53 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorV3Impl.java)
Replaced with one shared pool of at most 4 threads (further aborts queue) whose threads exit after 60 s idle. Driver.releaseResources() now shuts it down, and a later abort starts a new pool.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:53 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java:126)
Added MonitorService.stopAndRemove(class, key, expectedMonitor), which removes the entry only if it still holds that instance. It is atomic in MonitorServiceImpl; the interface default is a best-effort fallback so other implementations keep compiling. The service uses it now, and there's a test in MonitorServiceImplTest.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:53 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/ConnectionPluginChainBuilder.java:91)
Added.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-18 22:53 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
(on wrapper/src/main/java/software/amazon/jdbc/plugin/efm/v3/HostMonitorServiceV3Impl.java)
Applied your suggestion, thanks.
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237 (approved): feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 23:40 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: aaron-congo (@aaron-congo) @ 2026-11-18 23:40 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237 (approved): feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-18 23:41 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sophia-bq (@sophia-bq) @ 2026-11-18 23:41 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-19 00:28 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-19 00:28 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
@sergiyvamz merged this (commit 0db9da69dca2).
^ permalink raw [nested|flat] 39+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
@ 2026-11-19 00:28 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
37 siblings, 0 replies; 39+ messages in thread
From: sergiyvamz (@sergiyvamz) @ 2026-11-19 00:28 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
@sergiyvamz closed this.
^ permalink raw [nested|flat] 39+ messages in thread
end of thread, other threads:[~2026-11-19 00:28 UTC | newest]
Thread overview: 39+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-13 20:13 [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 19:57 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 20:39 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 20:40 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 20:55 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 20:56 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 20:57 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 21:02 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 21:10 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 21:13 ` "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-18 21:45 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 21:49 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:00 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:04 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:07 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:09 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:16 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:17 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:21 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:22 ` "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:52 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 22:53 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-18 23:40 ` Re: [aws/aws-advanced-jdbc-wrapper] PR #2237 (approved): feat: add efm3 host monitoring plugin with per-connection registration "aaron-congo (@aaron-congo)" <noreply+aaron-congo@github.com>
2026-11-18 23:41 ` Re: [aws/aws-advanced-jdbc-wrapper] PR #2237 (approved): feat: add efm3 host monitoring plugin with per-connection registration "sophia-bq (@sophia-bq)" <noreply+sophia-bq@github.com>
2026-11-19 00:28 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
2026-11-19 00:28 ` "sergiyvamz (@sergiyvamz)" <noreply+sergiyvamz@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox