pg.ddx.io  aws/aws-advanced-jdbc-wrapper GitHub issues and pull requests  
help / color / mirror / Atom feed
From: sergiyvamz (@sergiyvamz) <noreply+sergiyvamz@github.com>
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
Subject: [aws/aws-advanced-jdbc-wrapper] PR #2237: feat: add efm3 host monitoring plugin with per-connection registration
Date: Fri, 13 Nov 2026 20:13:37 +0000
Message-ID: <gh-aws-aws-advanced-jdbc-wrapper-2237@github.com> (raw)

## Summary

Adds `efm3`, a new Enhanced Failure Monitoring plugin that registers the connection with the host monitor once instead of registering every JDBC call. `efm` and `efm2` are unchanged and `efm2` remains the default.

## Why

In `efm`/`efm2` every network-bound call takes a JVM-wide context-pool lock twice, runs a contended `ConcurrentHashMap.compute` on the host key, and (`efm2`) allocates a `WeakReference`. Monitor memory and CPU scale with calls per second (`calls/s x failureDetectionTime` live contexts). Several reported leaks and races (#675, #966, #893, #2061, #2195) come from the per-call context lifecycle.

## How `efm3` works

- One `MonitoredConnectionSlot` per physical connection, holding the connection weakly and an `AtomicLong inFlightSince`. Per call: `getAndSet(now)` on entry, `getAndSet(0)` on exit. No allocation, no lock, no shared writes.
- One monitor thread per host scans slots (O(connections)), probes only once some call has outlived `failureDetectionTime`, and after `failureDetectionCount` consecutive failed probes claims each due connection with `compareAndSet(start, ABORTED)` before aborting it through `dialect.abortConnection`. The claim/exit pair is linearizable, so a completed call is never reported aborted and no connection is aborted twice.
- The monitor thread updates the heartbeat itself, so `MonitorService` stuck detection works. Slots idle longer than `monitorDisposalTime` are swept; the monitor expires when it has no slots.
- Monitoring connection timeouts come from `monitoring-connectTimeout` / `monitoring-socketTimeout`, defaulting to `failureDetectionInterval` when unset. The application's own timeouts are not inherited.
- Host resolution happens once per connection and never calls `setCurrentConnection` or `setRoutedHostSpec`; identification errors surface as `SQLException`.
- New parameter `abortIdleConnectionsOnFailure` (default `false`) also aborts idle connections to an unhealthy host.

## Measurements

JDK 17, 14 cores, fake target driver (`benchmarks/`). Only the comparison between plugins is meaningful; `executionTime` is a trivial subscribing plugin that prices the pipeline itself.

Many connections to one host, one real plugin chain per thread (`EfmContentionBenchmarks`, ops/ms over all threads, GC time per 5 s window):

| threads | executionTime | efm2 | efm3 | GC time efm2 / efm3 |
|---|---|---|---|---|
| 1 | 166 | 127 | 168 | 545 ms / 23 ms |
| 4 | 493 | 252 | 534 | 1,511 / 83 |
| 8 | 741 | 258 | 685 | 1,468 / 135 |
| 14 | 698 | 268 | 737 | 1,811 / 145 |
| 32 | 125* | 246 | 694 | 1,752 / 177 |

efm2 stops scaling at 4 threads; efm3 tracks the pipeline floor. (*`executionTime` writes a `static long` from every thread, its own contention.)

Steady state, 64 connections x 1,000 calls/s to one host for 90 s (`EfmSoakHarness`): efm2 holds ~1.95M context references and 124-126 MB heap after GC with 2 monitoring threads; efm3 holds 64 slots and 5-6 MB with 1 thread. With a connection replaced every 5,000 calls and half of them leaked without `releaseResources`, efm3's registered slots oscillated 128-156 and returned to 64 at shutdown (sweep after `monitorDisposalTime`).

Single thread (`RealPluginChainBenchmarks`): efm2 6,733 +- 1,657 ns/op, efm3 4,732 +- 306 ns/op, pipeline floor 4,940 +- 1,479.

## Changes

- `plugin/efm/v3`: `HostMonitoringConnectionPluginV3` (+ factory), `HostMonitorServiceV3Impl`, `HostMonitorV3Impl`, `MonitoredConnectionSlot`.
- `ConnectionPluginChainBuilder`: code `efm3`, abbreviation `e3`, weight 1550.
- Docs: new "Host Monitoring Plugin v3" section, plugin table row, telemetry metrics, compatibility matrix.
- Tests: 45 unit tests for the slot protocol, monitor loop, service and plugin; `EFM3Test` container test mirroring `EFM2Test`; `PerformanceTest` gains `efm3` and `failover2,efm3` detection-time rounds.
- Benchmarks: `EfmContentionBenchmarks`, `EfmSoakHarness`, shared `RealPluginChain` support, `efm3` chain in `RealPluginChainBenchmarks`. `BenchmarkServices` now honours the context service a plugin installs, so `efm`/`efm2` are measured with the production pooled service rather than the simple one.

## Verification

`checkstyleMain/Test`, `spotbugsMain/Test`, `compileJava -PenableCheckerFramework`, `:benchmarks:jmhJar`, and `test --tests "software.amazon.jdbc.plugin.efm.*"` (137 tests) pass locally. Detection timing on a real cluster (`PerformanceTest`) still needs the Aurora environment.


view thread (39+ messages)  latest in thread

Message-ID: <gh-aws-aws-advanced-jdbc-wrapper-2237@github.com>
Permalink:  ../gh-aws-aws-advanced-jdbc-wrapper-2237@github.com/

 · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox