pg.ddx.io aws/aws-advanced-jdbc-wrapper GitHub issues and pull requests
help / color / mirror / Atom feed[aws/aws-advanced-jdbc-wrapper] issue #2258: "skipFailoverOnInterruptedThread" is bypassed when the interrupted connection is already closed: failover starts anyway
2+ messages / 1 participants
[nested] [flat]
* [aws/aws-advanced-jdbc-wrapper] issue #2258: "skipFailoverOnInterruptedThread" is bypassed when the interrupted connection is already closed: failover starts anyway
@ 2026-11-19 13:08 "bauna (@bauna)" <noreply+bauna@github.com>
0 siblings, 0 replies; 2+ messages in thread
From: bauna (@bauna) @ 2026-11-19 13:08 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
### Describe the bug
## Summary
`skipFailoverOnInterruptedThread=true` (added for #1283 by #1284) is checked in one place only:
`shouldExceptionTriggerConnectionSwitch`. The failover plugins have two other ways into `failover()` that do
not check it. Both are reached when the current connection `isClosed()`:
| Plugin | Method | Path to `failover()` |
|---|---|---|
| `failover2` | `execute()` | `getCurrentConnection().isClosed()` → `pickNewConnection()` → `failover()` |
| `failover2` | `invalidInvocationOnClosedConnection()` | `pickNewConnection()` → `failover()` |
| `failover` (v1) | `execute()` | `getCurrentConnection().isClosed()` → `pickNewConnection()` |
| `failover` (v1) | `execute()` → `invalidInvocationOnClosedConnection()` | `pickNewConnection()` |
When a virtual thread that is blocked in a socket read is interrupted, the JDK closes the socket
(`java.net.SocketException: Closed by interrupt`). The first exception goes through
`dealWithOriginalException`, and failover is skipped as expected. But the physical connection is now closed. The next
JDBC call on the same wrapper connection that is not `close`/`abort` (for example, the `rollback()` that the
connection pool or the transaction manager sends) goes into `execute()`. There, the `isClosed()` branch calls
`failover()` with no check of the interrupt flag.
So with `skipFailoverOnInterruptedThread=true`, an interrupted thread still starts a writer failover.
## Impact
1. **False failover on a healthy cluster.** The `failover2` writer failover calls
`pluginService.forceRefreshHostList(true, failoverTimeoutMs)`. The thread is still interrupted, so
`ClusterTopologyMonitorImpl.waitForTopologyUpdate` catches `InterruptedException` on its first `wait(1000)`,
resets the flag and returns `null`. The plugin logs at SEVERE
`Failover.unableToRefreshHostList` ("The request to discover the new topology timed out or was unsuccessful.")
and throws `FailoverFailedSQLException`. Error monitoring shows this as a failed database failover, but
the database is healthy.
2. **Topology monitor goes into panic mode for no reason.** `forceRefresh(verifyTopology=true, …)` sets
`monitoringConnection` to `null` ("Enter panic mode") **before** it waits. Each interrupted query therefore
makes the monitor verify the topology again, and node threads open one connection per host. When an
application cancels work often (request timeouts, structured-concurrency deadlines, reactive
cancellation), this makes extra connections to every instance in the cluster.
3. The exception that the application gets is `FailoverFailedSQLException` instead of the original I/O error,
which hides the real cause (cancellation).
## Environment
- aws-advanced-jdbc-wrapper 4.4.0. The same code is on `main` at `9ca9a48` (4.6.0-dev, 2026-11-19).
- PostgreSQL (pgJDBC), Aurora PostgreSQL, `wrapperPlugins` includes `failover2`, `wrapperDialect=aurora-pg`.
- `skipFailoverOnInterruptedThread=true`, `failoverTimeoutMs=30000`.
- Java 25, virtual threads. HikariCP as the pool.
- The interrupt comes from `StructuredTaskScope` with `withTimeout(...)`: when the timeout expires, the scope interrupts
the forked virtual threads. Any other interrupt source gives the same result (`Future.cancel(true)`, Reactor
cancellation as in #1283, `ExecutorService.shutdownNow()`).
## Observed sequence (production logs, same thread, same millisecond)
```
12:27:06.662 WARN com.zaxxer.hikari.pool.ProxyConnection WRITER - Connection org.postgresql.jdbc.PgConnection@4c2950b0
marked as broken because of SQLSTATE(08006), ErrorCode(0) cause: Closed by interrupt
12:27:06.667 WARN org.hibernate.orm.jdbc.error An I/O error occurred while sending to the backend.
12:27:06.674 ERROR software.amazon.jdbc.plugin.failover2.FailoverConnectionPlugin
The request to discover the new topology timed out or was unsuccessful.
12:27:06.680 ERROR <application> Failed to process ... : Closed by interrupt
```
The query started 45 s earlier. 45 s is the application's structured-concurrency deadline. The topology error comes
12 ms after the I/O error, so the 30 s `failoverTimeoutMs` was not used. `waitForTopologyUpdate` stopped
at once because the thread was interrupted. This matches the path above. The cluster had no failover or
writer change at that time.
## Steps to reproduce
1. Configure a `failover2` connection to an Aurora PostgreSQL cluster with `skipFailoverOnInterruptedThread=true`.
2. On a virtual thread, run `select pg_sleep(30)` in a transaction (`autoCommit=false`).
3. From another thread, interrupt the virtual thread while the query runs.
4. Catch the `SQLException` (SQLSTATE `08006`, cause `SocketException: Closed by interrupt`) and, still on the
interrupted thread, call `connection.rollback()`. This is what Spring and HikariCP do on an exception.
**Expected:** `rollback()` fails or is a no-op. No failover. No `Failover.unableToRefreshHostList`.
The topology monitor does not enter panic mode.
**Actual:** `rollback()` → `execute()` → `isClosed()` is `true` → `pickNewConnection()` → `failover()` →
`failoverWriter()` → `forceRefreshHostList(true, …)` puts the monitor into panic mode → returns `false` → SEVERE
`Failover.unableToRefreshHostList` → `FailoverFailedSQLException`.
## Relevant code (`main`, `9ca9a48`)
`wrapper/src/main/java/software/amazon/jdbc/plugin/failover2/FailoverConnectionPlugin.java`:
```java
// execute(), ~L236
if (this.pluginService.getCurrentConnection() != null
&& !this.canDirectExecute(methodName)
&& !this.closedExplicitly
&& this.pluginService.getCurrentConnection().isClosed()) {
this.pickNewConnection(); // no interrupt check
}
// ~L792
protected void pickNewConnection() throws SQLException {
if (this.isClosed && this.closedExplicitly) { ... return; }
this.failover(); // no interrupt check
}
// ~L801: the only place the property is checked
protected boolean shouldExceptionTriggerConnectionSwitch(final Throwable t) {
...
if (this.skipFailoverOnInterruptedThread && Thread.currentThread().isInterrupted()) {
return false;
}
```
`wrapper/src/main/java/software/amazon/jdbc/hostlistprovider/ClusterTopologyMonitorImpl.java`:
```java
public List<HostSpec> forceRefresh(final boolean verifyTopology, final long timeoutMs) {
if (verifyTopology) {
this.monitoringConnection.set(null); // panic mode, even if the caller is interrupted
}
return this.waitForTopologyUpdate(timeoutMs); // returns null at once on InterruptedException
}
```
`wrapper/src/main/java/software/amazon/jdbc/plugin/failover/FailoverConnectionPlugin.java` (v1) has the same
pattern in `execute()` (~L250 and ~L270).
## Related
- #1283 — The Wrapper initiates a failover when using virtual threads and a thread is interrupted
- #1284 — skip failover on interrupted thread (covers only `shouldExceptionTriggerConnectionSwitch`)
### Expected Behavior
When skipFailoverOnInterruptedThread=true and the calling thread is interrupted, the failover plugin should not start a failover on any code path. That covers the exception handler, and also any later JDBC call on the same connection after the interrupt has closed the underlying socket, such as the rollback() that a connection pool or transaction manager sends during cleanup. That call should fail fast with a SQLException (for example SQLSTATE 08003), so the pool discards the connection, as the property’s documentation says (“the Connection should be disposed”). The plugin should not call failover(). It should not log Failover.unableToRefreshHostList at SEVERE, and it should not put the cluster topology monitor into panic mode. A cancelled query on a healthy cluster should never look like a failed database failover, and it should not cause extra connections to every instance in the cluster.
### What plugins are used? What other connection properties were set?
iam, initialConnection, auroraConnectionTracker, failover2, efm2 with wrapperDialect=aurora-pg.
### Current Behavior
With `skipFailoverOnInterruptedThread=true`, the first exception after the interrupt is handled correctly:
`shouldExceptionTriggerConnectionSwitch` sees the interrupt flag and skips failover. But the interrupt has closed the
physical connection's socket. The next JDBC call on the same wrapper connection that is not `close` or `abort`
(for example, the `rollback()` from HikariCP or the transaction manager) goes through `execute()`. There
`getCurrentConnection().isClosed()` is `true`, so the plugin calls `pickNewConnection()` → `failover()` without
checking the interrupt flag. `failover2` starts a writer failover and calls `forceRefreshHostList(true, failoverTimeoutMs)`.
That puts the cluster topology monitor into panic mode, and the monitor then returns at once, because the waiting
thread is interrupted. The plugin then logs `Failover.unableToRefreshHostList` at SEVERE ("The request to discover the
new topology timed out or was unsuccessful.") and throws `FailoverFailedSQLException`. All of this happens within
about 12 ms of the original I/O error, on a healthy cluster with no writer change.
### Reproduction Steps
```java
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.sql.Statement;
import java.util.Properties;
import java.util.logging.ConsoleHandler;
import java.util.logging.Level;
import java.util.logging.Logger;
/**
* Reproduces: failover2 starts a failover on an interrupted thread although skipFailoverOnInterruptedThread=true.
*
* <p>Run (Java 21+, aws-advanced-jdbc-wrapper and pgjdbc on the classpath):
* <pre>
* java -cp aws-advanced-jdbc-wrapper-4.4.0.jar:postgresql-42.7.8.jar Repro.java \
* "jdbc:aws-wrapper:postgresql://<cluster-endpoint>:5432/postgres" <user> <password> aurora-pg
* </pre>
*/
public class Repro {
public static void main(String[] args) throws Exception {
Logger wrapperLogger = Logger.getLogger("software.amazon.jdbc");
wrapperLogger.setLevel(Level.FINE);
ConsoleHandler handler = new ConsoleHandler();
handler.setLevel(Level.FINE);
wrapperLogger.addHandler(handler);
Properties props = new Properties();
props.setProperty("user", args[1]);
props.setProperty("password", args[2]);
props.setProperty("wrapperDialect", args[3]);
props.setProperty("wrapperPlugins", "failover2");
props.setProperty("skipFailoverOnInterruptedThread", "true");
props.setProperty("failoverTimeoutMs", "30000");
Thread worker = Thread.ofVirtual().start(() -> {
try (Connection conn = DriverManager.getConnection(args[0], props)) {
conn.setAutoCommit(false);
try (Statement stmt = conn.createStatement()) {
stmt.execute("select pg_sleep(30)");
} catch (SQLException e) {
// Expected: 08006 "An I/O error occurred while sending to the backend." (Closed by interrupt), no failover.
System.out.printf("query failed: %s %s, interrupted=%s, isClosed=%s%n",
e.getSQLState(), e.getMessage(), Thread.currentThread().isInterrupted(), conn.isClosed());
// Same cleanup a pool or a transaction manager does. This call goes through
// FailoverConnectionPlugin.execute() -> isClosed() -> pickNewConnection() -> failover().
try {
conn.rollback();
System.out.println("rollback: ok");
} catch (SQLException e2) {
System.out.printf("rollback failed: %s: %s%n", e2.getClass().getName(), e2.getMessage());
}
}
} catch (SQLException e) {
System.out.printf("connection failed: %s%n", e);
}
});
Thread.sleep(2_000);
worker.interrupt();
worker.join();
}
}
```
### Possible Solution
## Proposed fix
Check the property in `pickNewConnection()` in both plugins, so that all paths to `failover()` honor it:
```java
protected void pickNewConnection() throws SQLException {
if (this.isClosed && this.closedExplicitly) {
LOGGER.fine(() -> Messages.get("Failover.connectionClosedExplicitly"));
return;
}
if (this.skipFailoverOnInterruptedThread && Thread.currentThread().isInterrupted()) {
LOGGER.fine(() -> Messages.get("Failover.skipFailoverOnInterruptedThread"));
throw new SQLException(Messages.get("Failover.skipFailoverOnInterruptedThread"), "08003");
}
this.failover();
}
```
Throwing `08003` (connection does not exist) keeps the contract that the documentation gives for the property: "This
may leave the Connection in an invalid state so the Connection should be disposed". The pool then discards the
connection.
Two smaller changes would also help:
- In `ClusterTopologyMonitorImpl.forceRefresh`, do not enter panic mode (`monitoringConnection.set(null)`) when the
calling thread is already interrupted. That caller cannot wait for the result.
- Add a test for the sequence "interrupt during the query → the next non-close call on the same connection" with
`skipFailoverOnInterruptedThread=true`. Assert that `failover()` is not called.
### Additional Information/Context
_No response_
### The AWS Advanced JDBC Wrapper version used
4.4.0
### JDK version used
Java 25
### Operating System and version
linux/arm64
^ permalink raw [nested|flat] 2+ messages in thread
* Re: [aws/aws-advanced-jdbc-wrapper] issue #2258: "skipFailoverOnInterruptedThread" is bypassed when the interrupted connection is already closed: failover starts anyway
@ 2026-11-19 13:08 "bauna (@bauna)" <noreply+bauna@github.com>
0 siblings, 0 replies; 2+ messages in thread
From: bauna (@bauna) @ 2026-11-19 13:08 UTC (permalink / raw)
To: aws/aws-advanced-jdbc-wrapper <noreply+aws-aws-advanced-jdbc-wrapper@github.com>
@bauna applied label "bug".
^ permalink raw [nested|flat] 2+ messages in thread
end of thread, other threads:[~2026-11-19 13:08 UTC | newest]
Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 13:08 [aws/aws-advanced-jdbc-wrapper] issue #2258: "skipFailoverOnInterruptedThread" is bypassed when the interrupted connection is already closed: failover starts anyway "bauna (@bauna)" <noreply+bauna@github.com>
2026-11-19 13:08 Re: [aws/aws-advanced-jdbc-wrapper] issue #2258: "skipFailoverOnInterruptedThread" is bypassed when the interrupted connection is already closed: failover starts anyway "bauna (@bauna)" <noreply+bauna@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox