From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
Date: Thu, 19 Nov 2026 19:49:43 +0000
Message-ID: <gh-patroni-patroni-3810@github.com> (raw)
## Problem
`tests/test_log.py::TestPatroniLogger::test_patroni_logger` is flaky. For example, [this `unit (ubuntu)` job](https://github.com/patroni/patroni/actions/runs/44597077266/job/135818151641) failed on Python 3.10 with:
```
> self.assertEqual(logger.records_lost, 0)
E AssertionError: 1 != 0
tests/test_log.py:86: AssertionError
```
After `logger.reload_config()`, the logger thread closes the old handler at the top of its next loop iteration. Because `logging.Handler.close` is mocked to raise, the thread then logs `Failed to close the old log handler`. This happens asynchronously. If the thread was descheduled long enough, it logged that record while the test had `Queue.put_nowait` patched to raise `Full` (to check that `shutdown()` exits with `SystemExit`), so `QueueHandler.emit()` counted the record as lost.
## Fix
Swap the two `shutdown()` checks. Call the real `logger.shutdown()` first, which joins the logger thread, and only then check the `SystemExit` path with `put_nowait` raising `Full`. The thread logs "Failed to close…" before it dequeues the `None` sentinel, so once it has been joined nothing else can log while `put_nowait` is mocked. Both `shutdown()` paths are still exercised, and coverage of `patroni/log.py` is unchanged.
## Testing
- I reproduced the failure deterministically with a throwaway pytest plugin that adds delays to `PatroniLogger._close_old_handlers()` (only while the old file handler is pending) and to `PatroniLogger.shutdown()`. With the plugin, the old test fails with `1 != 0` and the new one passes.
- 400 runs of the fixed test as separate processes (8 in parallel on 2 CPUs) all passed, with no hangs.
- `tests/test_log.py` passes, and `python setup.py flake8` is clean.
inline patch 1: tests/test_log.py (+3/-1 lines) download inline-1.patchdiff --git a/tests/test_log.py b/tests/test_log.pyindex 11e8178bc..5f9e795ca 100644--- a/tests/test_log.py+++ b/tests/test_log.py@@ -79,9 +79,11 @@ def test_patroni_logger(self):
Mock(side_effect=[logging.LogRecord('', logging.INFO, '', 0, '', (), None), Exception])):
logging.exception('test')
logging.error('test')
+ # Stop the logger thread before mocking put_nowait(), otherwise its asynchronous+ # "Failed to close the old log handler" record could be lost and counted in records_lost.+ self.assertRaises(Exception, logger.shutdown)
with patch.object(Queue, 'put_nowait', Mock(side_effect=Full)):
self.assertRaises(SystemExit, logger.shutdown)
- self.assertRaises(Exception, logger.shutdown)
self.assertLessEqual(logger.queue_size, 2) # "Failed to close the old log handler" could be still in the queue
self.assertEqual(logger.records_lost, 0)
del config['log']['traceback_level']