pg.ddx.io  patroni/patroni GitHub issues and pull requests  
help / color / mirror / Atom feed
From: CyberDem0n (@CyberDem0n) <noreply+CyberDem0n@github.com>
To: patroni/patroni <noreply+patroni-patroni@github.com>
Subject: [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
Date: Thu, 19 Nov 2026 19:49:43 +0000
Message-ID: <gh-patroni-patroni-3810@github.com> (raw)

## Problem

`tests/test_log.py::TestPatroniLogger::test_patroni_logger` is flaky. For example, [this `unit (ubuntu)` job](https://github.com/patroni/patroni/actions/runs/44597077266/job/135818151641) failed on Python 3.10 with:

```
>       self.assertEqual(logger.records_lost, 0)
E       AssertionError: 1 != 0
tests/test_log.py:86: AssertionError
```

After `logger.reload_config()`, the logger thread closes the old handler at the top of its next loop iteration. Because `logging.Handler.close` is mocked to raise, the thread then logs `Failed to close the old log handler`. This happens asynchronously. If the thread was descheduled long enough, it logged that record while the test had `Queue.put_nowait` patched to raise `Full` (to check that `shutdown()` exits with `SystemExit`), so `QueueHandler.emit()` counted the record as lost.

## Fix

Swap the two `shutdown()` checks. Call the real `logger.shutdown()` first, which joins the logger thread, and only then check the `SystemExit` path with `put_nowait` raising `Full`. The thread logs "Failed to close…" before it dequeues the `None` sentinel, so once it has been joined nothing else can log while `put_nowait` is mocked. Both `shutdown()` paths are still exercised, and coverage of `patroni/log.py` is unchanged.

## Testing

- I reproduced the failure deterministically with a throwaway pytest plugin that adds delays to `PatroniLogger._close_old_handlers()` (only while the old file handler is pending) and to `PatroniLogger.shutdown()`. With the plugin, the old test fails with `1 != 0` and the new one passes.
- 400 runs of the fixed test as separate processes (8 in parallel on 2 CPUs) all passed, with no hangs.
- `tests/test_log.py` passes, and `python setup.py flake8` is clean.


inline patch 1: tests/test_log.py (+3/-1 lines) download inline-1.patchdiff --git a/tests/test_log.py b/tests/test_log.py index 11e8178bc..5f9e795ca 100644 --- a/tests/test_log.py +++ b/tests/test_log.py @@ -79,9 +79,11 @@ def test_patroni_logger(self): Mock(side_effect=[logging.LogRecord('', logging.INFO, '', 0, '', (), None), Exception])): logging.exception('test') logging.error('test') + # Stop the logger thread before mocking put_nowait(), otherwise its asynchronous + # "Failed to close the old log handler" record could be lost and counted in records_lost. + self.assertRaises(Exception, logger.shutdown) with patch.object(Queue, 'put_nowait', Mock(side_effect=Full)): self.assertRaises(SystemExit, logger.shutdown) - self.assertRaises(Exception, logger.shutdown) self.assertLessEqual(logger.queue_size, 2) # "Failed to close the old log handler" could be still in the queue self.assertEqual(logger.records_lost, 0) del config['log']['traceback_level']

view thread (7+ messages)  latest in thread

Message-ID: <gh-patroni-patroni-3810@github.com>
Permalink:  ../gh-patroni-patroni-3810@github.com/

 ·  · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox