pg.ddx.io patroni/patroni GitHub issues and pull requests
help / color / mirror / Atom feed[patroni/patroni] PR #3810: Fix flaky test_patroni_logger
7+ messages / 3 participants
[nested] [flat]
* [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
@ 2026-11-19 19:49 "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
0 siblings, 0 replies; 7+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 19:49 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
## Problem
`tests/test_log.py::TestPatroniLogger::test_patroni_logger` is flaky. For example, [this `unit (ubuntu)` job](https://github.com/patroni/patroni/actions/runs/44597077266/job/135818151641) failed on Python 3.10 with:
```
> self.assertEqual(logger.records_lost, 0)
E AssertionError: 1 != 0
tests/test_log.py:86: AssertionError
```
After `logger.reload_config()`, the logger thread closes the old handler at the top of its next loop iteration. Because `logging.Handler.close` is mocked to raise, the thread then logs `Failed to close the old log handler`. This happens asynchronously. If the thread was descheduled long enough, it logged that record while the test had `Queue.put_nowait` patched to raise `Full` (to check that `shutdown()` exits with `SystemExit`), so `QueueHandler.emit()` counted the record as lost.
## Fix
Swap the two `shutdown()` checks. Call the real `logger.shutdown()` first, which joins the logger thread, and only then check the `SystemExit` path with `put_nowait` raising `Full`. The thread logs "Failed to close…" before it dequeues the `None` sentinel, so once it has been joined nothing else can log while `put_nowait` is mocked. Both `shutdown()` paths are still exercised, and coverage of `patroni/log.py` is unchanged.
## Testing
- I reproduced the failure deterministically with a throwaway pytest plugin that adds delays to `PatroniLogger._close_old_handlers()` (only while the old file handler is pending) and to `PatroniLogger.shutdown()`. With the plugin, the old test fails with `1 != 0` and the new one passes.
- 400 runs of the fixed test as separate processes (8 in parallel on 2 CPUs) all passed, with no hangs.
- `tests/test_log.py` passes, and `python setup.py flake8` is clean.
inline patch 1: tests/test_log.py (+3/-1 lines) download inline-1.patch
diff --git a/tests/test_log.py b/tests/test_log.py
index 11e8178bc..5f9e795ca 100644
--- a/tests/test_log.py
+++ b/tests/test_log.py
@@ -79,9 +79,11 @@ def test_patroni_logger(self):
Mock(side_effect=[logging.LogRecord('', logging.INFO, '', 0, '', (), None), Exception])):
logging.exception('test')
logging.error('test')
+ # Stop the logger thread before mocking put_nowait(), otherwise its asynchronous
+ # "Failed to close the old log handler" record could be lost and counted in records_lost.
+ self.assertRaises(Exception, logger.shutdown)
with patch.object(Queue, 'put_nowait', Mock(side_effect=Full)):
self.assertRaises(SystemExit, logger.shutdown)
- self.assertRaises(Exception, logger.shutdown)
self.assertLessEqual(logger.queue_size, 2) # "Failed to close the old log handler" could be still in the queue
self.assertEqual(logger.records_lost, 0)
del config['log']['traceback_level']
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
@ 2026-11-19 19:49 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: CyberDem0n (@CyberDem0n) @ 2026-11-19 19:49 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@CyberDem0n requested a review.
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3810 (approved): Fix flaky test_patroni_logger
@ 2026-11-19 19:57 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 19:57 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
@ 2026-11-19 19:57 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 19:57 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet applied label "ci".
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
@ 2026-11-19 19:59 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: codecov[bot] (bot) (@codecov[bot]) @ 2026-11-19 19:59 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
## [Codecov](https://app.codecov.io/gh/patroni/patroni/pull/3810?dropdown=coverage&src=pr&el=h1&utm_m...) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:loudspeaker: Thoughts on this report? [Let us know!](https://github.com/codecov/feedback/issues/255)
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
@ 2026-11-19 21:00 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 21:00 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet merged this (commit 213bc70a664d).
^ permalink raw [nested|flat] 7+ messages in thread
* Re: [patroni/patroni] PR #3810: Fix flaky test_patroni_logger
@ 2026-11-19 21:00 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
5 siblings, 0 replies; 7+ messages in thread
From: hughcapet (@hughcapet) @ 2026-11-19 21:00 UTC (permalink / raw)
To: patroni/patroni <noreply+patroni-patroni@github.com>
@hughcapet closed this.
^ permalink raw [nested|flat] 7+ messages in thread
end of thread, other threads:[~2026-11-19 21:00 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-19 19:49 [patroni/patroni] PR #3810: Fix flaky test_patroni_logger "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 19:49 ` "CyberDem0n (@CyberDem0n)" <noreply+CyberDem0n@github.com>
2026-11-19 19:57 ` Re: [patroni/patroni] PR #3810 (approved): Fix flaky test_patroni_logger "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 19:57 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 19:59 ` "codecov[bot] (bot) (@codecov[bot])" <noreply+codecov[bot]@github.com>
2026-11-19 21:00 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
2026-11-19 21:00 ` "hughcapet (@hughcapet)" <noreply+hughcapet@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox