pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedMI - WAL issue
5+ messages / 4 participants
[nested] [flat]
* MI - WAL issue
@ 2026-07-27 05:57 mahamood hussain <hussain.ieg@gmail.com>
0 siblings, 2 replies; 5+ messages in thread
From: mahamood hussain @ 2026-07-27 05:57 UTC (permalink / raw)
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi team,
I need some urgent assistance.
We're running *PostgreSQL 17.6*, and the application team has reported
an *export
issue*. While investigating, I found the following messages in the
PostgreSQL logs:
2026-07-26 10:58:52.902 MST [1183899] DETAIL: Parameters: $1 =
'1785088732902', $2 = 'ctsserver-11783137027378'
2026-07-26 10:58:56.136 MST [1529364] LOG: checkpoint starting: time
2026-07-26 10:58:56.447 MST [1529364] LOG: request to flush past end
of generated WAL; request 5D116/1, current position C8F/734E66C0
2026-07-26 10:58:56.447 MST [1529364] CONTEXT: writing block 4400857
of relation base/16384/1255337
2026-07-26 10:58:56.448 MST [1529364] ERROR: xlog flush request
5D116/1 is not satisfied --- flushed only to C8F/734E66C0
2026-07-26 10:58:56.448 MST [1529364] CONTEXT: writing block 4400857
of relation base/16384/1255337
Has anyone come across this issue before or know what could cause it? Any
guidance on the root cause or recommended troubleshooting steps would be
greatly appreciated.
This is a *Major Incident (MI)* and requires immediate assistance.
Thanks in advance.
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: MI - WAL issue
@ 2026-07-27 08:06 Jochen Bandhauer <jb@jbitc.de>
parent: mahamood hussain <hussain.ieg@gmail.com>
1 sibling, 0 replies; 5+ messages in thread
From: Jochen Bandhauer @ 2026-07-27 08:06 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
Hello,
is this a standby database? Is the database currently running?
Regards
Jochen
On 7/27/2026 7:57 AM, mahamood hussain wrote:
>
> Hi team,
>
> I need some urgent assistance.
>
> We're running *PostgreSQL 17.6*, and the application team has reported
> an *export issue*. While investigating, I found the following messages
> in the PostgreSQL logs:
>
> |2026-07-26 10:58:52.902 MST [1183899] DETAIL: Parameters: $1 =
> '1785088732902', $2 = 'ctsserver-11783137027378' 2026-07-26
> 10:58:56.136 MST [1529364] LOG: checkpoint starting: time 2026-07-26
> 10:58:56.447 MST [1529364] LOG: request to flush past end of generated
> WAL; request 5D116/1, current position C8F/734E66C0 2026-07-26
> 10:58:56.447 MST [1529364] CONTEXT: writing block 4400857 of relation
> base/16384/1255337 2026-07-26 10:58:56.448 MST [1529364] ERROR: xlog
> flush request 5D116/1 is not satisfied --- flushed only to
> C8F/734E66C0 2026-07-26 10:58:56.448 MST [1529364] CONTEXT: writing
> block 4400857 of relation base/16384/1255337 |
>
> Has anyone come across this issue before or know what could cause it?
> Any guidance on the root cause or recommended troubleshooting steps
> would be greatly appreciated.
>
> This is a *Major Incident (MI)* and requires immediate assistance.
>
> Thanks in advance.
>
--
Mit freundlichen Grüßen
Jochen Bandhauer
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: MI - WAL issue
@ 2026-07-27 08:45 Gaurav Shukla <gsricky391@gmail.com>
parent: mahamood hussain <hussain.ieg@gmail.com>
1 sibling, 1 reply; 5+ messages in thread
From: Gaurav Shukla @ 2026-07-27 08:45 UTC (permalink / raw)
To: mahamood hussain <hussain.ieg@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi,
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: MI - WAL issue
@ 2026-07-27 09:01 mahamood hussain <hussain.ieg@gmail.com>
parent: Gaurav Shukla <gsricky391@gmail.com>
0 siblings, 1 reply; 5+ messages in thread
From: mahamood hussain @ 2026-07-27 09:01 UTC (permalink / raw)
To: Gaurav Shukla <gsricky391@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Thanks for the suggestions.
A few weeks ago, we reduced the PostgreSQL checkpoint interval to 1 minute.
Could this change contribute to the issue we're seeing?
During our investigation, our infrastructure team identified what appears
to be an *NVMe storage issue* on the Azure VM. The kernel logs show
repeated NVMe controller resets followed by multiple read I/O errors across
several NVMe devices:
kernel: nvme: resetting controller due to persistent internal error
kernel: Read ... Host Aborted Command
kernel: I/O error, dev nvme..., op:(READ)
The PostgreSQL error occurred around the same timeframe, which makes us
suspect the underlying storage issue may have caused the WAL flush failure:
request to flush past end of generated WAL
xlog flush request ... is not satisfied
Our latest pgBackRest backup is from July 18, as we currently run weekly
full backups.
Has anyone encountered a similar issue where an NVMe/storage failure
resulted in PostgreSQL reporting WAL flush errors like this? Is there
anything else we should be checking from the PostgreSQL side while the
infrastructure team investigates the storage issue?
On Mon, Jul 27, 2026 at 2:15 PM Gaurav Shukla <gsricky391@gmail.com> wrote:
> Hi,
>
> From the logs, it looks like PostgreSQL is facing an issue while trying to
> flush WAL during a checkpoint. I would suggest checking a few things:
>
> - Check the PostgreSQL logs around the same time to see if there were
> any other WAL, checkpoint, or I/O-related errors.
> - Check the server/OS logs for any disk or filesystem issues around 10:58:56
> MST.
> - Check if there was any unexpected server or PostgreSQL restart,
> storage issue, snapshot/rollback, or failover.
> - Verify the disk and filesystem health and make sure there is enough
> free space.
> - Identify which table or index is related to base/16384/1255337, as
> PostgreSQL was writing to this relation when the error occurred.
> - Check if WAL archiving and replication are working fine, if
> configured.
> - Also, make sure the latest backup is healthy and available for
> recovery before taking any corrective action.
>
> The export issue may just be a symptom of an underlying database or
> storage problem, so I think these checks would be a good starting point.
>
> Thanks & Regards
> Gaurav Shukla
> Ph: +91-7022636963
>
> On Mon, Jul 27, 2026, 11:27 mahamood hussain <hussain.ieg@gmail.com>
> wrote:
>
>> Hi team,
>>
>> I need some urgent assistance.
>>
>> We're running *PostgreSQL 17.6*, and the application team has reported
>> an *export issue*. While investigating, I found the following messages
>> in the PostgreSQL logs:
>>
>> 2026-07-26 10:58:52.902 MST [1183899] DETAIL: Parameters: $1 = '1785088732902', $2 = 'ctsserver-11783137027378'
>> 2026-07-26 10:58:56.136 MST [1529364] LOG: checkpoint starting: time
>> 2026-07-26 10:58:56.447 MST [1529364] LOG: request to flush past end of generated WAL; request 5D116/1, current position C8F/734E66C0
>> 2026-07-26 10:58:56.447 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>> 2026-07-26 10:58:56.448 MST [1529364] ERROR: xlog flush request 5D116/1 is not satisfied --- flushed only to C8F/734E66C0
>> 2026-07-26 10:58:56.448 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>
>> Has anyone come across this issue before or know what could cause it? Any
>> guidance on the root cause or recommended troubleshooting steps would be
>> greatly appreciated.
>>
>> This is a *Major Incident (MI)* and requires immediate assistance.
>>
>> Thanks in advance.
>>
>
^ permalink raw reply [nested|flat] 5+ messages in thread
* Re: MI - WAL issue
@ 2026-07-27 11:38 sayli khedekar <saylikhedekar1@gmail.com>
parent: mahamood hussain <hussain.ieg@gmail.com>
0 siblings, 0 replies; 5+ messages in thread
From: sayli khedekar @ 2026-07-27 11:38 UTC (permalink / raw)
To: mahamood hussain <hussain.ieg@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi ,
Can't confirm the checkpoint_timeout change directly caused the WAL flush
error, but the 1-min checkpoint interval may add extra write I/O -- two
things worth checking to reduce that load:
1) HOT update ratio + fillfactor — for busiest tables, check
n_tup_hot_upd/n_tup_upd in pg_stat_user_tables. If low (and not due to
indexed columns changing), lower fillfactor to 70–90 to allow more HOT
updates.
2) Unused/duplicate indexes — check pg_stat_user_indexes for near-zero
idx_scan and check for duplicates on the same columns.
pg_wal size may also have increased due to more full page image, but I
don't see its relevance here.
Regards,
Sayli
On Mon, Jul 27, 2026 at 2:32 PM mahamood hussain <hussain.ieg@gmail.com>
wrote:
> Thanks for the suggestions.
>
> A few weeks ago, we reduced the PostgreSQL checkpoint interval to 1
> minute. Could this change contribute to the issue we're seeing?
>
> During our investigation, our infrastructure team identified what appears
> to be an *NVMe storage issue* on the Azure VM. The kernel logs show
> repeated NVMe controller resets followed by multiple read I/O errors across
> several NVMe devices:
>
> kernel: nvme: resetting controller due to persistent internal error
> kernel: Read ... Host Aborted Command
> kernel: I/O error, dev nvme..., op:(READ)
>
> The PostgreSQL error occurred around the same timeframe, which makes us
> suspect the underlying storage issue may have caused the WAL flush failure:
>
> request to flush past end of generated WAL
> xlog flush request ... is not satisfied
>
> Our latest pgBackRest backup is from July 18, as we currently run weekly
> full backups.
>
> Has anyone encountered a similar issue where an NVMe/storage failure
> resulted in PostgreSQL reporting WAL flush errors like this? Is there
> anything else we should be checking from the PostgreSQL side while the
> infrastructure team investigates the storage issue?
>
> On Mon, Jul 27, 2026 at 2:15 PM Gaurav Shukla <gsricky391@gmail.com>
> wrote:
>
>> Hi,
>>
>> From the logs, it looks like PostgreSQL is facing an issue while trying
>> to flush WAL during a checkpoint. I would suggest checking a few things:
>>
>> - Check the PostgreSQL logs around the same time to see if there were
>> any other WAL, checkpoint, or I/O-related errors.
>> - Check the server/OS logs for any disk or filesystem issues around 10:58:56
>> MST.
>> - Check if there was any unexpected server or PostgreSQL restart,
>> storage issue, snapshot/rollback, or failover.
>> - Verify the disk and filesystem health and make sure there is enough
>> free space.
>> - Identify which table or index is related to base/16384/1255337, as
>> PostgreSQL was writing to this relation when the error occurred.
>> - Check if WAL archiving and replication are working fine, if
>> configured.
>> - Also, make sure the latest backup is healthy and available for
>> recovery before taking any corrective action.
>>
>> The export issue may just be a symptom of an underlying database or
>> storage problem, so I think these checks would be a good starting point.
>>
>> Thanks & Regards
>> Gaurav Shukla
>> Ph: +91-7022636963
>>
>> On Mon, Jul 27, 2026, 11:27 mahamood hussain <hussain.ieg@gmail.com>
>> wrote:
>>
>>> Hi team,
>>>
>>> I need some urgent assistance.
>>>
>>> We're running *PostgreSQL 17.6*, and the application team has reported
>>> an *export issue*. While investigating, I found the following messages
>>> in the PostgreSQL logs:
>>>
>>> 2026-07-26 10:58:52.902 MST [1183899] DETAIL: Parameters: $1 = '1785088732902', $2 = 'ctsserver-11783137027378'
>>> 2026-07-26 10:58:56.136 MST [1529364] LOG: checkpoint starting: time
>>> 2026-07-26 10:58:56.447 MST [1529364] LOG: request to flush past end of generated WAL; request 5D116/1, current position C8F/734E66C0
>>> 2026-07-26 10:58:56.447 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>> 2026-07-26 10:58:56.448 MST [1529364] ERROR: xlog flush request 5D116/1 is not satisfied --- flushed only to C8F/734E66C0
>>> 2026-07-26 10:58:56.448 MST [1529364] CONTEXT: writing block 4400857 of relation base/16384/1255337
>>>
>>> Has anyone come across this issue before or know what could cause it?
>>> Any guidance on the root cause or recommended troubleshooting steps would
>>> be greatly appreciated.
>>>
>>> This is a *Major Incident (MI)* and requires immediate assistance.
>>>
>>> Thanks in advance.
>>>
>>
^ permalink raw reply [nested|flat] 5+ messages in thread
end of thread, other threads:[~2026-07-27 11:38 UTC | newest]
Thread overview: 5+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-07-27 05:57 MI - WAL issue mahamood hussain <hussain.ieg@gmail.com>
2026-07-27 08:06 ` Jochen Bandhauer <jb@jbitc.de>
2026-07-27 08:45 ` Gaurav Shukla <gsricky391@gmail.com>
2026-07-27 09:01 ` mahamood hussain <hussain.ieg@gmail.com>
2026-07-27 11:38 ` sayli khedekar <saylikhedekar1@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox