Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1mjlSP-0000nC-S7 for pgsql-admin@arkaria.postgresql.org; Sun, 07 Nov 2021 16:55:50 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1mjlSO-0005wc-Ci for pgsql-admin@arkaria.postgresql.org; Sun, 07 Nov 2021 16:55:48 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <70camero@live.com>) id 1mjlSN-0005wS-Da for pgsql-admin@lists.postgresql.org; Sun, 07 Nov 2021 16:55:48 +0000 Received: from mail-bn8nam11olkn2053.outbound.protection.outlook.com ([40.92.20.53] helo=NAM11-BN8-obe.outbound.protection.outlook.com) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from <70camero@live.com>) id 1mjlSI-0006YB-67 for pgsql-admin@postgresql.org; Sun, 07 Nov 2021 16:55:44 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=Xo4s4v/grvmBcg1GtR4+3/zb1NK3BO9zRQFtuwIVFJtIrJq+Eb+TMFLTa185ZdcK45gjlBGVE0rKLL4cENLeU83TQKZuuAlWUd68l7x/X6YV+hRuZeR3a1/gudxpIQZy5aH7EMd5yiJ8kax3Xs2cF52WGl0rHpIXnE+o9JsHu+11K/6ewrkWWuqibCviuFhFjlsx9aIWsMS9PJySDPXIOzgTOWCwm2ujWdXVSJ3KoM9FwESWhK/T3oc99J6t3KIprCFisNQIs83KXX/so1o42D0XJupWLgFiFi2K0ink7UwDiI3Wx+Ve3PJ7ivGrZOG+psxWNLOn7px+XLH52pYYLw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=3hbZPu04fm4RCqNVSUuZWR3rIoVCVGOzAsp1rEwP4/s=; b=D+3Wto0+mEGR8orrkzO76bc7Tz/3uq074vn7eLcOX7RMF1kghBoOBuIZWbrnDDFNccrQvwUBxAGPRnVeAhYOWerW9LH146GmU+eLyqFpj1ieV/C6hxe7KsAr03rWwqIaPtFJVpN2r2QAXrgD36nN8+81A6VCPSeBvHDnU4sdlxW7EYBZdeB3+sYMWPDQxk1r8+ubtnvmgHsGDhmQ64+PNw3o8g+Z3dxagKllFaImiw4t8sDjxxgiXfDruwg/TgtCsJu/XA/eDNTYDQ569SMPghYGliClvSqf7K+4t5aRxnt1UE177wkBwJg0j7sM+pwBdCFiNO1oV+3jzKATxKPLcg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=none; dmarc=none; dkim=none; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=live.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=3hbZPu04fm4RCqNVSUuZWR3rIoVCVGOzAsp1rEwP4/s=; b=oWTEMlpIHr8Cr1mHI2VjdbTsy1t1WhFIJj+HFIoj2vh7oo4vuwPfAGXN6vL6WgnsO4F4C8UW4OALQvddpcrzCqzFHcpeR/fjy1AXSCQYII9fNua63XLqb5gIyAVyDgsywQjs9EbddFK2wULEkm2ddUr7J6NWtkN6iOugven7eg8wpUmFhCJAf+flIVCp8tE1qhcfgn+fmDUkdPFCQ4Wp88aLn5jDiw2ZxKzqfxHYvgvpWZOXda8WYoSaBuuj+epaPVl/TB20IllniJkUFwkyvt6dH0+kUIFcMsarPXoOsLhiFp5D7iR3r6UjK/XRNeWhHNglfSqbEaj5MYe8IjuTxQ== Received: from BYAPR02MB6438.namprd02.prod.outlook.com (2603:10b6:a03:11d::33) by SJ0PR02MB7325.namprd02.prod.outlook.com (2603:10b6:a03:294::24) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4669.10; Sun, 7 Nov 2021 16:55:27 +0000 Received: from BYAPR02MB6438.namprd02.prod.outlook.com ([fe80::6948:8b09:f03c:b114]) by BYAPR02MB6438.namprd02.prod.outlook.com ([fe80::6948:8b09:f03c:b114%5]) with mapi id 15.20.4669.016; Sun, 7 Nov 2021 16:55:27 +0000 From: Satoshi Nakamoto <70camero@live.com> To: Nikhil Shetty , pgsql-admin Subject: Re: Stuck Checkpoint Process Thread-Topic: Stuck Checkpoint Process Thread-Index: AQHX0JO1hg15Z1ockUum9D+ZNh84GKv0Y2QAgAPrOgQ= Date: Sun, 7 Nov 2021 16:55:27 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-tmn: [WUmbPY196u/rAijVUp+gm8k0KkLvvJKm] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: 6e19caf6-edf4-495f-580e-08d9a20f6697 x-ms-traffictypediagnostic: SJ0PR02MB7325: x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: wT7s2mFzaM/Cjn90jIA3AprBxf0L62coJ2C37loiLkpuHjR58zB6W85KnM1MxyDQiJRPPNMTPg8UvpOajx0n1KobBr2LJFR/PvqNWG4hnON2/V4exkh+x5r1jI9DG62dHwj359sGQPxCa+ZAs2ErE1irIC0XL137MlMRgNNl6TvmzqPda9zgYOM9Gs6/uBRnrSJDVCNDY5ne089y6fWjktOAO3ZDBJ5C2pMQvG9EIMlu6f2bvuD8PY21qOdjGAlFBCzGauEIyhxncGcGoJFtnRNLgpgf+GxYXllliFWBEiEAu5x7fchOihB9rUdYfcApC6uIoNFjUPFDHDu8F3s7lkh8U/Y3dtdxux0GIrbjgmsN7/0x0wiBOPfSqt//24GWP4u+tPedpbApTpxeOFwlhCq2YVX2+87O/de7iTg+1EVnxQIPfwfF4yjeTrjqBsojAVUYrf///0dkaTODuTriBPhxL7eKph7rQfQxiZQJoq6PjmhPwK2WRrqwlaHZUIG5pJ8ZjOSs1jvIBrEw7JWCzsCw7/oMdraBXzOa2N44iG4PQ3VnKLzosbPJkUTQWuiduZZCBkGiu0wEp/bfncr+jQ== x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: Qj1nt/S406UOTZlHkVYxKC1+eA/6kvLEvuXmlkeQbql/pd7P1mLXKC7dNXrPQQIq8ITyQFYO5oxijYg1U2gkheoXvtY7+J08P/CN5bxhE/JYQrdw6XplmO54UOTjuUE2iiGOMq3FFYF8SG/7qP/GsaFN8G2f6aeovBh8pwsANy7l4gThO1MvxyJTT19AXjN7ePAn43mvbixY7voFti8jDwJnhlwFIXrbDT2k5SxTg+/ezG0IUSFSnUoZC7Ngp5SFdqhjIctZl7tGO2bf2cEYer9KAgDFyurObeZNCA0VtDn3v1bDfUFFC/qcds+WCkwf2tzjZ/cDDkETEW+hpdrGSicygWjKdbg+PYnpUVzPf9iCsnoaaA6QQTXNa8dx1WH6gzEWe56ttlfD7FA8lDNACM2XpGRUxO2yq562DOrcuoz3Owwsk7Swu1i1VYsE4F6Bj5Y1kfwewK7C3ujxtDw/X0eZYtWwoVVUwkvaAjf0tDsd2+EgHMckArzYGMilKqpu7rKSUrMcfrxbyFy+TAJf9Fhw97diV04PgJXnhqs8303XzWQIHCbuc2dFiikkCj6GYTK8k2h4/CGlgxGdmdAl5XQH+Upqjm8o0xJkD4PjTlPUb1GyDPOmGOBhHuhXPv6/YHO3SgBpRkF0UymzDXOrAcEGbM6X3XUTzODTjrg2125U9cMoDroItcVnvnnRgeJZWNmoSX7Ugf1Xm4OFPyGWRw== Content-Type: multipart/alternative; boundary="_000_BYAPR02MB6438FD478ECC8954EC61E89BB3909BYAPR02MB6438namp_" MIME-Version: 1.0 X-OriginatorOrg: sct-15-20-3174-20-msonline-outlook-cec7a.templateTenant X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: BYAPR02MB6438.namprd02.prod.outlook.com X-MS-Exchange-CrossTenant-RMS-PersistedConsumerOrg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-CrossTenant-Network-Message-Id: 6e19caf6-edf4-495f-580e-08d9a20f6697 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Nov 2021 16:55:27.8090 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa X-MS-Exchange-CrossTenant-rms-persistedconsumerorg: 00000000-0000-0000-0000-000000000000 X-MS-Exchange-Transport-CrossTenantHeadersStamped: SJ0PR02MB7325 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --_000_BYAPR02MB6438FD478ECC8954EC61E89BB3909BYAPR02MB6438namp_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Restart then upload the fiol an run it then see what you can do you can put= it in a block header an work on it with out crashing an burningthermit a = termitburn is going to bet the ne upgrad to block chain. Get Outlook for Android ________________________________ From: Nikhil Shetty Sent: Thursday, November 4, 2021 10:02:04 PM To: pgsql-admin Subject: Re: Stuck Checkpoint Process Hi Team, Any assist here would be helpful Thanks, Nikhil On Wed, 3 Nov 2021 at 2:47 PM, Nikhil Shetty > wrote: Hi Team, PostgreSQL Version - 11.7 OS - RHEL 7.9 Problem: We see that the backups were not happening on one of the instances. On furt= her checking, we saw that the checkpoint process was not working. Last chec= kpoint was 7 days ago. Manual checkpoint goes into hang state. Checkpoint Process > pid | backend_start | wait_event_type | wait_event= | backend_type > -------+-------------------------------+-----------------+---------------= ------+-------------- > 21237 | 2021-06-29 21:27:51.806746+00 | LWLock | replication_sl= ot_io | checkpointer > (1 row) Time of latest checkpoint: Wed 27 Oct 2021 03:45:09 PM PDT I see it is waiting for I/O on replication slots. I see no issue on standby= and replication is working fine. Need help to further debug this Replication Slots > dbname=3D# select pg_current_wal_lsn(); > pg_current_wal_lsn > -------------------- > 2B0/75042DE0 > dbname=3D# select * from pg_replication_slots ; > slot_name | plugin | slot_type | datoid | database | temporary | act= ive | active_pid | xmin | catalog_xmin | restart_lsn | confirmed_flush_lsn > ---------------+--------+-----------+--------+----------+-----------+----= ----+------------+------+--------------+--------------+--------------------= - > repmgr_slot_5 | | physical | | | f | t = | 4285 | | | 2B0/75042DE0 | > repmgr_slot_1 | | physical | | | f | t = | 3402 | | | 2B0/75042DE0 | Thanks and Regards, Nikhil --_000_BYAPR02MB6438FD478ECC8954EC61E89BB3909BYAPR02MB6438namp_ Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable
Restart then upload the fiol an run it then see what you can do you can put= it in a block header an work on it with out crashing  an burningtherm= it a termitburn is going to bet the ne upgrad to block chain. 

From: Nikhil Shetty <nik= hil.dba04@gmail.com>
Sent: Thursday, November 4, 2021 10:02:04 PM
To: pgsql-admin <pgsql-admin@postgresql.org>
Subject: Re: Stuck Checkpoint Process
 

Hi Team,

Any assist here would be helpful 

Thanks,
Nikhil

On Wed, 3 Nov 2021 at 2:47 PM, Nikh= il Shetty <nikhil.dba04@gmail.= com> wrote:
Hi Team,

PostgreSQL Version - 11.7
OS - RHEL 7.9

Problem:
We see that the backups were not happening on one of the instances. On= further checking, we saw that the checkpoint process was not working. Last= checkpoint was 7 days ago. Manual checkpoint goes into hang state.

Checkpoint Process
>   pid  |         backend_start   &n= bsp;     | wait_event_type |     wait_event   &nbs= p;  | backend_type
> -------+-------------------------------+-----------------+------------= ---------+--------------
>  21237 | 2021-06-29 21:27:51.806746+00 | LWLock     &nb= sp;    | replication_slot_io | checkpointer
> (1 row)

Time of latest checkpoint:            Wed 27 = Oct 2021 03:45:09 PM PDT

I see it is waiting for I/O on replication slots. I see no issue on st= andby and replication is working fine. Need help to further debug this

Replication Slots
> dbname=3D# select pg_current_wal_lsn();
>  pg_current_wal_lsn
> --------------------
>  2B0/75042DE0

> dbname=3D# select * from pg_replication_slots ;
>    slot_name   | plugin | slot_type | datoid | database= | temporary | active | active_pid | xmin | catalog_xmin | restart_lsn &nbs= p;| confirmed_flush_lsn
> ---------------+--------+-----------+--------+----------+-----------+-= -------+------------+------+--------------+--------------+-----------------= ----
>  repmgr_slot_5 |        | physical  | &n= bsp;      |          | f   &nb= sp;     | t      |       4285 | &nb= sp;    |              | 2B0/75= 042DE0 |
>  repmgr_slot_1 |        | physical  | &n= bsp;      |          | f   &nb= sp;     | t      |       3402 | &nb= sp;    |              | 2B0/75= 042DE0 |

Thanks and Regards,
Nikhil 
--_000_BYAPR02MB6438FD478ECC8954EC61E89BB3909BYAPR02MB6438namp_--