Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1udo3j-000ZL5-Ny for pgsql-general@arkaria.postgresql.org; Mon, 21 Jul 2025 10:47:53 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1udo3i-000PFU-SA for pgsql-general@arkaria.postgresql.org; Mon, 21 Jul 2025 10:47:51 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1udo3i-000PEW-2O for pgsql-general@lists.postgresql.org; Mon, 21 Jul 2025 10:47:50 +0000 Received: from mail-westeuropeazon11021108.outbound.protection.outlook.com ([52.101.70.108] helo=AS8PR04CU009.outbound.protection.outlook.com) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1udo3e-0000dc-2i for pgsql-general@lists.postgresql.org; Mon, 21 Jul 2025 10:47:49 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector10001; d=microsoft.com; cv=none; b=WheOj6DvTQZmJJwFxgfdyTBTJKURRU9YMZegrVCGmOXSq/omLWfj1kJ6wi+jsWCiNDFx3A+gHh/ll+0haTAy7AhJ0q6H0OFRp/P65FwIVHadE9G80VgFkbb3QNhX/TFi/HJgbH6yYdCM4717pSqzprdOK099uR2jXEgsBpafx4jOAfL/nPXtAkW+pH4Eh7iQpOyU+mfaSPB4E0xPB9m1K0xYZm91mtlPd/ja73vZGVNsrfLC4IQvqfh8AQdCyvFSSLLMSRMub+3A/Tk6+aRKgjof2aTdmWEQ1dgTZJ8DYqNRF72MjlaFTNLUMqKWVz1uL2qHzKni8Z1++UuBBZL5Gg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector10001; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=/tm3HZUKABEQWCn8JOhe+VfNw97IJj+IDCUSLsCdUvg=; b=d3zkLlxTiwZItby8CC7ThveXSwptyPfplzTtaWLwrcnxH/apP8npCOIWYcZGTmx0YDeDvEHteqCq/KOuz3eiwSYkF9+sVsIK8Q1DLV968PnPiIXUWO20KjiE/GJDC/VUihiAmuaNbHGEwzvIxcm92+wlSH+KNHSIei76RAZvCz2l4FfMzADywRsGI0NwjJyF5vD6gKBYaB9yd4lisX4vw6noiGiBq0F0ncGpFoiVAl/p94YfXtiV9zNHpCWGEiqAYOGOWzrME7CEbP1qwFx9k1XQlmA2FVdhcplSpFzOJv8TaVqIBU43qP9rWJjPYCyfgd0gS+W8+2i6sKgOCo4jnw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=nic.at; dmarc=pass action=none header.from=nic.at; dkim=pass header.d=nic.at; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nic.at; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=/tm3HZUKABEQWCn8JOhe+VfNw97IJj+IDCUSLsCdUvg=; b=odm9vTq+XpSiyeJaPyu8xj1cJyV/xNWCmCVGDsbtdZNYcEivENZVbe781lgOqouUpHes3rmrfIEDlDqtmER1F7kDk+5rZgJHIS+CY6mB2gOPMSQ7hxnfJtl50AtLzF0FOqV0Enyl8nQ7iVhZr9YV2Hj7lD3lgbSSXNYZrG2QoLWqfEY2i7CbQXr/UdiBgeIrNBqKsoLyP6Zt9EyiEfyl4n1nh7hWnD/Y8p7O/rH1CZ9hNQkdxqYAshRsnJloOmIDQvwm3GG4kjQIgZr9ugsJw1Y8R+ENxTE+4ee6MU42RY/DoItiIoPkng7h72lsRhmodx1xcuBr9pBvGx0Ff0KlLQ== Received: from DBAPR03MB6358.eurprd03.prod.outlook.com (2603:10a6:10:192::20) by DB8PR03MB6250.eurprd03.prod.outlook.com (2603:10a6:10:136::8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.8922.32; Mon, 21 Jul 2025 10:47:40 +0000 Received: from DBAPR03MB6358.eurprd03.prod.outlook.com ([fe80::6cf7:babb:d18f:f1a9]) by DBAPR03MB6358.eurprd03.prod.outlook.com ([fe80::6cf7:babb:d18f:f1a9%7]) with mapi id 15.20.8943.029; Mon, 21 Jul 2025 10:47:40 +0000 From: Klaus Darilion To: "pgsql-general@lists.postgresql.org" Subject: Postgresql 16.9 fast shutdown hangs with walsenders eating 100% CPU Thread-Topic: Postgresql 16.9 fast shutdown hangs with walsenders eating 100% CPU Thread-Index: Adv6LAVs27nN85/RQeaGseQkSk9XHg== Date: Mon, 21 Jul 2025 10:47:40 +0000 Message-ID: Accept-Language: de-DE, en-US Content-Language: de-DE X-MS-Has-Attach: yes X-MS-TNEF-Correlator: authentication-results: dkim=none (message not signed) header.d=none;dmarc=none action=none header.from=nic.at; x-ms-publictraffictype: Email x-ms-traffictypediagnostic: DBAPR03MB6358:EE_|DB8PR03MB6250:EE_ x-ms-office365-filtering-correlation-id: 8da893bd-e7ed-4023-3f8f-08ddc84403fc x-ms-exchange-senderadcheck: 1 x-ms-exchange-antispam-relay: 0 x-microsoft-antispam: BCL:0;ARA:13230040|376014|366016|1800799024|13003099007|4053099003|8096899003|38070700018; x-microsoft-antispam-message-info: =?iso-8859-1?Q?Y6faIJZ7M6nIVuVKGUPqPNieTV5te8vvZ3tpUHaNSg4J5XcwUHf8e8KRlr?= =?iso-8859-1?Q?Ybptg4hhGmh6iik9ERaKSTxUHbIGkUIWnJ8Uzt4TdD0C5T0/Y4P2CaXB8V?= =?iso-8859-1?Q?RIhu/lkbePTxoCOxJmy++1n37TBaDdXNi47+QhU7o7xG+dEu5RQW/TYvJ8?= =?iso-8859-1?Q?XeeOrPqS/kltkFvFvJtZO4wCTPQKyGBgJ8HnGe2cwfx00H1crJL0D/enYX?= =?iso-8859-1?Q?UmCuo+j2s4kWzrAwWfxM6gCBNFnYOQCS/gq/hwRI8Ucq1jW8u3w+ulrW3B?= =?iso-8859-1?Q?6t9+ryrC8Ahl90tePuX1VOhFUaSmGSUKsJ3eGAAVT8+HzQkOA+uu8evxod?= =?iso-8859-1?Q?JZkg9GWhne+/JC9P0iacw8Qr5knckc/tdvwI50DX8ndKou6zoKrXuGz9FT?= =?iso-8859-1?Q?0otzLL0Mhk5WtWrnrb7lXyPtKVfBFJz4vn3C857bWX9YB1BQX7i5ovaCvO?= =?iso-8859-1?Q?Bue+zpuHT/cu8Qx/+y45utgtfSX+JadXb39g1SQ6odLmZLTcr/qr8bXFSn?= =?iso-8859-1?Q?tmF1qAGvuSGtxV65azklNKI/36UeB+z7iJQ4DbLabamTt1SjKyXNejhsTZ?= =?iso-8859-1?Q?xLwmr53thI7Gw+j5uuvip/0zDcMiNel0GfwH0Lfa2zKL6YimBYWA6VOc0k?= =?iso-8859-1?Q?VGE4aw85U3zqM+s3hezgtqmhGS2gUZ5RMlMMPL1b1wf69OgK2lR7iO1aP8?= =?iso-8859-1?Q?f8+XmYk3++hgp9Aeg+/8agFQYNmeTEOkhrkdgpWU23nMzUY1IQzD7LD1BD?= =?iso-8859-1?Q?SzkPTkwuw+8/6U56u2OTsijetcQhS7Rux9VNBUTwbl/WYUiWmDV9aN7/14?= =?iso-8859-1?Q?2x1C/b27Y5AlOu+aah2y/7Rsf0+fZDqQlDa9czntnHkXDEcZ5HX2AbL555?= =?iso-8859-1?Q?3XeZsCaivmjBqOXjlRDUEMpDMZbd6z8N7hjZ/L877OSMnJCbikFPpOtoP+?= =?iso-8859-1?Q?9KkjJsD4r9hXF4g5l9wJxxfqEM6GSpRSowLo5WfY2WZw4LQl3XWvdv2bWN?= =?iso-8859-1?Q?qLn4zBkROPMoCN1303NJPSPxdnxV53keOcQjj3+ZJzDxgKkjmd0nqwuNlS?= =?iso-8859-1?Q?xLHZPND4Y17hBC0MFB9097MpW+cKGXl0vem4GXsgNOfWO2AEP96pbSM2+Z?= =?iso-8859-1?Q?wZlqpOqXq/8rv7cPYhcO5ZI4lFz6UaSRe6MFZ+NFoiTLW1H42v+6NR/zjs?= =?iso-8859-1?Q?+2NrRbqVjq1MjQ+yumLhKValtdLd/znMXcxjb8GfJNb1hpIuUM5nd64Igf?= =?iso-8859-1?Q?6h3CBa0TVepBYEnqDcomZMgyZv5gf89ypAHBquFz5IQ1fGS+yh57zaxQA9?= =?iso-8859-1?Q?d9FlR7M+32TTK4JyGQDB1/tI9u4hg6rNd/NWp0FmVJXFg82+evnxtaALiC?= =?iso-8859-1?Q?Fq//TAFBTxsOhxw9+NbArFYAnN15DfaMo3lusINOfY/zImEPQhQZ2pDI0B?= =?iso-8859-1?Q?1VEfyRrDiWQknKf/KcHZNkJt80Q09IA9nwYHA+Mwv6DqAdmhJIhFU+B29P?= =?iso-8859-1?Q?A=3D?= x-forefront-antispam-report: CIP:255.255.255.255;CTRY:;LANG:en;SCL:1;SRV:;IPV:NLI;SFV:NSPM;H:DBAPR03MB6358.eurprd03.prod.outlook.com;PTR:;CAT:NONE;SFS:(13230040)(376014)(366016)(1800799024)(13003099007)(4053099003)(8096899003)(38070700018);DIR:OUT;SFP:1102; x-ms-exchange-antispam-messagedata-chunkcount: 1 x-ms-exchange-antispam-messagedata-0: =?iso-8859-1?Q?w4dA9iIgRKPt3negfuWOh1KjeHVFljBeOiqwVPyK4ozwmj5N7h2llqw9Z/?= =?iso-8859-1?Q?Hc7pAT0HEQsTx9PByuK4aaiCCrnt9ogKSK+Tox93FUDlHqQu+iDkz4p6jL?= =?iso-8859-1?Q?kzDUUHq7Pz6WB/KVFV4VlNJPjQNLhbVsLQnbf4hBwoMzttds3d7wu1aieo?= =?iso-8859-1?Q?yMzAQVcH32Tx4JFkYcgeIEnWTCCt/76+3OQSYVviPqB2NAAEAiZoc0a6N2?= =?iso-8859-1?Q?6Wno4eau0Z3qLKY2iGq+tyHjfxIu4x3Y2Bn7Zc9VHajgyz4kJ1wQ901xzW?= =?iso-8859-1?Q?XxRGouWHfIo5p0rcWswcs62fVDigSfx0ZAxtOybP+eWXnsd9Z3rcgqVVYr?= =?iso-8859-1?Q?+kyFItvY99tzOI6i37Lwh5AN55YL7Xss3W39XMyfXJW4u2EPzErGpwqGoQ?= =?iso-8859-1?Q?JjLA3zamh/3M4/BeyVTSorpmnD1zmJPwjeJcMeejtdZOMmxeSLZthEjzJI?= =?iso-8859-1?Q?lczOMc4AYcZdQi2v8eMqP0KxrbLb0Sjc0iaSuNhbD3I7E6KQKQrz8Zp6Qy?= =?iso-8859-1?Q?r08GLly+JPnWSHeX7jN0In/basvMMJuCYd9gfBIvp3QHgC+O2iEpQZSKIr?= =?iso-8859-1?Q?hpH21DEquQgEg17STPW+ex4q2XY+FWARgkqV05eagWv2++9J03VzXHAc5x?= =?iso-8859-1?Q?/xOHx8T6VNEXtSK8BOncb3Hunm/xbQtuNZPODrfbAPNHRVD937oR+jlp+J?= =?iso-8859-1?Q?l2HLirRAoryA2UFSPjZ1KZ6fWkjpgntpbCiFZHmVLzyDjnS/NVIm0vgUsb?= =?iso-8859-1?Q?ptidkYUz3qjRy7BEEQu4MbFHLcHIJ9HgvzyOg0ZrOtmzs97NipbCNxZmzh?= =?iso-8859-1?Q?hBdJC6c5Xw7b5PPwV/QcSEdbDjSQFj8dzufxlkgekXhzhFAwHqTa/eIq8/?= =?iso-8859-1?Q?F3xMym1nlfLKH5ZAO46ujgb5aaZo0rpWnMcOyue7Aq1kbvTLm3mcJ0jAPO?= =?iso-8859-1?Q?Qz6nhVBW35xZJZcsLHKPpRZEd8+6PPHCte9v1rQQSeW7yDQHrgLwm2FObI?= =?iso-8859-1?Q?1LIrIQZX4Qp7lSvObPTW3VMuXjxhzTPXn1dzRXpUAyIhINIPbdReYVdlxl?= =?iso-8859-1?Q?RmdhPWE3SOBlJbIIsGCNTpgY0/KYmQW5TeBH81tFfrOt0XN6okVl55YbdP?= =?iso-8859-1?Q?r5Qs0gqHqlWPz5UxCcMBLYs7w2Jol8b9lRPiI2BrvftkeyHCu/8lTEyWGc?= =?iso-8859-1?Q?5ULjM64eEzs5MjaywhHXfQ/TNnisNlh7QjyAWRtida0tYJggIkRxyZ+6kd?= =?iso-8859-1?Q?ickBv3IdTWTj9d9QTtPyeOCISq3PDVnym3RmON/MsOQElB2DMWB7hF1VWH?= =?iso-8859-1?Q?UdFdkrk0/Q72fmnZMSM6uWwEatPI8FwSXTwt8U9e7HjFAFxwAI+eGOTmVL?= =?iso-8859-1?Q?QBIHlkg0fSe9TAc6k+5Bpa5MEyD8Py7xgEy8xjoVpHhQt/vkxlrKsH76Wb?= =?iso-8859-1?Q?Or8jJxFRotgNQ+22RBDKRPza8SF/x+ohceMBfIvgJ89ZNb0APxkXM3P3Ue?= =?iso-8859-1?Q?6tnQ9rDS26fxU1enCKborVirdUC6mAkJGx4ubtJ61YlAfLAoC/Nb5KS9Dr?= =?iso-8859-1?Q?5V0mnTsEEW5sbJDL4ta3BMQ6lh78herc4Q6bElNwdTeS7UYDDSYH54YaOu?= =?iso-8859-1?Q?pV+Nq2AAzlZG5rkkAC1t+mGYJsYeyw7w8l?= Content-Type: multipart/mixed; boundary="_004_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_" MIME-Version: 1.0 X-OriginatorOrg: nic.at X-MS-Exchange-CrossTenant-AuthAs: Internal X-MS-Exchange-CrossTenant-AuthSource: DBAPR03MB6358.eurprd03.prod.outlook.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8da893bd-e7ed-4023-3f8f-08ddc84403fc X-MS-Exchange-CrossTenant-originalarrivaltime: 21 Jul 2025 10:47:40.6240 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 6da68bb1-2470-49a3-8e7a-a343d51e9234 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: OsoWHrP+RSPZ1MGdgdfUyi7XNY/qYTK832tr5bqPjhy/BUdCaloAyQvPasJHYseGDEiphmCl1vj5r6AH88sp/Q== X-MS-Exchange-Transport-CrossTenantHeadersStamped: DB8PR03MB6250 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --_004_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_ Content-Type: multipart/alternative; boundary="_000_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_" --_000_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_ Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable (Note: I have also attached the whole email for better readability of the l= ogs) Hello! Our setup: 5 Node Patroni Cluster with PostgreSQL 16.9. db1: current leader db2: sync-replica db3/4/5: replica The replicas connect to the leader using the host IP of the leader. So ther= e are 4 walsender for patroni, 1 sync and 3 async. The patroni cluster utilizes a service IP-address (VIP). The VIP is used by= all clients connecting to the current leader. These clients are: - some web-apps doing normal DB queries (read/write) - 2 barman backup clients using streaming replication - 58 logical replication clients Additionally we use https://github.com/EnterpriseDB/pg_failover_slots to sy= nc and advance the logical replication slots on the replicas. The failover_= slots plugin periodically connects to leader using the VIP. We had a planned maintenance and wanted to switch the leader from db1 to db= 2: 12:32:18: patronictl switchover --leader db1 --candidate db2 So postmaster received the fast shutdown request from Patroni and started s= hutting down the client connection processes: 12:32:42 db1 patroni[2748]: patroni_postgres: 2025-07-16 12:32:42.224 UTC 2= 748 LOG: received fast shutdown request 12:32:42 db1 patroni[2748]: patroni_postgres: 2025-07-16 12:32:42.224 UTC 2= 748 LOG: aborting any active transactions 12:32:42 db1 patroni[1842780]: patroni_postgres: 2025-07-16 12:32:42.224 UT= C 1842780 FATAL: terminating connection due to administrator command 12:32:42 db1 patroni[1842702]: patroni_postgres: 2025-07-16 12:32:42.224 UT= C 1842702 FATAL: terminating connection due to administrator command ... 12:32:42 db1 patroni[2748]: patroni_postgres: 2025-07-16 12:32:42.555 UTC 2= 748 LOG: background worker "logical replication launcher" (PID 99854) exit= ed with exit code 1 12:32:42 db1 patroni[3935]: patroni_postgres: 2025-07-16 12:32:42.573 UTC 3= 935 FATAL: terminating background worker "pg_failover_slots worker" due to= administrator command 12:32:42 db1 patroni[2748]: patroni_postgres: 2025-07-16 12:32:42.626 UTC 2= 748 LOG: background worker "pg_failover_slots worker" (PID 3935) exited wi= th exit code 1 At that time, all the normal clients were disconnected and we only see in t= he logs new connection attempts that were refused, eg: 12:32:42 db1 patroni[1846807]: patroni_postgres: 2025-07-16 12:32:42.796 UT= C 1846807 LOG: connection received: host=3Dxxxx:xxx:8::13 port=3D53704 12:32:42 db1 patroni[1846807]: patroni_postgres: 2025-07-16 12:32:42.800 UT= C 1846807 FATAL: the database system is shutting down 12:32:43 db1 patroni[2988]: patroni_postgres: 2025-07-16 12:32:43.259 UTC 2= 988 LOG: shutting down We might have missed some logs that might be real patroni or postgres logs = :-( 12:33:39 db1 systemd-journald[1613]: Suppressed 10375 messages from patroni= .service Usually the switchover only takes a few seconds. After waiting a few minute= s we became anxious and started debugging. Using "ps -Alf|grep postgres" we saw that there were no more normal client = connections, but still 58 logical replicaton walsender processes and 6 stre= aming replication walsenders. "top" revealed that the walsenders were eating CPU. I know for sure that I = saw plenty logical replication walsenders that were eating 100% CPU, but I = have not checked if also the streaming replication walsenders were eating C= PU. I also straced the checkpointer process, and that was doing no I/O, just lo= oping or polling (I can't remember the details). Our Performance Monitoring= showed a general drop of I/O activity during the shutdown. So this sounds somehow like the walsenders were stuck in a state that in th= e postgres mailing lists is called "busy loop". Then we decided to remove the VIP (which is used by the logical replication= clients) from db1 to trigger replication errors: systemctl stop vip-manager.service 12:50:11 db1 commandlog[2276907]: root/root : ip a d x.x.x.x/24 dev bond0.2= 03 12:50:11 db1 commandlog[2276907]: root/root : ip -6 a d xxxx:xxx:x:x::10/64= dev bond0.203 Then, either caused by that or by something else, the connections timed out= . We are using the default timeouts: wal_sender_timeout: 60s Frankly, the timeouts started not 60s after removing the VIP, but already a= fter 45 seconds. But I guess the timeout triggered a bit eralier as there w= as no more traffic on the walsender connections except for the status keepa= live from the clients. 12:50:55 db1 patroni[2586700]: patroni_postgres: 2025-07-16 12:50:55.977 UT= C 2586700 LOG: terminating walsender process due to replication timeout 12:50:55 db1 patroni[2586700]: patroni_postgres: 2025-07-16 12:50:55.977 UT= C 2586700 STATEMENT: START_REPLICATION SLOT "barman_sbg" 7DFD/5F000000 TIM= ELINE 22 12:50:56 db1 patroni[2579309]: patroni_postgres: 2025-07-16 12:50:56.027 UT= C 2579309 LOG: terminating walsender process due to replication timeout 12:50:56 db1 patroni[2579309]: patroni_postgres: 2025-07-16 12:50:56.027 UT= C 2579309 STATEMENT: START_REPLICATION SLOT "reg_tam1" LOGICAL 75D4/F7FDBB= 8 (proto_version '4', origin 'any', publication_names '"mypublications"') 12:50:56 db1 patroni[827455]: patroni_postgres: 2025-07-16 12:50:56.212 UTC= 827455 LOG: terminating walsender process due to replication timeout 12:50:56 db1 patroni[827455]: patroni_postgres: 2025-07-16 12:50:56.212 UTC= 827455 STATEMENT: START_REPLICATION SLOT "reg_lis1" LOGICAL 7C86/593393A0= (proto_version '4', origin 'any', publication_names '"mypublications"') 12:50:56 db1 patroni[3390803]: patroni_postgres: 2025-07-16 12:50:56.506 UT= C 3390803 LOG: terminating walsender process due to replication timeout 12:50:56 db1 patroni[3390803]: patroni_postgres: 2025-07-16 12:50:56.506 UT= C 3390803 STATEMENT: START_REPLICATION SLOT "reg_rey1" LOGICAL 77A5/205F5F= 0 (proto_version '4', origin 'any', publication_names '"mypublications"') 12:50:56 db1 patroni[137758]: patroni_postgres: 2025-07-16 12:50:56.672 UTC= 137758 LOG: terminating walsender process due to replication timeout 12:50:56 db1 patroni[137758]: patroni_postgres: 2025-07-16 12:50:56.672 UTC= 137758 STATEMENT: START_REPLICATION SLOT "reg_tlv1" LOGICAL 7DC2/CEAB27A0= (proto_version '4', origin 'any', publication_names '"mypublications"') 12:50:56 db1 patroni[2586624]: patroni_postgres: 2025-07-16 12:50:56.929 UT= C 2586624 LOG: terminating walsender process due to replication timeout 12:50:56 db1 patroni[2586624]: patroni_postgres: 2025-07-16 12:50:56.929 UT= C 2586624 STATEMENT: START_REPLICATION SLOT "barman" 7DFD/60000000 TIMELIN= E 22 ...and additional 54 logical walsender that timed out and were terminated 12:51:05 db1 patroni[2988]: patroni_postgres: 2025-07-16 12:51:05.902 UTC 2= 988 LOG: checkpoint starting: shutdown immediate 12:51:05 db1 patroni[2988]: patroni_postgres: 2025-07-16 12:51:05.926 UTC 2= 988 LOG: checkpoint complete: wrote 11 buffers (0.0%); 0 WAL file(s) added= , 0 removed, 0 recycled; write=3D0.021 s, sync=3D0.001 s, total=3D0.026 s; = sync files=3D9, longest=3D0.001 s, average=3D0.001 s; distance=3D55 kB, est= imate=3D1080354 kB; lsn=3D7E53/18554B28, redo lsn=3D7E53/18554B28 12:51:07 db1 patroni[2748]: patroni_postgres: 2025-07-16 12:51:07.159 UTC 2= 748 LOG: database system is shut down 12:51:07 db1 patroni[2389]: 2025-07-16 12:51:07,954 INFO: Got response from= db2 https://db2:8008/patroni: {"state": "running", "postmaster_start_time"= : "2025-07-16 12:17:25.758501+00:00", "role": "replica", "server_version": = 160009, "xlog": {"received_location": 138895355628448, "replayed_location":= 138895355628448, "replayed_timestamp": "2025-07-16 12:32:42.142954+00:00",= "paused": false}, "sync_standby": true, "timeline": 22, "dcs_last_seen": 1= 752670265, "database_system_identifier": "7386557262250076198", "patroni": = {"version": "4.0.5", "scope": "regdns_patroni_cluster_1", "name": "db2"}} After closing all walsenders (that used the VIP) due to timeout (58 logical= senders and 2 streaming senders to barman), the database was shut down and= the switchover happened. I do not know for sure, but I think the walsenders to the patroni replicas,= that were using the host IP and not the VIP, were still active wenn we rem= oved the VIP. But it seems they have shut down nicely after all the other w= alsenders terminated due to timeout. So what caused the hanging? We found bug https://www.postgresql.org/message= -id/aD0Xvh4Tj_Vk7pCA%40paquier.xyz but it seems that only affected logical = walsender in standby mode. We can not reproduce it - after a few minutes we made another switchover ba= ck to db1 and this time it was done within a few seconds. Do you have any ideas what could be the problem, how we could solve it and = how to debug if it happens again? Thanks Klaus -- Klaus Darilion, Head of Operations nic.at GmbH, Jakob-Haringer-Stra=DFe 8/V 5020 Salzburg, Austria --_000_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_ Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable

(Note: I have also = attached the whole email for better readability of the logs)

 

Hello!

 

Our setup: 5 Node P= atroni Cluster with PostgreSQL 16.9.

db1: current leader=

db2: sync-replica

db3/4/5: replica

 

The replicas connec= t to the leader using the host IP of the leader. So there are 4 walsender f= or patroni, 1 sync and 3 async.

 

The patroni cluster= utilizes a service IP-address (VIP). The VIP is used by all clients connec= ting to the current leader. These clients are:

- some web-apps doi= ng normal DB queries (read/write)

- 2 barman backup c= lients using streaming replication

- 58 logical replic= ation clients

 

Additionally we use= https://github.com/EnterpriseDB/pg_failover_slots to sync and advance t= he logical replication slots on the replicas. The failover_slots plugin per= iodically connects to leader using the VIP.

 

 

We had a planned ma= intenance and wanted to switch the leader from db1 to db2:

12:32:18: patronict= l switchover --leader db1 --candidate db2

 

So postmaster recei= ved the fast shutdown request from Patroni and started shutting down the cl= ient connection processes:

12:32:42 db1 patron= i[2748]: patroni_postgres: 2025-07-16 12:32:42.224 UTC 2748 LOG:  rece= ived fast shutdown request

12:32:42 db1 patron= i[2748]: patroni_postgres: 2025-07-16 12:32:42.224 UTC 2748 LOG:  abor= ting any active transactions

12:32:42 db1 patron= i[1842780]: patroni_postgres: 2025-07-16 12:32:42.224 UTC 1842780 FATAL:&nb= sp; terminating connection due to administrator command

12:32:42 db1 patron= i[1842702]: patroni_postgres: 2025-07-16 12:32:42.224 UTC 1842702 FATAL:&nb= sp; terminating connection due to administrator command

...

12:32:42 db1 patron= i[2748]: patroni_postgres: 2025-07-16 12:32:42.555 UTC 2748 LOG:  back= ground worker "logical replication launcher" (PID 99854) exited w= ith exit code 1

12:32:42 db1 patron= i[3935]: patroni_postgres: 2025-07-16 12:32:42.573 UTC 3935 FATAL:  te= rminating background worker "pg_failover_slots worker" due to adm= inistrator command

12:32:42 db1 patron= i[2748]: patroni_postgres: 2025-07-16 12:32:42.626 UTC 2748 LOG:  back= ground worker "pg_failover_slots worker" (PID 3935) exited with e= xit code 1

 

At that time, all t= he normal clients were disconnected and we only see in the logs new connect= ion attempts that were refused, eg:

12:32:42 db1 patron= i[1846807]: patroni_postgres: 2025-07-16 12:32:42.796 UTC 1846807 LOG: = ; connection received: host=3Dxxxx:xxx:8::13 port=3D53704=

12:32:42 db1 patron= i[1846807]: patroni_postgres: 2025-07-16 12:32:42.800 UTC 1846807 FATAL:&nb= sp; the database system is shutting down

 

12:32:43 db1 patron= i[2988]: patroni_postgres: 2025-07-16 12:32:43.259 UTC 2988 LOG:  shut= ting down

 

 

We might have misse= d some logs that might be real patroni or postgres logs :-(

12:33:39 db1 system= d-journald[1613]: Suppressed 10375 messages from patroni.service=

 

 

 

Usually the switcho= ver only takes a few seconds. After waiting a few minutes we became anxious= and started debugging.

 

 

Using "ps -Alf= |grep postgres" we saw that there were no more normal client connectio= ns, but still 58 logical replicaton walsender processes and 6 streaming rep= lication walsenders.

"top" rev= ealed that the walsenders were eating CPU. I know for sure that I saw plent= y logical replication walsenders that were eating 100% CPU, but I have not = checked if also the streaming replication walsenders were eating CPU.

I also straced the = checkpointer process, and that was doing no I/O, just looping or polling (I= can't remember the details). Our Performance Monitoring showed a general d= rop of I/O activity during the shutdown.

 

So this sounds some= how like the walsenders were stuck in a state that in the postgres mailing = lists is called "busy loop".

 

Then we decided to = remove the VIP (which is used by the logical replication clients) from db1 = to trigger replication errors:

systemctl stop vip-= manager.service

12:50:11 db1 comman= dlog[2276907]: root/root : ip a d x.x.x.x/24 dev bond0.203

12:50:11 db1 comman= dlog[2276907]: root/root : ip -6 a d xxxx:xxx:x:x::10/64 dev bond0.203=

 

Then, either caused= by that or by something else, the connections timed out. We are using the = default timeouts:

wal_sender_timeout:= 60s

 

Frankly, the timeou= ts started not 60s after removing the VIP, but already after 45 seconds. Bu= t I guess the timeout triggered a bit eralier as there was no more traffic = on the walsender connections except for the status keepalive from the clients.

 

12:50:55 db1 patron= i[2586700]: patroni_postgres: 2025-07-16 12:50:55.977 UTC 2586700 LOG: = ; terminating walsender process due to replication timeout

12:50:55 db1 patron= i[2586700]: patroni_postgres: 2025-07-16 12:50:55.977 UTC 2586700 STATEMENT= :  START_REPLICATION SLOT "barman_sbg" 7DFD/5F000000 TIMELIN= E 22

12:50:56 db1 patron= i[2579309]: patroni_postgres: 2025-07-16 12:50:56.027 UTC 2579309 LOG: = ; terminating walsender process due to replication timeout

12:50:56 db1 patron= i[2579309]: patroni_postgres: 2025-07-16 12:50:56.027 UTC 2579309 STATEMENT= :  START_REPLICATION SLOT "reg_tam1" LOGICAL 75D4/F7FDBB8 (p= roto_version '4', origin 'any', publication_names '"mypublications&quo= t;')

12:50:56 db1 patron= i[827455]: patroni_postgres: 2025-07-16 12:50:56.212 UTC 827455 LOG:  = terminating walsender process due to replication timeout<= /p>

12:50:56 db1 patron= i[827455]: patroni_postgres: 2025-07-16 12:50:56.212 UTC 827455 STATEMENT:&= nbsp; START_REPLICATION SLOT "reg_lis1" LOGICAL 7C86/593393A0 (pr= oto_version '4', origin 'any', publication_names '"mypublications"= ;')

12:50:56 db1 patron= i[3390803]: patroni_postgres: 2025-07-16 12:50:56.506 UTC 3390803 LOG: = ; terminating walsender process due to replication timeout

12:50:56 db1 patron= i[3390803]: patroni_postgres: 2025-07-16 12:50:56.506 UTC 3390803 STATEMENT= :  START_REPLICATION SLOT "reg_rey1" LOGICAL 77A5/205F5F0 (p= roto_version '4', origin 'any', publication_names '"mypublications&quo= t;')

12:50:56 db1 patron= i[137758]: patroni_postgres: 2025-07-16 12:50:56.672 UTC 137758 LOG:  = terminating walsender process due to replication timeout<= /p>

12:50:56 db1 patron= i[137758]: patroni_postgres: 2025-07-16 12:50:56.672 UTC 137758 STATEMENT:&= nbsp; START_REPLICATION SLOT "reg_tlv1" LOGICAL 7DC2/CEAB27A0 (pr= oto_version '4', origin 'any', publication_names '"mypublications"= ;')

12:50:56 db1 patron= i[2586624]: patroni_postgres: 2025-07-16 12:50:56.929 UTC 2586624 LOG: = ; terminating walsender process due to replication timeout

12:50:56 db1 patron= i[2586624]: patroni_postgres: 2025-07-16 12:50:56.929 UTC 2586624 STATEMENT= :  START_REPLICATION SLOT "barman" 7DFD/60000000 TIMELINE 22=

 

...and additional 5= 4 logical walsender that timed out and were terminated

 

12:51:05 db1 patron= i[2988]: patroni_postgres: 2025-07-16 12:51:05.902 UTC 2988 LOG:  chec= kpoint starting: shutdown immediate

12:51:05 db1 patron= i[2988]: patroni_postgres: 2025-07-16 12:51:05.926 UTC 2988 LOG:  chec= kpoint complete: wrote 11 buffers (0.0%); 0 WAL file(s) added, 0 removed, 0= recycled; write=3D0.021 s, sync=3D0.001 s, total=3D0.026 s; sync files=3D9, longest=3D0.001 s, average=3D0.001 s; distance=3D55 kB,= estimate=3D1080354 kB; lsn=3D7E53/18554B28, redo lsn=3D7E53/18554B28<= /o:p>

12:51:07 db1 patron= i[2748]: patroni_postgres: 2025-07-16 12:51:07.159 UTC 2748 LOG:  data= base system is shut down

12:51:07 db1 patron= i[2389]: 2025-07-16 12:51:07,954 INFO: Got response from db2 https://db2:8008/patroni: {"s= tate": "running", "postmaster_start_time": "2= 025-07-16 12:17:25.758501+00:00", "role": "replica"= ;, "server_version": 160009, "xlog": {"received_lo= cation": 138895355628448, "replayed_location": 138895355628448, "replayed_timestamp": "2025-07-16 12:32:42= .142954+00:00", "paused": false}, "sync_standby": = true, "timeline": 22, "dcs_last_seen": 1752670265, &quo= t;database_system_identifier": "7386557262250076198", "= patroni": {"version": "4.0.5", "scope": = "regdns_patroni_cluster_1", "name": "db2"}}

 

 

After closing all w= alsenders (that used the VIP) due to timeout (58 logical senders and 2 stre= aming senders to barman), the database was shut down and the switchover hap= pened.

 

I do not know for s= ure, but I think the walsenders to the patroni replicas, that were using th= e host IP and not the VIP, were still active wenn we removed the VIP. But i= t seems they have shut down nicely after all the other walsenders terminated due to timeout.

 

So what caused the = hanging? We found bug https://www.postgresql.org/message-id/aD0Xvh4Tj_Vk7pCA%40paquier.xyz<= /a> but it seems that only affected logical walsender in standby mode.=

 

We can not reproduc= e it - after a few minutes we made another switchover back to db1 and this = time it was done within a few seconds.

 

Do you have any ide= as what could be the problem, how we could solve it and how to debug if it = happens again?

 

Thanks

Klaus

 

= --

= Klaus Darilion, Head of Operations

nic.at GmbH, Jakob-Haringer-Stra=DFe 8/V

5020 Salzburg, Austria

 

--_000_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_-- --_004_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_ Content-Type: text/plain; name="postgresql-shutdown-hangs.txt" Content-Description: postgresql-shutdown-hangs.txt Content-Disposition: attachment; filename="postgresql-shutdown-hangs.txt"; size=8694; creation-date="Mon, 21 Jul 2025 10:47:20 GMT"; modification-date="Mon, 21 Jul 2025 10:47:40 GMT" Content-Transfer-Encoding: base64 KE5vdGU6IEkgaGF2ZSBhbHNvIGF0dGFjaGVkIHRoZSB3aG9sZSBlbWFpbCBmb3IgYmV0dGVyIHJl YWRhYmlsaXR5IG9mIHRoZSBsb2dzKQ0KDQpIZWxsbyENCg0KT3VyIHNldHVwOiA1IE5vZGUgUGF0 cm9uaSBDbHVzdGVyIHdpdGggUG9zdGdyZVNRTCAxNi45Lg0KZGIxOiBjdXJyZW50IGxlYWRlcg0K ZGIyOiBzeW5jLXJlcGxpY2ENCmRiMy80LzU6IHJlcGxpY2ENCg0KVGhlIHJlcGxpY2FzIGNvbm5l Y3QgdG8gdGhlIGxlYWRlciB1c2luZyB0aGUgaG9zdCBJUCBvZiB0aGUgbGVhZGVyLiBTbyB0aGVy ZSBhcmUgNCB3YWxzZW5kZXIgZm9yIHBhdHJvbmksIDEgc3luYyBhbmQgMyBhc3luYy4NCg0KVGhl IHBhdHJvbmkgY2x1c3RlciB1dGlsaXplcyBhIHNlcnZpY2UgSVAtYWRkcmVzcyAoVklQKS4gVGhl IFZJUCBpcyB1c2VkIGJ5IGFsbCBjbGllbnRzIGNvbm5lY3RpbmcgdG8gdGhlIGN1cnJlbnQgbGVh ZGVyLiBUaGVzZSBjbGllbnRzIGFyZToNCi0gc29tZSB3ZWItYXBwcyBkb2luZyBub3JtYWwgREIg cXVlcmllcyAocmVhZC93cml0ZSkNCi0gMiBiYXJtYW4gYmFja3VwIGNsaWVudHMgdXNpbmcgc3Ry ZWFtaW5nIHJlcGxpY2F0aW9uDQotIDU4IGxvZ2ljYWwgcmVwbGljYXRpb24gY2xpZW50cw0KDQpB ZGRpdGlvbmFsbHkgd2UgdXNlIGh0dHBzOi8vZ2l0aHViLmNvbS9FbnRlcnByaXNlREIvcGdfZmFp bG92ZXJfc2xvdHMgdG8gc3luYyBhbmQgYWR2YW5jZSB0aGUgbG9naWNhbCByZXBsaWNhdGlvbiBz bG90cyBvbiB0aGUgcmVwbGljYXMuIFRoZSBmYWlsb3Zlcl9zbG90cyBwbHVnaW4gcGVyaW9kaWNh bGx5IGNvbm5lY3RzIHRvIGxlYWRlciB1c2luZyB0aGUgVklQLg0KDQoNCldlIGhhZCBhIHBsYW5u ZWQgbWFpbnRlbmFuY2UgYW5kIHdhbnRlZCB0byBzd2l0Y2ggdGhlIGxlYWRlciBmcm9tIGRiMSB0 byBkYjI6DQoxMjozMjoxODogcGF0cm9uaWN0bCBzd2l0Y2hvdmVyIC0tbGVhZGVyIGRiMSAtLWNh bmRpZGF0ZSBkYjINCg0KU28gcG9zdG1hc3RlciByZWNlaXZlZCB0aGUgZmFzdCBzaHV0ZG93biBy ZXF1ZXN0IGZyb20gUGF0cm9uaSBhbmQgc3RhcnRlZCBzaHV0dGluZyBkb3duIHRoZSBjbGllbnQg Y29ubmVjdGlvbiBwcm9jZXNzZXM6DQoxMjozMjo0MiBkYjEgcGF0cm9uaVsyNzQ4XTogcGF0cm9u aV9wb3N0Z3JlczogMjAyNS0wNy0xNiAxMjozMjo0Mi4yMjQgVVRDIDI3NDggTE9HOiAgcmVjZWl2 ZWQgZmFzdCBzaHV0ZG93biByZXF1ZXN0DQoxMjozMjo0MiBkYjEgcGF0cm9uaVsyNzQ4XTogcGF0 cm9uaV9wb3N0Z3JlczogMjAyNS0wNy0xNiAxMjozMjo0Mi4yMjQgVVRDIDI3NDggTE9HOiAgYWJv cnRpbmcgYW55IGFjdGl2ZSB0cmFuc2FjdGlvbnMNCjEyOjMyOjQyIGRiMSBwYXRyb25pWzE4NDI3 ODBdOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjMyOjQyLjIyNCBVVEMgMTg0Mjc4 MCBGQVRBTDogIHRlcm1pbmF0aW5nIGNvbm5lY3Rpb24gZHVlIHRvIGFkbWluaXN0cmF0b3IgY29t bWFuZA0KMTI6MzI6NDIgZGIxIHBhdHJvbmlbMTg0MjcwMl06IHBhdHJvbmlfcG9zdGdyZXM6IDIw MjUtMDctMTYgMTI6MzI6NDIuMjI0IFVUQyAxODQyNzAyIEZBVEFMOiAgdGVybWluYXRpbmcgY29u bmVjdGlvbiBkdWUgdG8gYWRtaW5pc3RyYXRvciBjb21tYW5kDQouLi4NCjEyOjMyOjQyIGRiMSBw YXRyb25pWzI3NDhdOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjMyOjQyLjU1NSBV VEMgMjc0OCBMT0c6ICBiYWNrZ3JvdW5kIHdvcmtlciAibG9naWNhbCByZXBsaWNhdGlvbiBsYXVu Y2hlciIgKFBJRCA5OTg1NCkgZXhpdGVkIHdpdGggZXhpdCBjb2RlIDENCjEyOjMyOjQyIGRiMSBw YXRyb25pWzM5MzVdOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjMyOjQyLjU3MyBV VEMgMzkzNSBGQVRBTDogIHRlcm1pbmF0aW5nIGJhY2tncm91bmQgd29ya2VyICJwZ19mYWlsb3Zl cl9zbG90cyB3b3JrZXIiIGR1ZSB0byBhZG1pbmlzdHJhdG9yIGNvbW1hbmQNCjEyOjMyOjQyIGRi MSBwYXRyb25pWzI3NDhdOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjMyOjQyLjYy NiBVVEMgMjc0OCBMT0c6ICBiYWNrZ3JvdW5kIHdvcmtlciAicGdfZmFpbG92ZXJfc2xvdHMgd29y a2VyIiAoUElEIDM5MzUpIGV4aXRlZCB3aXRoIGV4aXQgY29kZSAxDQoNCkF0IHRoYXQgdGltZSwg YWxsIHRoZSBub3JtYWwgY2xpZW50cyB3ZXJlIGRpc2Nvbm5lY3RlZCBhbmQgd2Ugb25seSBzZWUg aW4gdGhlIGxvZ3MgbmV3IGNvbm5lY3Rpb24gYXR0ZW1wdHMgdGhhdCB3ZXJlIHJlZnVzZWQsIGVn Og0KMTI6MzI6NDIgZGIxIHBhdHJvbmlbMTg0NjgwN106IHBhdHJvbmlfcG9zdGdyZXM6IDIwMjUt MDctMTYgMTI6MzI6NDIuNzk2IFVUQyAxODQ2ODA3IExPRzogIGNvbm5lY3Rpb24gcmVjZWl2ZWQ6 IGhvc3Q9eHh4eDp4eHg6ODo6MTMgcG9ydD01MzcwNA0KMTI6MzI6NDIgZGIxIHBhdHJvbmlbMTg0 NjgwN106IHBhdHJvbmlfcG9zdGdyZXM6IDIwMjUtMDctMTYgMTI6MzI6NDIuODAwIFVUQyAxODQ2 ODA3IEZBVEFMOiAgdGhlIGRhdGFiYXNlIHN5c3RlbSBpcyBzaHV0dGluZyBkb3duDQoNCjEyOjMy OjQzIGRiMSBwYXRyb25pWzI5ODhdOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjMy OjQzLjI1OSBVVEMgMjk4OCBMT0c6ICBzaHV0dGluZyBkb3duDQoNCg0KV2UgbWlnaHQgaGF2ZSBt aXNzZWQgc29tZSBsb2dzIHRoYXQgbWlnaHQgYmUgcmVhbCBwYXRyb25pIG9yIHBvc3RncmVzIGxv Z3MgOi0oDQoxMjozMzozOSBkYjEgc3lzdGVtZC1qb3VybmFsZFsxNjEzXTogU3VwcHJlc3NlZCAx MDM3NSBtZXNzYWdlcyBmcm9tIHBhdHJvbmkuc2VydmljZQ0KDQoNCg0KVXN1YWxseSB0aGUgc3dp dGNob3ZlciBvbmx5IHRha2VzIGEgZmV3IHNlY29uZHMuIEFmdGVyIHdhaXRpbmcgYSBmZXcgbWlu dXRlcyB3ZSBiZWNhbWUgYW54aW91cyBhbmQgc3RhcnRlZCBkZWJ1Z2dpbmcuDQoNCg0KVXNpbmcg InBzIC1BbGZ8Z3JlcCBwb3N0Z3JlcyIgd2Ugc2F3IHRoYXQgdGhlcmUgd2VyZSBubyBtb3JlIG5v cm1hbCBjbGllbnQgY29ubmVjdGlvbnMsIGJ1dCBzdGlsbCA1OCBsb2dpY2FsIHJlcGxpY2F0b24g d2Fsc2VuZGVyIHByb2Nlc3NlcyBhbmQgNiBzdHJlYW1pbmcgcmVwbGljYXRpb24gd2Fsc2VuZGVy cy4NCiJ0b3AiIHJldmVhbGVkIHRoYXQgdGhlIHdhbHNlbmRlcnMgd2VyZSBlYXRpbmcgQ1BVLiBJ IGtub3cgZm9yIHN1cmUgdGhhdCBJIHNhdyBwbGVudHkgbG9naWNhbCByZXBsaWNhdGlvbiB3YWxz ZW5kZXJzIHRoYXQgd2VyZSBlYXRpbmcgMTAwJSBDUFUsIGJ1dCBJIGhhdmUgbm90IGNoZWNrZWQg aWYgYWxzbyB0aGUgc3RyZWFtaW5nIHJlcGxpY2F0aW9uIHdhbHNlbmRlcnMgd2VyZSBlYXRpbmcg Q1BVLg0KSSBhbHNvIHN0cmFjZWQgdGhlIGNoZWNrcG9pbnRlciBwcm9jZXNzLCBhbmQgdGhhdCB3 YXMgZG9pbmcgbm8gSS9PLCBqdXN0IGxvb3Bpbmcgb3IgcG9sbGluZyAoSSBjYW4ndCByZW1lbWJl ciB0aGUgZGV0YWlscykuIE91ciBQZXJmb3JtYW5jZSBNb25pdG9yaW5nIHNob3dlZCBhIGdlbmVy YWwgZHJvcCBvZiBJL08gYWN0aXZpdHkgZHVyaW5nIHRoZSBzaHV0ZG93bi4NCg0KU28gdGhpcyBz b3VuZHMgc29tZWhvdyBsaWtlIHRoZSB3YWxzZW5kZXJzIHdlcmUgc3R1Y2sgaW4gYSBzdGF0ZSB0 aGF0IGluIHRoZSBwb3N0Z3JlcyBtYWlsaW5nIGxpc3RzIGlzIGNhbGxlZCAiYnVzeSBsb29wIi4N Cg0KVGhlbiB3ZSBkZWNpZGVkIHRvIHJlbW92ZSB0aGUgVklQICh3aGljaCBpcyB1c2VkIGJ5IHRo ZSBsb2dpY2FsIHJlcGxpY2F0aW9uIGNsaWVudHMpIGZyb20gZGIxIHRvIHRyaWdnZXIgcmVwbGlj YXRpb24gZXJyb3JzOg0Kc3lzdGVtY3RsIHN0b3AgdmlwLW1hbmFnZXIuc2VydmljZQ0KMTI6NTA6 MTEgZGIxIGNvbW1hbmRsb2dbMjI3NjkwN106IHJvb3Qvcm9vdCA6IGlwIGEgZCB4LngueC54LzI0 IGRldiBib25kMC4yMDMNCjEyOjUwOjExIGRiMSBjb21tYW5kbG9nWzIyNzY5MDddOiByb290L3Jv b3QgOiBpcCAtNiBhIGQgeHh4eDp4eHg6eDp4OjoxMC82NCBkZXYgYm9uZDAuMjAzDQoNClRoZW4s IGVpdGhlciBjYXVzZWQgYnkgdGhhdCBvciBieSBzb21ldGhpbmcgZWxzZSwgdGhlIGNvbm5lY3Rp b25zIHRpbWVkIG91dC4gV2UgYXJlIHVzaW5nIHRoZSBkZWZhdWx0IHRpbWVvdXRzOg0Kd2FsX3Nl bmRlcl90aW1lb3V0OiA2MHMNCg0KRnJhbmtseSwgdGhlIHRpbWVvdXRzIHN0YXJ0ZWQgbm90IDYw cyBhZnRlciByZW1vdmluZyB0aGUgVklQLCBidXQgYWxyZWFkeSBhZnRlciA0NSBzZWNvbmRzLiBC dXQgSSBndWVzcyB0aGUgdGltZW91dCB0cmlnZ2VyZWQgYSBiaXQgZXJhbGllciBhcyB0aGVyZSB3 YXMgbm8gbW9yZSB0cmFmZmljIG9uIHRoZSB3YWxzZW5kZXIgY29ubmVjdGlvbnMgZXhjZXB0IGZv ciB0aGUgc3RhdHVzIGtlZXBhbGl2ZSBmcm9tIHRoZSBjbGllbnRzLg0KDQoxMjo1MDo1NSBkYjEg cGF0cm9uaVsyNTg2NzAwXTogcGF0cm9uaV9wb3N0Z3JlczogMjAyNS0wNy0xNiAxMjo1MDo1NS45 NzcgVVRDIDI1ODY3MDAgTE9HOiAgdGVybWluYXRpbmcgd2Fsc2VuZGVyIHByb2Nlc3MgZHVlIHRv IHJlcGxpY2F0aW9uIHRpbWVvdXQNCjEyOjUwOjU1IGRiMSBwYXRyb25pWzI1ODY3MDBdOiBwYXRy b25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjUwOjU1Ljk3NyBVVEMgMjU4NjcwMCBTVEFURU1F TlQ6ICBTVEFSVF9SRVBMSUNBVElPTiBTTE9UICJiYXJtYW5fc2JnIiA3REZELzVGMDAwMDAwIFRJ TUVMSU5FIDIyDQoxMjo1MDo1NiBkYjEgcGF0cm9uaVsyNTc5MzA5XTogcGF0cm9uaV9wb3N0Z3Jl czogMjAyNS0wNy0xNiAxMjo1MDo1Ni4wMjcgVVRDIDI1NzkzMDkgTE9HOiAgdGVybWluYXRpbmcg d2Fsc2VuZGVyIHByb2Nlc3MgZHVlIHRvIHJlcGxpY2F0aW9uIHRpbWVvdXQNCjEyOjUwOjU2IGRi MSBwYXRyb25pWzI1NzkzMDldOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjUwOjU2 LjAyNyBVVEMgMjU3OTMwOSBTVEFURU1FTlQ6ICBTVEFSVF9SRVBMSUNBVElPTiBTTE9UICJyZWdf dGFtMSIgTE9HSUNBTCA3NUQ0L0Y3RkRCQjggKHByb3RvX3ZlcnNpb24gJzQnLCBvcmlnaW4gJ2Fu eScsIHB1YmxpY2F0aW9uX25hbWVzICcibXlwdWJsaWNhdGlvbnMiJykNCjEyOjUwOjU2IGRiMSBw YXRyb25pWzgyNzQ1NV06IHBhdHJvbmlfcG9zdGdyZXM6IDIwMjUtMDctMTYgMTI6NTA6NTYuMjEy IFVUQyA4Mjc0NTUgTE9HOiAgdGVybWluYXRpbmcgd2Fsc2VuZGVyIHByb2Nlc3MgZHVlIHRvIHJl cGxpY2F0aW9uIHRpbWVvdXQNCjEyOjUwOjU2IGRiMSBwYXRyb25pWzgyNzQ1NV06IHBhdHJvbmlf cG9zdGdyZXM6IDIwMjUtMDctMTYgMTI6NTA6NTYuMjEyIFVUQyA4Mjc0NTUgU1RBVEVNRU5UOiAg U1RBUlRfUkVQTElDQVRJT04gU0xPVCAicmVnX2xpczEiIExPR0lDQUwgN0M4Ni81OTMzOTNBMCAo cHJvdG9fdmVyc2lvbiAnNCcsIG9yaWdpbiAnYW55JywgcHVibGljYXRpb25fbmFtZXMgJyJteXB1 YmxpY2F0aW9ucyInKQ0KMTI6NTA6NTYgZGIxIHBhdHJvbmlbMzM5MDgwM106IHBhdHJvbmlfcG9z dGdyZXM6IDIwMjUtMDctMTYgMTI6NTA6NTYuNTA2IFVUQyAzMzkwODAzIExPRzogIHRlcm1pbmF0 aW5nIHdhbHNlbmRlciBwcm9jZXNzIGR1ZSB0byByZXBsaWNhdGlvbiB0aW1lb3V0DQoxMjo1MDo1 NiBkYjEgcGF0cm9uaVszMzkwODAzXTogcGF0cm9uaV9wb3N0Z3JlczogMjAyNS0wNy0xNiAxMjo1 MDo1Ni41MDYgVVRDIDMzOTA4MDMgU1RBVEVNRU5UOiAgU1RBUlRfUkVQTElDQVRJT04gU0xPVCAi cmVnX3JleTEiIExPR0lDQUwgNzdBNS8yMDVGNUYwIChwcm90b192ZXJzaW9uICc0Jywgb3JpZ2lu ICdhbnknLCBwdWJsaWNhdGlvbl9uYW1lcyAnIm15cHVibGljYXRpb25zIicpDQoxMjo1MDo1NiBk YjEgcGF0cm9uaVsxMzc3NThdOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjUwOjU2 LjY3MiBVVEMgMTM3NzU4IExPRzogIHRlcm1pbmF0aW5nIHdhbHNlbmRlciBwcm9jZXNzIGR1ZSB0 byByZXBsaWNhdGlvbiB0aW1lb3V0DQoxMjo1MDo1NiBkYjEgcGF0cm9uaVsxMzc3NThdOiBwYXRy b25pX3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjUwOjU2LjY3MiBVVEMgMTM3NzU4IFNUQVRFTUVO VDogIFNUQVJUX1JFUExJQ0FUSU9OIFNMT1QgInJlZ190bHYxIiBMT0dJQ0FMIDdEQzIvQ0VBQjI3 QTAgKHByb3RvX3ZlcnNpb24gJzQnLCBvcmlnaW4gJ2FueScsIHB1YmxpY2F0aW9uX25hbWVzICci bXlwdWJsaWNhdGlvbnMiJykNCjEyOjUwOjU2IGRiMSBwYXRyb25pWzI1ODY2MjRdOiBwYXRyb25p X3Bvc3RncmVzOiAyMDI1LTA3LTE2IDEyOjUwOjU2LjkyOSBVVEMgMjU4NjYyNCBMT0c6ICB0ZXJt aW5hdGluZyB3YWxzZW5kZXIgcHJvY2VzcyBkdWUgdG8gcmVwbGljYXRpb24gdGltZW91dA0KMTI6 NTA6NTYgZGIxIHBhdHJvbmlbMjU4NjYyNF06IHBhdHJvbmlfcG9zdGdyZXM6IDIwMjUtMDctMTYg MTI6NTA6NTYuOTI5IFVUQyAyNTg2NjI0IFNUQVRFTUVOVDogIFNUQVJUX1JFUExJQ0FUSU9OIFNM T1QgImJhcm1hbiIgN0RGRC82MDAwMDAwMCBUSU1FTElORSAyMg0KDQouLi5hbmQgYWRkaXRpb25h bCA1NCBsb2dpY2FsIHdhbHNlbmRlciB0aGF0IHRpbWVkIG91dCBhbmQgd2VyZSB0ZXJtaW5hdGVk DQoNCjEyOjUxOjA1IGRiMSBwYXRyb25pWzI5ODhdOiBwYXRyb25pX3Bvc3RncmVzOiAyMDI1LTA3 LTE2IDEyOjUxOjA1LjkwMiBVVEMgMjk4OCBMT0c6ICBjaGVja3BvaW50IHN0YXJ0aW5nOiBzaHV0 ZG93biBpbW1lZGlhdGUNCjEyOjUxOjA1IGRiMSBwYXRyb25pWzI5ODhdOiBwYXRyb25pX3Bvc3Rn cmVzOiAyMDI1LTA3LTE2IDEyOjUxOjA1LjkyNiBVVEMgMjk4OCBMT0c6ICBjaGVja3BvaW50IGNv bXBsZXRlOiB3cm90ZSAxMSBidWZmZXJzICgwLjAlKTsgMCBXQUwgZmlsZShzKSBhZGRlZCwgMCBy ZW1vdmVkLCAwIHJlY3ljbGVkOyB3cml0ZT0wLjAyMSBzLCBzeW5jPTAuMDAxIHMsIHRvdGFsPTAu MDI2IHM7IHN5bmMgZmlsZXM9OSwgbG9uZ2VzdD0wLjAwMSBzLCBhdmVyYWdlPTAuMDAxIHM7IGRp c3RhbmNlPTU1IGtCLCBlc3RpbWF0ZT0xMDgwMzU0IGtCOyBsc249N0U1My8xODU1NEIyOCwgcmVk byBsc249N0U1My8xODU1NEIyOA0KMTI6NTE6MDcgZGIxIHBhdHJvbmlbMjc0OF06IHBhdHJvbmlf cG9zdGdyZXM6IDIwMjUtMDctMTYgMTI6NTE6MDcuMTU5IFVUQyAyNzQ4IExPRzogIGRhdGFiYXNl IHN5c3RlbSBpcyBzaHV0IGRvd24NCjEyOjUxOjA3IGRiMSBwYXRyb25pWzIzODldOiAyMDI1LTA3 LTE2IDEyOjUxOjA3LDk1NCBJTkZPOiBHb3QgcmVzcG9uc2UgZnJvbSBkYjIgaHR0cHM6Ly9kYjI6 ODAwOC9wYXRyb25pOiB7InN0YXRlIjogInJ1bm5pbmciLCAicG9zdG1hc3Rlcl9zdGFydF90aW1l IjogIjIwMjUtMDctMTYgMTI6MTc6MjUuNzU4NTAxKzAwOjAwIiwgInJvbGUiOiAicmVwbGljYSIs ICJzZXJ2ZXJfdmVyc2lvbiI6IDE2MDAwOSwgInhsb2ciOiB7InJlY2VpdmVkX2xvY2F0aW9uIjog MTM4ODk1MzU1NjI4NDQ4LCAicmVwbGF5ZWRfbG9jYXRpb24iOiAxMzg4OTUzNTU2Mjg0NDgsICJy ZXBsYXllZF90aW1lc3RhbXAiOiAiMjAyNS0wNy0xNiAxMjozMjo0Mi4xNDI5NTQrMDA6MDAiLCAi cGF1c2VkIjogZmFsc2V9LCAic3luY19zdGFuZGJ5IjogdHJ1ZSwgInRpbWVsaW5lIjogMjIsICJk Y3NfbGFzdF9zZWVuIjogMTc1MjY3MDI2NSwgImRhdGFiYXNlX3N5c3RlbV9pZGVudGlmaWVyIjog IjczODY1NTcyNjIyNTAwNzYxOTgiLCAicGF0cm9uaSI6IHsidmVyc2lvbiI6ICI0LjAuNSIsICJz Y29wZSI6ICJyZWdkbnNfcGF0cm9uaV9jbHVzdGVyXzEiLCAibmFtZSI6ICJkYjIifX0NCg0KDQpB ZnRlciBjbG9zaW5nIGFsbCB3YWxzZW5kZXJzICh0aGF0IHVzZWQgdGhlIFZJUCkgZHVlIHRvIHRp bWVvdXQgKDU4IGxvZ2ljYWwgc2VuZGVycyBhbmQgMiBzdHJlYW1pbmcgc2VuZGVycyB0byBiYXJt YW4pLCB0aGUgZGF0YWJhc2Ugd2FzIHNodXQgZG93biBhbmQgdGhlIHN3aXRjaG92ZXIgaGFwcGVu ZWQuDQoNCkkgZG8gbm90IGtub3cgZm9yIHN1cmUsIGJ1dCBJIHRoaW5rIHRoZSB3YWxzZW5kZXJz IHRvIHRoZSBwYXRyb25pIHJlcGxpY2FzLCB0aGF0IHdlcmUgdXNpbmcgdGhlIGhvc3QgSVAgYW5k IG5vdCB0aGUgVklQLCB3ZXJlIHN0aWxsIGFjdGl2ZSB3ZW5uIHdlIHJlbW92ZWQgdGhlIFZJUC4g QnV0IGl0IHNlZW1zIHRoZXkgaGF2ZSBzaHV0IGRvd24gbmljZWx5IGFmdGVyIGFsbCB0aGUgb3Ro ZXIgd2Fsc2VuZGVycyB0ZXJtaW5hdGVkIGR1ZSB0byB0aW1lb3V0Lg0KDQpTbyB3aGF0IGNhdXNl ZCB0aGUgaGFuZ2luZz8gV2UgZm91bmQgYnVnIGh0dHBzOi8vd3d3LnBvc3RncmVzcWwub3JnL21l c3NhZ2UtaWQvYUQwWHZoNFRqX1ZrN3BDQSU0MHBhcXVpZXIueHl6IGJ1dCBpdCBzZWVtcyB0aGF0 IG9ubHkgYWZmZWN0ZWQgbG9naWNhbCB3YWxzZW5kZXIgaW4gc3RhbmRieSBtb2RlLg0KDQpXZSBj YW4gbm90IHJlcHJvZHVjZSBpdCAtIGFmdGVyIGEgZmV3IG1pbnV0ZXMgd2UgbWFkZSBhbm90aGVy IHN3aXRjaG92ZXIgYmFjayB0byBkYjEgYW5kIHRoaXMgdGltZSBpdCB3YXMgZG9uZSB3aXRoaW4g YSBmZXcgc2Vjb25kcy4NCg0KRG8geW91IGhhdmUgYW55IGlkZWFzIHdoYXQgY291bGQgYmUgdGhl IHByb2JsZW0sIGhvdyB3ZSBjb3VsZCBzb2x2ZSBpdCBhbmQgaG93IHRvIGRlYnVnIGlmIGl0IGhh cHBlbnMgYWdhaW4/IA0KDQpUaGFua3MNCktsYXVz --_004_DBAPR03MB6358854AD71C8ABA5CA10A8DF15DADBAPR03MB6358eurp_--