Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r1VNz-00BZil-ST for pgsql-admin@arkaria.postgresql.org; Fri, 10 Nov 2023 17:33:40 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1r1VNx-004TSi-CF for pgsql-admin@arkaria.postgresql.org; Fri, 10 Nov 2023 17:33:37 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1r1VNw-004TSS-4E for pgsql-admin@lists.postgresql.org; Fri, 10 Nov 2023 17:33:36 +0000 Received: from sonic305-20.consmr.mail.ir2.yahoo.com ([77.238.177.82]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1r1VNs-004wEP-HZ for pgsql-admin@lists.postgresql.org; Fri, 10 Nov 2023 17:33:34 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.co.uk; s=s2048; t=1699637609; bh=xYGDaMzajXpVR+C0Ad1FwUhhuxEXEYkBaGYWjCnBmkI=; h=Date:Subject:To:References:From:In-Reply-To:From:Subject:Reply-To; b=TC0BJeX8947j0+cgqR7/4ofQCXL8Uuj2m1W0m0RoQUF994L4fPShcNJEmJ7B05cHS5JiV19swsmw6H9ydgQd3Be2pufNHT2NbAuBD5KA7uBW/M9WWbBfBIMnqdXxIpGgK2iMYSFHFPATNmkIHezmWpr+AIdyZtpcdlBlnkJMJE0MJQh7n9rfKvkGH+h53yyyA6Q+6//hwWS9SNgUar8rpclBIn5JNtzYeOR6QXgM5n9M+C4mvcHd4HHXUiNlL7ijSoj8arSVBeEm+kK/bfNwo/LTqd+MCN1Fir9AefbiVkMa3fTjNcY4ljO+JnK33La7QCzgedsuHn2dPTqVpc4AiQ== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1699637609; bh=t55m58sS6i7DIQwot4sHrs6VWumA0A/KCfQwK/z46hZ=; h=X-Sonic-MF:Date:Subject:To:From:From:Subject; b=k4NwVQsHJVsPGechdwSYHV+cngpreMpGPTDeT0h2LwJGqwZTzqZl76zI9qdKROd5meiX4df62P5r+oSJCNgycYGoVXyf0cxj0VZ2gB88Cc6fJI0sCPzyWbNvJsb6YK6yZ3mysI+PzsmmL85sl0BqPc/yxNyc5hXJvUc1DnrU87OYafbmaXYIuKr8Tz1ACYNUDcjMv64IlXS9pYka6Jp/PVObyEzgf9XMxrS0ewWvFtvq6cF+WnI/cnKwyrbzyzx2+656hNzl7QFy9JJPdvkl8TvTqMFdeezV3GuyPfmXgbeNuioVPjs61zbmoNVw+esHypKajn4vLhmzRilburHguw== X-YMail-OSG: v.aHfTMVM1mQTy2K9Dy4PTm0ptJsmQqFcoxOOKoGWNPn1Y8aGWDGP1kJhgPJCir B7wWku6dVGFfxlRn8MWU.Jwd602hMY7VKqqdd7bRfSzIFJYzTxeg34K.QnJewFI30j4UMK6UGV93 65Q5bSy.UrIcRjZUGc0GF7JhBLLBhOtA9DZnkS.gvQzE.gTVywF7IG4.uPVcQ0c3ZiPXiHbUAbE6 xABQVviOHFomS45WNOVzxtfWJLFB9IdnezSNdYrCytwUksUElTWGN2WIlyqRgLocW5zkHlmOOplx aAal610sBf6S3NeJ00_riBJLprByqya483Ag9.kmI6lTw2PG6F8DiuqD31PnbGa_Bp3BAtVz_Z7C hZ.RNUbaclpWjqI_ud6VTkRCgU55UVHfAP8DPYV3ubuSFroawpKq0gJN77KXKjVhQxcIqSepUvaP sXEX0VHCgBcNLwlIexr5nvRpfJFAdDU5duEjB7klTRF9Zqn.cBDk2dpUzZFCwGr6qaHlZcP1awor vzEDHvLqsCbIXqKRj1xVz8k0MqPFPVEEeUMFEnw_op1ZHqCrroQjaidIOrIzCf6W4e.3n02_gjDr Y2YKhLQ8hgOOXOPZHpELNaytAw5PPllgGzgcDysJa8PqvgcdEB6F1TnjWG4YCi5Q2mHVdjbKJXmQ c4JHxEY_3YCN9MbrfUENV1IgjEEWf9UaUiZUQ3rQP3_xOmx0h2MQR1sXdfcbekkCHDtsYtF7g4XX On2_bSBXbzeZzvEnM7fVBKhW9m472erKRf8mbLXELKKzMIhlgrGp1RpSBNEWAV0qKLGTKcw62tXU 8DjfVvvv6amUUy6gRZnJngujSwquILVlyn5XS40sryHI.6FpYoqeIm9Kr18vuTvPXKyYYNACoFKR yXs5M8hdA9YLJ.v6rLdl.jnOItnNL5R9g78u1RnuKmGO7WAGAOd8BRSDmGGS901iUmeXn3BJwpOp JEg3UWHs0r.ZW.V7ZZy2ePzzX_D_jA3DbAsat.UNa_D1YyWtX_vuZzFFSUgOrosz.WJqOrIH06AT BuLVXXNvF9FKjzcuGP6YFTaSrA8MHs.SEkbNK6NTbsgL4KM2NRdIVfhLjkKJx4bi5qTm8RsiRhyZ CV8RTIU_kO0x1bpZvPNhkQsIBfzydtW0no_SiDBuEpW0BbrxgWhlFFMoGJMSzNQkNnusloufbP2y 3FoPiXqpYetSUHstH30atDoaHDI_KcWTd9AKBPi3JSrd9VjZEQOacHO_ZGCohLcCc3teEEA9ESfB c77hX5.H33FLF5Nvn6JDxSJkURWr66xUPKFLo5mQIKbt0.ofj_vkWEL2Qtdki7N8l2c0y5P.vqra SBhylP.6YQJCRU9DpCfs.sHDimlx9Q8tA7xT63BwH5stfpCQ_wV6qK4vjHbR3nXTL9hhNBYRlLnw GBpfCZ6gEQqTz99IglMCOLVZb7Ys9Kg4XjHqwzvq3rTw3InD8vxoKCouAeuUvngzyt6uo5JIoFaV 2tyNfaRbPC.3pOsrXKSb5e8ornGXHr0.pQI49Y7iLbg_Z55n0GD5bcqgadfUblIGBb86mSVK9Eir a0ARIwZrDCp9YjOFeStsQHd9f2QE3Bo5N74o_8gg9ePvzrMOgWsx_oEXRRbJEQxGdnCH15zJ3uCI leamOJEoLcx6gkrqQM8usrdFpcHePy168zO6xXGE1AZjTn4Mv3NOnz4MMkFhUSpb0xX5dLGW5AvB MQoRUmSFJ6tXPjUYwTLHVTy6.OTfoD3ByujYjnpasRhxtP6FOdRilB9ChW4VXRD1b2ovMdJlgpPM BrdoTXoNmynHgLNGGJZKuhmW8f3_KcCvgAI47rgagcKqo705vRBtWN3NwFjK6htcfnGpkbwjjPzb RZTmlI_wgmjCZh3b.q9ksl52fvyo5MYIPAxhXRJMIELii1P.YOLN20oKwXCO4QuIPfBml87mNv6d Sq9tLw_h5oHD90KXLoiUOxWAgGhxtKFVLo0KyVg.NW1A28tBM.dMUpdPzegoQi6w_NEe_g08kVzR SwY7zeGulKgim6nG6gtpOtYvadr3vimzX1N79xKvXbPiatUHmkj7m4wN3e6xFeXbVLl5I0GTCGvW TyxPVMarhF9DBkrgsRoWSkgnDNeVoXkllIKbVM9BLASrceTwN9D8lQlAVUBpUQPDkfDXtu7JdcuW I_CZuKA_LOk4ojaLRxGI8VucWUQHOtJczHzBb6HTpGoAtelR3UhaurHpO7s76Nt4.9Iv_FyfxUnv lMVeYhgGsDvzMqorFjCrPOyCQXahtefeqjosfwP8VMwAtc700E4INh_mwhO7U4u1bjmPjQkJLlwz fWJ6eRlCr.w-- X-Sonic-MF: X-Sonic-ID: 169717ea-e96d-49a0-9e40-e3b88e64cc5c Received: from sonic.gate.mail.ne1.yahoo.com by sonic305.consmr.mail.ir2.yahoo.com with HTTP; Fri, 10 Nov 2023 17:33:29 +0000 Received: by hermes--production-ir2-6755f9ff7c-ltc4c (Yahoo Inc. Hermes SMTP Server) with ESMTPA ID 5877dc897d5d3ecaa8beafbc6ac08cc7; Fri, 10 Nov 2023 17:33:24 +0000 (UTC) Message-ID: <6fc34508-2b22-488e-a908-047039b549d6@yahoo.co.uk> Date: Fri, 10 Nov 2023 18:33:23 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: fails to start - pg_stat_tmp ? Content-Language: en-US To: pgsql-admin@lists.postgresql.org References: <3c1964cc-ac89-4378-9b4f-841d8cb0fe01.ref@yahoo.co.uk> <3c1964cc-ac89-4378-9b4f-841d8cb0fe01@yahoo.co.uk> <2874e719d29908405c46ddd4273222aa3544afde.camel@cybertec.at> From: lejeczek In-Reply-To: <2874e719d29908405c46ddd4273222aa3544afde.camel@cybertec.at> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Mailer: WebService/1.1.21896 mail.backend.jedi.jws.acl:role.jedi.acl.token.atz.jws.hermes.yahoo Content-Length: 1444 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 10/11/2023 13:20, Laurenz Albe wrote: > On Fri, 2023-11-10 at 11:33 +0100, lejeczek wrote: >> I have simple cluster managed by a popular toolset, I believe, and I get a standby/slave node which >> after it was shutdown, rebooted, is unable to start, not by the that cluster-manager nor manually. > Thatks for telling us all the detail... > >>  FATAL:  could not open directory "/var/run/postgresql/14-paf.pg_stat_tmp": No such file or directory >>  LOG:  startup process (PID 54860) exited with exit code 1 >> >> Indeed that dir, that path does not exist on the node - from what I can see, among the nodes >> comprising the cluster this path exists only on the _master_ node. > Looks like somebody randomly deleted directories on that machine. > > Create it again, and make sure it belongs to the postgres user and > nobody else has write access. > > Yours, > Laurenz Albe > > Indeed something removes at/after reboot the path, last bit of with stats - I started fiddling with _systemd_ tmpfiles which I thought was the culprit but it seems systemd neither creates nor cleans it. I also have bucardo - cleaned it up, did not help nor after I completely removed bucardo pg_ctlcluster start - does work, with this wrapper path gets created and server starts OS journal boot logs do not mention this path neither. It's a true sticky-wicket for me right now - to add, other two virtually identical nodes seem free from this issue.