Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pPB8e-0006Wv-8d for pgsql-hackers@arkaria.postgresql.org; Mon, 06 Feb 2023 23:43:08 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1pPB8d-0002f6-5B for pgsql-hackers@arkaria.postgresql.org; Mon, 06 Feb 2023 23:43:07 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pPB8c-0002eq-SL for pgsql-hackers@lists.postgresql.org; Mon, 06 Feb 2023 23:43:06 +0000 Received: from out4-smtp.messagingengine.com ([66.111.4.28]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1pPB8a-0006zB-A3 for pgsql-hackers@lists.postgresql.org; Mon, 06 Feb 2023 23:43:06 +0000 Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 5888D5C01A1; Mon, 6 Feb 2023 18:43:03 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 06 Feb 2023 18:43:03 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=anarazel.de; h= cc:cc:content-transfer-encoding:content-type:date:date:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to; s=fm3; t=1675726983; x= 1675813383; bh=CI7cfT+1RW9KsakFUlZjPPthdoKoK+apj3N/L0P20w8=; b=4 9KIQJgqYZgWLsLbbJMJF/VokcAmZeMsOO7VOx4gNgsmPvgda7pdQtLEou0o6h4R9 tp1PZVvmUPd2QE2scDpaGcEkm2jnzm34GzWcSWnvwYr37CJkAkZAjDQyg6A1Mfmg UpKR97OOh5ZNB9A2Uo4JkrTuoWIN1lW58daRfE26y4RBGTLLmTAHBAzf3nAgjFmC l+kllp1fsEtojyP8OJ0EFkg5q6od6HpPLcKutMBSHwfh3tFuWfHVCqkG4QPflPRJ j0vhEdcIKSSVyRj96A+u7uNX674v6cK5v87jBfXJG7Kwed4K6WRHcYoWqviuXe5D gbcV79kkUlcMiAQJJDioA== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-transfer-encoding :content-type:date:date:feedback-id:feedback-id:from:from :in-reply-to:in-reply-to:message-id:mime-version:references :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm3; t=1675726983; x= 1675813383; bh=CI7cfT+1RW9KsakFUlZjPPthdoKoK+apj3N/L0P20w8=; b=U VxO6+g3CY1ftK+RARAFkPGM4MnubyVVYcMSqH72CPPypiz4qW7EXfFxzOUpcEBLu FcNvv1DJSt81o8UyUcHYL5EcWItqlQF/m6Z3HZqLm/nDVuy8C0TjShh0W/QTN4xP f/9WKBYT9GbiqbShUYUoJr179B6Q2vCjm5c3rtpnL1I1pcDIsxC/ICtFLNyQorMY ch36z4A5rSi4Wle6PhJzd5TxJ1TA58anLQa5/GvlI0NiSRKwCW9Z/iUrgn2y1BYI DmV82fhQSLpKktQoD0S/MpvJhyxpIMdBPxEn6W0U8p/SX7cSOLNKHyN3h3s90ylh LRh/fOQ6IeCEkFCRPN51w== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvhedrudegjedgudefucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne goufhushhpvggtthffohhmrghinhculdegledmnecujfgurhepfffhvfevuffkfhggtggu gfgjsehtqhertddttddvnecuhfhrohhmpeetnhgurhgvshcuhfhrvghunhguuceorghnug hrvghssegrnhgrrhgriigvlhdruggvqeenucggtffrrghtthgvrhhnpedvudekueekfffg hfetudfgffdtteeutdefveelieduhfevffdufeejjeevueffleenucffohhmrghinhepgh hoohhglhgvrghpihhsrdgtohhmnecuvehluhhsthgvrhfuihiivgeptdenucfrrghrrghm pehmrghilhhfrhhomheprghnughrvghssegrnhgrrhgriigvlhdruggv X-ME-Proxy: Feedback-ID: id4a34324:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Mon, 6 Feb 2023 18:43:02 -0500 (EST) Date: Mon, 6 Feb 2023 15:43:01 -0800 From: Andres Freund To: Thomas Munro Cc: Justin Pryzby , Michael Paquier , pgsql-hackers@lists.postgresql.org, Nazir Bilal Yavuz Subject: Re: pg_upgrade test failure Message-ID: <20230206234301.33c5vvtd7f3rs6qv@awork3.anarazel.de> References: <20230131172806.GM22427@telsasoft.com> <3FC7ACA2-5A65-4CAB-AC3D-8F9695A97A2C@anarazel.de> <20230206221422.ykidlne7pmywu4ay@awork3.anarazel.de> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable In-Reply-To: <20230206221422.ykidlne7pmywu4ay@awork3.anarazel.de> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On 2023-02-06 14:14:22 -0800, Andres Freund wrote: > On 2023-02-07 11:03:18 +1300, Thomas Munro wrote: > > What I see is that there were 1254 FreeBSD tasks run in that window, of > > which 163 failed, and (more interestingly) 111 of those failures succee= ded > > on every other platform. And clicking on a few on cfbot's page reveals= that > > it's the new running stuff, and I'm still trying to find the interesting > > logs... >=20 > I think I figured out why the logs frequently fail to upload - the server= is > still running, so the size changes during the upload, causing the upload = to > fail with errors like: >=20 > [12:46:43.552] Failed to upload artifacts: Put "https://storage.googleapi= s.com/cirrus-ci-5309429912436736-3271c9/artifacts/postgresql-cfbot/postgres= ql/6729936359129088/testrun/build/testrun/runningcheck.log?X-Goog-Algorithm= =3DGOOG4-RSA-SHA256&X-Goog-Credential=3Dcirrus-ci%40cirrus-ci-community.iam= =2Egserviceaccount.com%2F20230206%2Fauto%2Fstorage%2Fgoog4_request&X-Goog-D= ate=3D20230206T124536Z&X-Goog-Expires=3D600&X-Goog-SignedHeaders=3Dhost%3Bx= -goog-content-length-range%3Bx-goog-meta-created_by_task&X-Goog-Signature= =3D8e84192cbc754180b8baa6c00c41b463f580fe7183f0e7113c253aac13cc2458b835caef= 7940b91e102e96d54cff2b5714c77390e74244e2fb88c00c9957a801e33cbee2ac960e0db8a= 01fe08ee945bedf4616881e6beafa3a162c22948ac0b9a9359d93e1f461fc9f49385b784b75= d633f1b01805b987d9d53bc7fb55263917ec85180a2140659d50990f066160f03e8bb8984e8= d2aadb64c875c253167cf24da152a18d69fcd3d941edce145931e4feb23dc8cf43de7b7bbfc= 565786c1c692406f2a0a127f30385a8c4b66f96709b51d26d3c71617991c731b0e7206ee390= 6338dedf6359412edd024f8c76bd33400f4c9320c2bde9512fa8bcd6289e54d52": http2: = request body larger than specified content length >=20 > I'm testing adding a pg_ctl stop to the on_failure right now. Pushed the fix doing so.