Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jXRjO-00041B-Us for pgsql-bugs@arkaria.postgresql.org; Sat, 09 May 2020 15:49:38 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jXRjM-0004iD-Ok for pgsql-bugs@arkaria.postgresql.org; Sat, 09 May 2020 15:49:36 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jXRjM-0004i2-J2 for pgsql-bugs@lists.postgresql.org; Sat, 09 May 2020 15:49:36 +0000 Received: from sss.pgh.pa.us ([66.207.139.130]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jXRjG-0005K1-CJ for pgsql-bugs@lists.postgresql.org; Sat, 09 May 2020 15:49:35 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.14.4/8.14.4) with ESMTP id 049FnTGE005217; Sat, 9 May 2020 11:49:29 -0400 From: Tom Lane To: Rod Taylor cc: Thusitha Maheepala , pgsql-bugs@lists.postgresql.org Subject: Re: Postgredb issue In-reply-to: References: Comments: In-reply-to Rod Taylor message dated "Sat, 09 May 2020 11:05:58 -0400" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <5215.1589039369.1@sss.pgh.pa.us> Date: Sat, 09 May 2020 11:49:29 -0400 Message-ID: <5216.1589039369@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Rod Taylor writes: > It appears all errors are related to the c_acctprocessorlog_pkey index? Run > the below to ditch the current index and rebuild with data from the table > files. > REINDEX INDEX c_acctprocessorlog_pkey; That appears to have been the original problem, but at the end of the log we see that the database is refusing to start, which suggests some other filesystem corruption has manifested. I'm suspecting some combination of unreliable storage, ancient/buggy Postgres version etc. It's troubling that the "last completed transaction" log messages suggest that no checkpoint has completed since last July ... has the OP somehow turned off checkpoints? Ordinarily the only thing that could prevent checkpoint completion is I/O errors, but there's no trace of such in the log. I wonder how much WAL has accumulated in the pg_wal/pg_xlog area. It's possible that a professional data recovery service could help you get something out of this mess. If the data is worth money to you, see [1] for some places to try. Otherwise, chalk it up to experience and start over. regards, tom lane [1] https://www.postgresql.org/support/professional_support/