Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1V4JCu-0005i3-UM for pgsql-admin@arkaria.postgresql.org; Tue, 30 Jul 2013 23:19:57 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.80) (envelope-from ) id 1V4JCu-0003i2-DD for pgsql-admin@arkaria.postgresql.org; Tue, 30 Jul 2013 23:19:56 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1V4JCt-0003hv-3D for pgsql-admin@postgresql.org; Tue, 30 Jul 2013 23:19:55 +0000 Received: from mail-ob0-f173.google.com ([209.85.214.173]) by magus.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1V4JCp-0001Vp-BQ for pgsql-admin@postgresql.org; Tue, 30 Jul 2013 23:19:54 +0000 Received: by mail-ob0-f173.google.com with SMTP id ta17so46405obb.18 for ; Tue, 30 Jul 2013 16:19:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=message-id:date:from:user-agent:mime-version:to:cc:subject :content-type:x-gm-message-state; bh=n6jvXE50/gBjld90VwnWlPNvaGYT/1F/8qF4YcEZN3I=; b=GXiYDuomvQX5wlraXHFeVxbBzaNh5S+6og2RTP6tu/S5/+G4+LmF9QtIT9E7Zu9wO8 rD8VsLyPymiGElp0gTYw1b5RBmlZuafMnLudEVSTaVu0HUWhhXN1GctHqVfU/R7w1BRk zNlMQ+Fcq1sSpetpgHQRbAGidCoyHRMjyvLPW8AGOtnpRAcy7SYXcoJKgBSY4VV4SnF/ iqfu4pOer3zMa/Osql+7fEgl43MhjTdOU7oF3zL+VdzOZKSB0oZOD4l/kLt9VZDEaM3h FIekWt9GXcG9krOC5r/tNvCPkaLs9vGaDrR26jF2ZR4qbRvvVZkMv5PhgYaCZaVIOWNy tNYA== X-Received: by 10.182.87.170 with SMTP id az10mr59200809obb.10.1375226389261; Tue, 30 Jul 2013 16:19:49 -0700 (PDT) Received: from Kevins-MacBook-Pro.local ([65.100.173.26]) by mx.google.com with ESMTPSA id j7sm19708544oew.5.2013.07.30.16.19.47 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 30 Jul 2013 16:19:48 -0700 (PDT) Message-ID: <51F84A14.4000801@consistentstate.com> Date: Tue, 30 Jul 2013 17:19:48 -0600 From: CS DBA User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.8; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: "pgsql-admin@postgresql.org" CC: cs_dba@consistentstate.com Subject: db crash, streaming rep slave will not start Content-Type: multipart/alternative; boundary="------------020200060509040405000209" X-Gm-Message-State: ALoCoQnOoNyHrn+zOlBowGGCh5IDiNgfRku667yPq5OuBG+GGKGYivl8i9I3+UMadeyP2omcuXVw X-Pg-Spam-Score: -2.6 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-admin Precedence: bulk Sender: pgsql-admin-owner@postgresql.org This is a multi-part message in MIME format. --------------020200060509040405000209 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Hi All; A client's master database crashed, they tried to startup the streaming replication slave and it refuses to start. See the log details below... thanks in advance for any help Master log: 2013-07-30 16:23:01 MDT PANIC: corrupted page pointers: lower = 0, upper = 0, special = 0 2013-07-30 16:23:02 MDT LOG: server process (PID 17539) was terminated by signal 6: Abort trap 2013-07-30 16:23:02 MDT LOG: terminating any other active server processes 2013-07-30 16:23:02 MDT [local]WARNING: terminating connection because of crash of another server process 2013-07-30 16:23:02 MDT [local]DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2013-07-30 16:23:02 MDT [local]HINT: In a moment you should be able to reconnect to the database and repeat your command. 2013-07-30 16:23:02 MDT [local]WARNING: terminating connection because of crash of another server process 2013-07-30 16:23:02 MDT [local]DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2013-07-30 16:23:02 MDT [local]HINT: In a moment you should be able to reconnect to the database and repeat your command. 2013-07-30 16:23:02 MDT [local]WARNING: terminating connection because of crash of another server process 2013-07-30 16:23:02 MDT [local]DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2013-07-30 16:23:04 MDT [local]FATAL: the database system is in recovery mode 2013-07-30 16:23:04 MDT LOG: archiver process (PID 1826) exited with exit code 1 2013-07-30 16:23:04 MDT 192.168.131.2FATAL: the database system is in recovery mode 2013-07-30 16:23:04 MDT LOG: all server processes terminated; reinitializing 2013-07-30 16:23:04 MDT LOG: database system was interrupted; last known up at 2013-07-30 16:21:33 MDT 2013-07-30 16:23:04 MDT LOG: database system was not properly shut down; automatic recovery in progress 2013-07-30 16:23:04 MDT LOG: consistent recovery state reached at 1179D/8B7E7EF8 2013-07-30 16:23:04 MDT LOG: redo starts at 1179A/C1001EA8 2013-07-30 16:26:48 MDT LOG: record with zero length at 1179D/AC2591A8 2013-07-30 16:26:48 MDT LOG: redo done at 1179D/AC259168 2013-07-30 16:26:48 MDT LOG: last completed transaction was at log time 2013-07-30 16:23:02.11493-06 2013-07-30 16:26:48 MDT WARNING: page 476 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 493 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 1023 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 708 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 1075 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 590 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 832 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 1742 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 238 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 334 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 1131 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 434 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 772 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 259 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 498 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 948 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 1743 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 96 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT WARNING: page 559 of relation base/603188/199093492 did not exist 2013-07-30 16:26:48 MDT PANIC: WAL contains references to invalid pages 2013-07-30 16:26:48 MDT LOG: startup process (PID 17546) was terminated by signal 6: Abort trap 2013-07-30 16:26:48 MDT LOG: aborting startup due to startup process failure Slave log: 2013-07-30 16:41:48 MDT FATAL: could not connect to the primary server: could not connect to server: Operation timed out Is the server running on host "192.168.131.1" and accepting TCP/IP connections on port 5432? 2013-07-30 16:42:59 MDT LOG: trigger file found: /pgdata/data/failover 2013-07-30 16:42:59 MDT FATAL: terminating walreceiver process due to administrator command 2013-07-30 16:42:59 MDT LOG: redo done at 1179D/AC11DF00 2013-07-30 16:42:59 MDT LOG: last completed transaction was at log time 2013-07-30 16:23:01.986951-06 2013-07-30 16:42:59 MDT LOG: selected new timeline ID: 2 2013-07-30 16:42:59 MDT LOG: archive recovery complete 2013-07-30 16:42:59 MDT WARNING: page 98 of relation base/603188/4268050827 did not exist 2013-07-30 16:42:59 MDT WARNING: page 476 of relation base/603188/199093492 did not exist 2013-07-30 16:42:59 MDT WARNING: page 571 of relation base/603188/2775093183 did not exist 2013-07-30 16:42:59 MDT WARNING: page 202 of relation base/603188/4268050827 did not exist 2013-07-30 16:42:59 MDT WARNING: page 202 of relation base/603188/3435025974 did not exist 2013-07-30 16:42:59 MDT WARNING: page 493 of relation base/603188/199093492 did not exist 2013-07-30 16:42:59 MDT WARNING: page 1023 of relation base/603188/199093492 did not exist 2013-07-30 16:42:59 MDT WARNING: page 163 of relation base/603188/3476677873 did not exist 2013-07-30 16:42:59 MDT WARNING: page 15 of relation base/603188/3435025974 did not exist 2013-07-30 16:42:59 MDT PANIC: WAL contains references to invalid pages 2013-07-30 16:43:00 MDT LOG: startup process (PID 41056) was terminated by signal 6: Abort trap 2013-07-30 16:43:00 MDT LOG: terminating any other active server processes 2013-07-30 16:43:00 MDT 10.254.254.23WARNING: terminating connection because of crash of another server process 2013-07-30 16:43:00 MDT 10.254.254.23DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. --------------020200060509040405000209 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Hi All;

A client's master database crashed, they tried to startup the streaming replication slave and it refuses to start.

See the log details below... thanks in advance for any help





Master log:

2013-07-30 16:23:01 MDT PANIC: corrupted page pointers: lower = 0, upper = 0, special = 0

2013-07-30 16:23:02 MDT LOG: server process (PID 17539) was terminated by signal 6: Abort trap

2013-07-30 16:23:02 MDT LOG: terminating any other active server processes


2013-07-30 16:23:02 MDT [local]WARNING: terminating connection because of crash of another server process

2013-07-30 16:23:02 MDT [local]DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2013-07-30 16:23:02 MDT [local]HINT: In a moment you should be able to reconnect to the database and repeat your command.

2013-07-30 16:23:02 MDT [local]WARNING: terminating connection because of crash of another server process

2013-07-30 16:23:02 MDT [local]DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.

2013-07-30 16:23:02 MDT [local]HINT: In a moment you should be able to reconnect to the database and repeat your command.

2013-07-30 16:23:02 MDT [local]WARNING: terminating connection because of crash of another server process

2013-07-30 16:23:02 MDT [local]DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.


2013-07-30 16:23:04 MDT [local]FATAL: the database system is in recovery mode

2013-07-30 16:23:04 MDT LOG: archiver process (PID 1826) exited with exit code 1

2013-07-30 16:23:04 MDT 192.168.131.2FATAL: the database system is in recovery mode

2013-07-30 16:23:04 MDT LOG: all server processes terminated; reinitializing

2013-07-30 16:23:04 MDT LOG: database system was interrupted; last known up at 2013-07-30 16:21:33 MDT

2013-07-30 16:23:04 MDT LOG: database system was not properly shut down; automatic recovery in progress

2013-07-30 16:23:04 MDT LOG: consistent recovery state reached at 1179D/8B7E7EF8

2013-07-30 16:23:04 MDT LOG: redo starts at 1179A/C1001EA8


2013-07-30 16:26:48 MDT LOG: record with zero length at 1179D/AC2591A8

2013-07-30 16:26:48 MDT LOG: redo done at 1179D/AC259168

2013-07-30 16:26:48 MDT LOG: last completed transaction was at log time 2013-07-30 16:23:02.11493-06

2013-07-30 16:26:48 MDT WARNING: page 476 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 493 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 1023 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 708 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 1075 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 590 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 832 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 1742 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 238 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 334 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 1131 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 434 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 772 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 259 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 498 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 948 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 1743 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 96 of relation base/603188/199093492 did not exist

2013-07-30 16:26:48 MDT WARNING: page 559 of relation base/603188/199093492 did not exist


2013-07-30 16:26:48 MDT PANIC: WAL contains references to invalid pages

2013-07-30 16:26:48 MDT LOG: startup process (PID 17546) was terminated by signal 6: Abort trap

2013-07-30 16:26:48 MDT LOG: aborting startup due to startup process failure




Slave log:

2013-07-30 16:41:48 MDT FATAL: could not connect to the primary server: could not connect to server: Operation timed out

Is the server running on host "192.168.131.1" and accepting

TCP/IP connections on port 5432?


2013-07-30 16:42:59 MDT LOG: trigger file found: /pgdata/data/failover

2013-07-30 16:42:59 MDT FATAL: terminating walreceiver process due to administrator command

2013-07-30 16:42:59 MDT LOG: redo done at 1179D/AC11DF00

2013-07-30 16:42:59 MDT LOG: last completed transaction was at log time 2013-07-30 16:23:01.986951-06

2013-07-30 16:42:59 MDT LOG: selected new timeline ID: 2

2013-07-30 16:42:59 MDT LOG: archive recovery complete

2013-07-30 16:42:59 MDT WARNING: page 98 of relation base/603188/4268050827 did not exist

2013-07-30 16:42:59 MDT WARNING: page 476 of relation base/603188/199093492 did not exist

2013-07-30 16:42:59 MDT WARNING: page 571 of relation base/603188/2775093183 did not exist

2013-07-30 16:42:59 MDT WARNING: page 202 of relation base/603188/4268050827 did not exist

2013-07-30 16:42:59 MDT WARNING: page 202 of relation base/603188/3435025974 did not exist

2013-07-30 16:42:59 MDT WARNING: page 493 of relation base/603188/199093492 did not exist

2013-07-30 16:42:59 MDT WARNING: page 1023 of relation base/603188/199093492 did not exist

2013-07-30 16:42:59 MDT WARNING: page 163 of relation base/603188/3476677873 did not exist

2013-07-30 16:42:59 MDT WARNING: page 15 of relation base/603188/3435025974 did not exist


2013-07-30 16:42:59 MDT PANIC: WAL contains references to invalid pages

2013-07-30 16:43:00 MDT LOG: startup process (PID 41056) was terminated by signal 6: Abort trap

2013-07-30 16:43:00 MDT LOG: terminating any other active server processes

2013-07-30 16:43:00 MDT 10.254.254.23WARNING: terminating connection because of crash of another server process

2013-07-30 16:43:00 MDT 10.254.254.23DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.










--------------020200060509040405000209--