pg.ddx.io pgsql-admin@postgresql.org mailing list archivehelp / color / mirror / Atom feed
Failing replication 2+ messages / 1 participants [nested] [flat]
* Failing replication @ 2014-06-02 23:49 Scott Whitney <scott@journyx.com> 2014-06-03 00:35 ` Re: Failing replication Scott Whitney <scott@journyx.com> 0 siblings, 1 reply; 2+ messages in thread From: Scott Whitney @ 2014-06-02 23:49 UTC (permalink / raw) To: pgsql-admin So, I have 3 servers: 1) Production. 2) "Production" training server. Replicates #1. At same data center. 2 postmasters on separate ports. 3) Internal "production" server. 3 postmasters on separate ports. Replicates #2 and #1 (both from #2). This has happened to me several times in the past few weeks. ^ permalink raw reply [nested|flat] 2+ messages in thread
* Re: Failing replication 2014-06-02 23:49 Failing replication Scott Whitney <scott@journyx.com> @ 2014-06-03 00:35 ` Scott Whitney <scott@journyx.com> 0 siblings, 0 replies; 2+ messages in thread From: Scott Whitney @ 2014-06-03 00:35 UTC (permalink / raw) To: Scott Whitney <swhitney@journyx.com>; +Cc: pgsql-admin rsync has completed. Cascading (#3 -> #1 via #2) is up and running again. Log as of now: LOG: consistent recovery state reached at 31C/1C979FD8 LOG: database system is ready to accept read only connections LOG: record with zero length at 31C/1CCDF680 LOG: streaming replication successfully connected to primary === me....it's working 7:29 pm === scott again...restarting pg -> db01 at 6/2/14 7:33 just to see what happens. LOG: database system was shut down in recovery at 2014-06-02 19:34:07 CDT LOG: entering standby mode LOG: redo starts at 31C/1F443578 LOG: consistent recovery state reached at 31C/20278980 LOG: record with incorrect prev-link 319/60278768 at 31C/20278980 LOG: database system is ready to accept read only connections LOG: streaming replication successfully connected to primary ----- Original Message ----- > So, I have 3 servers: > 1) Production. > 2) "Production" training server. Replicates #1. At same data center. 2 > postmasters on separate ports. > 3) Internal "production" server. 3 postmasters on separate ports. Replicates > #2 and #1 (both from #2). > This has happened to me several times in the past few weeks. > From #3: > LOG: entering standby mode > LOG: redo starts at 31B/FDFFC0D8 > WARNING: page 7843 of relation base/25999670/33421425 is uninitialized > PANIC: WAL contains references to invalid pages > LOG: startup process (PID 4580) was terminated by signal 6: Aborted > LOG: terminating any other active server processes > === > here is me restarting...adding this since there are no timestamps in this > log... > === > LOG: database system was interrupted while in recovery at log time 2014-06-02 > 17:31:05 CDT > HINT: If this has occurred more than once some data might be corrupted and > you might need to choose an earlier recovery target. > LOG: entering standby mode > LOG: redo starts at 31B/FDFFC0D8 > WARNING: page 7843 of relation base/25999670/33421425 is uninitialized > PANIC: WAL contains references to invalid pages > LOG: startup process (PID 4838) was terminated by signal 6: Aborted > LOG: terminating any other active server processes > There is no activity in #2's pg_log postgresql-Mon.log file consummate to > this event. > I have wal segments keeping 1,000. I have pg_xlog files dated yesterday. > This happened earlier today as well. It happened twice last week. > The "warning page" stuff is different from the pg_controldata stuff earlier > today. > Here is pg_controldata from server 3 wrt server 1 replication. > pg_control version number: 922 > Catalog version number: 201204301 > Database system identifier: 5868266298097360799 > Database cluster state: in archive recovery > pg_control last modified: Mon Jun 2 18:42:18 2014 > Latest checkpoint location: 31B/FE08DB50 > Prior checkpoint location: 31B/FE08DB50 > Latest checkpoint's REDO location: 31B/FDFFC0D8 > Latest checkpoint's TimeLineID: 1 > Latest checkpoint's full_page_writes: on > Latest checkpoint's NextXID: 0/441709396 > Latest checkpoint's NextOID: 33432042 > Latest checkpoint's NextMultiXactId: 28077 > Latest checkpoint's NextMultiOffset: 57507 > Latest checkpoint's oldestXID: 241800232 > Latest checkpoint's oldestXID's DB: 16573 > Latest checkpoint's oldestActiveXID: 441709396 > Time of latest checkpoint: Mon Jun 2 17:31:05 2014 > Minimum recovery ending location: 31C/4E1C18 > Backup start location: 0/0 > Backup end location: 0/0 > End-of-backup record required: no > Current wal_level setting: hot_standby > Current max_connections setting: 4000 > Current max_prepared_xacts setting: 0 > Current max_locks_per_xact setting: 64 > Maximum data alignment: 8 > Database block size: 8192 > Blocks per segment of large relation: 131072 > WAL block size: 8192 > Bytes per WAL segment: 16777216 > Maximum length of identifiers: 64 > Maximum columns in an index: 32 > Maximum size of a TOAST chunk: 1996 > Date/time type storage: 64-bit integers > Float4 argument passing: by value > Float8 argument passing: by value > I have to redo the base backup and bring it back up again, so the rest of the > stuff is about to be rsycnhed awak, but it has now failed twice today. It > might well, again. > What more should I provide to try to find a solution? > Server 2 -> 1 replication has no issues. > Server 3 -> 2 (replicating server 1) is the one that's failing. > 65Mbps line between them. > PG 9.28 across the board on Centos 6.5 64bit all across. ^ permalink raw reply [nested|flat] 2+ messages in thread
end of thread, other threads:[~2014-06-03 00:35 UTC | newest] Thread overview: 2+ messages (download: mbox mbox.gz follow: Atom feed) -- links below jump to the message on this page -- 2014-06-02 23:49 Failing replication Scott Whitney <scott@journyx.com> 2014-06-03 00:35 ` Scott Whitney <scott@journyx.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions for how to clone and mirror all data and code used for this inbox