Received: from makus.postgresql.org (makus.postgresql.org [98.129.198.125]) by mail.postgresql.org (Postfix) with ESMTP id 5CA721345932 for ; Mon, 11 Jun 2012 00:35:36 -0300 (ADT) Received: from ec2-23-23-181-247.compute-1.amazonaws.com ([23.23.181.247] helo=nebula.troocorp.com) by makus.postgresql.org with esmtp (Exim 4.72) (envelope-from ) id 1SdvPf-0000K8-SX for pgsql-general@postgresql.org; Mon, 11 Jun 2012 03:35:35 +0000 Received: from localhost (localhost [127.0.0.1]) by nebula.troocorp.com (Postfix) with ESMTP id 8600F776002 for ; Mon, 11 Jun 2012 03:34:52 +0000 (UTC) X-Virus-Scanned: amavisd-new at troocorp.com Received: from nebula.troocorp.com ([127.0.0.1]) by localhost (nebula.troocorp.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CH8wEykln9a6 for ; Mon, 11 Jun 2012 03:34:48 +0000 (UTC) Received: from nebula.troocorp.com (nebula.troocorp.com [10.114.218.85]) by nebula.troocorp.com (Postfix) with ESMTP id EFBC7776001 for ; Mon, 11 Jun 2012 03:34:47 +0000 (UTC) Date: Mon, 11 Jun 2012 03:34:47 +0000 (UTC) From: Benson Jin To: pgsql-general@postgresql.org Message-ID: <843420470.254405.1339385687611.JavaMail.root@troo.com> In-Reply-To: <676924118.254401.1339385164386.JavaMail.root@troo.com> Subject: Segmentation Fault MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_Part_254404_399341335.1339385687610" X-Originating-IP: [99.245.249.129] X-Mailer: Zimbra 7.2.0_GA_2669 (ZimbraWebClient - GC19 (Win)/7.2.0_GA_2669) X-Pg-Spam-Score: 0.1 (/) X-Archive-Number: 201206/201 X-Sequence-Number: 187981 ------=_Part_254404_399341335.1339385687610 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Hi All, We are having a problem with our streaming replication read only node. It has crashed a few times with a couple of different reasons, mostly "segmentation fault". The latest log are listed below: 2012-05-30 23:56:37.385 UTC::: LOG: server process (PID 19476) was terminated by signal 11: Segmentation fault 2012-05-30 23:56:37.385 UTC::: LOG: terminating any other active server processes 2012-05-30 23:56:37.385 UTC:10.43.6.61:webmaster:panorama WARNING: terminating connection because of crash of another server process 2012-05-30 23:56:37.385 UTC:10.43.6.61:webmaster:panorama DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2012-05-30 23:56:37.385 UTC:10.43.6.61:webmaster:panorama HINT: In a moment you should be able to reconnect to the database and repeat your command. 2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama WARNING: terminating connection because of crash of another server process 2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama HINT: In a moment you should be able to reconnect to the database and repeat your command. 2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama WARNING: terminating connection because of crash of another server process 2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama DETAIL: The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory. 2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama HINT: In a moment you should be able to reconnect to the database and repeat your command. 2012-05-30 23:56:37.575 UTC:10.43.6.81:webmaster:panorama FATAL: the database system is in recovery mode Our setup: 2x physical server - Dell PE R815, 64GB ECC RAM, 2 CPUs (12 cores each), storing pgsql data on SAN backed volumes. CentOS 5.6 PostgreSQL 9.0.8, compiled *without* int64 datetime. Both servers are identically configured (or at least as much as we could ensure) One is master, another is streaming read-only node. The master runs two instances of postgreSQL, where the slave runs 5 instances of postgreSQL. 2 out of 5 are streaming replication from the master, rest 3 are streaming replication from other DB nodes. Those 2 instances serves clients as Read Only. The master node never had any crash so far. However, the 2 instances on slave have crashed 3 times by now, 1 time on one readonly instance, twice on another readonly instance. Above log was generated from one of the instances. All three crashes happened when the database was doing vacuuming. we automatically purge some data every night, and run vacuum analyze right after that... Our the CPU load is generally on 40%-60% mark. I have run a complete set of hardware diagnostics on the slave , with no faulty hardware detected. Can someone kindly shed some lights on me? I am not sure where to look into at this point.... Cheers, Bo Jin Operating/IT Manager Troo Corporation [ www.troo.com ] 43 Auriga Drive, Suite 102, Ottawa, ON K2E 7Y8 Ph: +1 877.702.8766 x156 Fax: +1 855.726.8766 ------=_Part_254404_399341335.1339385687610 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: quoted-printable <= div style=3D'font-family: arial,helvetica,sans-serif; font-size: 12pt; colo= r: #666666'>Hi All,

We are having a problem with our streaming replication read = only node. It has crashed a few times with a couple of different reasons, m= ostly "segmentation fault". The latest log are listed below:

2012-05-30 23:56:3= 7.385 UTC::: LOG:  server process (PID 19476) was terminated by signal= 11: Segmentation fault
2012-05-30 23:56:3= 7.385 UTC::: LOG:  terminating any other active server processes
=
2012-05-30 23:56:37.385 UTC:10.43.6.61:webmaste= r:panorama WARNING:  terminating connection because of crash of anothe= r server process
2012-05-30 23:56:37.385 U= TC:10.43.6.61:webmaster:panorama DETAIL:  The postmaster has commanded= this server process to roll back the current transaction and exit, because= another server process exited abnormally and possibly corrupted shared mem= ory.
2012-05-30 23:56:37.385 UTC:10.43.6.6= 1:webmaster:panorama HINT:  In a moment you should be able to reconnec= t to the database and repeat your command.
2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama WARNING:  t= erminating connection because of crash of another server process
2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:pan= orama DETAIL:  The postmaster has commanded this server process to rol= l back the current transaction and exit, because another server process exi= ted abnormally and possibly corrupted shared memory.
2012-05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama HINT: =  In a moment you should be able to reconnect to the database and repea= t your command.
2012-05-30 23:56:37.385 UT= C:10.43.6.81:webmaster:panorama WARNING:  terminating connection becau= se of crash of another server process
2012= -05-30 23:56:37.385 UTC:10.43.6.81:webmaster:panorama DETAIL:  The pos= tmaster has commanded this server process to roll back the current transact= ion and exit, because another server process exited abnormally and possibly= corrupted shared memory.
2012-05-30 23:56= :37.385 UTC:10.43.6.81:webmaster:panorama HINT:  In a moment you shoul= d be able to reconnect to the database and repeat your command.
2012-05-30 23:56:37.575 UTC:10.43.6.81:webmaster:pano= rama FATAL:  the database system is in recovery mode


Our setup:
2x physical = server - Dell PE R815, 64GB ECC RAM, 2 CPUs (12 cores each),  storing = pgsql data on SAN backed volumes.
CentOS 5= .6
PostgreSQL 9.0.8, compiled *without* in= t64 datetime.
Both servers are identically= configured (or at least as much as we could ensure)
One is master, another is streaming read-only node.
The master runs two instances of postgreSQL, where the= slave runs 5 instances of postgreSQL. 2 out of 5 are streaming replication= from the master, rest 3 are streaming replication from other DB nodes. Tho= se 2 instances serves clients as Read Only. The master node never had any c= rash so far. However, the 2 instances on slave have crashed 3 times by now,= 1 time on one readonly instance, twice on another readonly instance. Above= log was generated from one of the instances.

All three crashes happened when t= he database was doing vacuuming. we automatically purge some data every nig= ht, and run vacuum analyze right after that... Our the CPU load is generall= y on 40%-60% mark.

I have run= a complete set of hardware diagnostics on the slave, with no faulty ha= rdware detected. Can someone kindly shed some lights on me? I am not sure w= here to look into at this point....


=
Cheers,  
&nbs= p;

Bo Jin 

Operating/IT Manager Troo Corporation [www.troo.com]
4= 3 Auriga Drive, Suite 102, Ottawa, ON K2E 7Y8
Ph: +1 877.702.8766 x156 =
Fax: +1 855.726.8766

------=_Part_254404_399341335.1339385687610--