Received: from malur.postgresql.org ([2a02:16a8:dc51::56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1ft8Eu-0004HJ-NI for pgsql-admin@arkaria.postgresql.org; Fri, 24 Aug 2018 09:18:45 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1ft8Es-00075i-Nf for pgsql-admin@arkaria.postgresql.org; Fri, 24 Aug 2018 09:18:42 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1ft8Es-00075b-8d for pgsql-admin@lists.postgresql.org; Fri, 24 Aug 2018 09:18:42 +0000 Received: from hermes.gatewaynet.com ([193.92.121.98]) by makus.postgresql.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1ft8En-0001uf-Jl for pgsql-admin@lists.postgresql.org; Fri, 24 Aug 2018 09:18:41 +0000 Received: from smadev.internal.net (pc131.internal.net [10.9.200.131]) by hermes.gatewaynet.com (8.14.3/8.14.3/Debian-9.4) with ESMTP id w7O9IRbs002398 for ; Fri, 24 Aug 2018 12:18:32 +0300 Received: from smadev.internal.net (smadev [10.9.200.131]) by smadev.internal.net (8.15.2/8.15.2) with ESMTP id w7O9IQGV005779 for ; Fri, 24 Aug 2018 12:18:27 +0300 (EEST) (envelope-from achill@matrix.gatewaynet.com) Subject: Re: Logical replication monitoring To: pgsql-admin@lists.postgresql.org References: From: Achilleas Mantzios Message-ID: Date: Fri, 24 Aug 2018 12:18:26 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.6.0 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------9EA4338213270828B93D5DF6" Content-Language: en-US X-Virus-Scanned: clamav-milter 0.98.7 at hermes X-Virus-Status: Clean X-Spam-Status: No, score=-35.8 required=5.0 tests=ALL_TRUSTED, DYN_HAM_CHARTERING6,DYN_HAM_COORD,DYN_HAM_IT,DYN_HAM_IT_PARTNERS, DYN_HAM_NAUTICAL,DYN_HAM_SIGNAT1,HTML_MESSAGE,TW_PG,TW_TN autolearn=unavailable version=3.3.1 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on hermes.gatewaynet.com List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------9EA4338213270828B93D5DF6 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit On 24/08/2018 11:49, AYahorau@ibagroup.eu wrote: > Hello, Hello > > Thank you for the suggestion. > I increased wal_receiver_timeout , wal_sender_timeout parameters and now this error does not occur. > Glad it worked! > I installed tail_n_mail utility, made a simple config started in debug mode. > I am constantly facing the same error: > I had my PITA moments with tail_n_mail myself, it seems it doesn't want to work with %c (session id) in the log_line_prefix. Anyways, some comments : - are you doing anything exotic with your log files? which such file patterns? why do you need to create a new file every second? This is bad for many reasons. - did you specify the same LOG_LINE_PREFIX in your tnm conf file as in postgresql.conf ? In any case, for tnm related problems you better ask the ppl you maintain it. For the time being just make sure that your replication works. > WARNING! Skipping non-existent file "/var/lib/pgsql/pg_log/postgresql.log-2018-08-23_154034" > Too many loops (20161): bailing: > > The configuration file tail_n_mail.conf is quiet standart: > EMAIL: someone@example.com > PGLOG: log > MAILSUBJECT: Acme HOST Postgres errors UNIQUE : NUMBER > INCLUDE: ERROR: > INCLUDE: FATAL: > INCLUDE: PANIC: > FILE1: /var/lib/pgsql/pg_log/postgresql.log-%Y-%m-%d_%H%M%S > LASTFILE1: /var/lib/pgsql/pg_log/postgresql.log-2018-08-23_154034 > > Could you please say is there anything wrong in my configuration or script usage? > > Thank you, > Andrei Yahorau > > > > From: Andrei Yahorau/IBA > To: pgsql-admin@postgresql.org, > Cc: Mikalai Keida/IBA@IBA > Date: 13/08/2018 13:16 > Subject: Re: Logical replication monitoring > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > Hello! > > Thank you for your suggestion. > I  afraid this approach is not suitable for me. As a rule my postgresql log  on subscriber side contains a  bunch of the following entries: > > *ERROR:  terminating logical replication worker due to timeout* > *00000 LOG:  worker process: logical replication worker for subscription 24578 (PID 6217) exited with exit code 1* > * > *How should I handle this situation? * > *As I understand this is quite normal situation. But why is severity for it  an ERROR ? > > I have another assumption. Could you correct me if I am wrong. > I found out in the source code that logical replication worker termination depends on *wal_receiver_timeout *paramer. > So I propose setting *wal_receiver_timeout *to 0. > In this case I think that monitoring of the following views *pg_stat_subscription, pg_publication *and *pg_stat_replication * is enough. > In case if there  is some problem with connection or with replication *pg_stat_replication * will show nothing because wal sender will not be working otherwise it will give some information. > Am I right? Are there any vulnerabilities in this approach ? > > Best regards, > Andrei Yahorau > > > > > From: Andrei Yahorau/IBA > To: pgsql-admin@postgresql.org, > Cc: Mikalai Keida/IBA@IBA > Date: 10/08/2018 13:05 > Subject: Logical replication monitoring > -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > > > > Hello PostgreSQL Community! > > I configured logical replication for PostgreSQL 10.4 on 2 machines, set wal_level to logical, created a publication on master node and created a subscription on standby node according to the > PostgreSQL documentation. > Could you please suggest an approach for replication state monitoring. > > According to my experience the monitoring of *pg_stat_subscription *and *pg_publication, pg_replication_slots *unfortunately is not enough for this aim. Moreover standby database does not prohibit > write operations by default and it can lead to some inconsistency between these databases. > > For example a chain of queries as > *SELECT pg_is_is_recovery()*, > *SELECT * FROM pg_stat_replication* and > *SELECT * FROM pg_stat_wal_receiver* > provide insight into replication state for hot_standby replication. > > So is there a reliable way of replication state monitoring for logical replication? > > Best regards, > Andrei Yahorau -- Achilleas Mantzios IT DEV Lead IT DEPT Dynacom Tankers Mgmt --------------9EA4338213270828B93D5DF6 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit
On 24/08/2018 11:49, AYahorau@ibagroup.eu wrote:
Hello,
Hello

Thank you for the suggestion.
I increased wal_receiver_timeout , wal_sender_timeout parameters and now this error does not occur.

Glad it worked!
I installed tail_n_mail utility, made a simple config started in debug mode.
I am constantly facing the same error:

I had my PITA moments with tail_n_mail myself, it seems it doesn't want to work with %c (session id) in the log_line_prefix. Anyways, some comments :
- are you doing anything exotic with your log files? which such file patterns? why do you need to create a new file every second? This is bad for many reasons.
- did you specify the same LOG_LINE_PREFIX in your tnm conf file as in postgresql.conf ?

In any case, for tnm related problems you better ask the ppl you maintain it. For the time being just make sure that your replication works.
WARNING! Skipping non-existent file "/var/lib/pgsql/pg_log/postgresql.log-2018-08-23_154034"
Too many loops (20161): bailing:

The configuration file tail_n_mail.conf is quiet standart:
EMAIL: someone@example.com
PGLOG: log
MAILSUBJECT: Acme HOST Postgres errors UNIQUE : NUMBER
INCLUDE: ERROR:
INCLUDE: FATAL:
INCLUDE: PANIC:
FILE1: /var/lib/pgsql/pg_log/postgresql.log-%Y-%m-%d_%H%M%S
LASTFILE1: /var/lib/pgsql/pg_log/postgresql.log-2018-08-23_154034

Could you please say is there anything wrong in my configuration or script usage?

Thank you,
Andrei Yahorau



From:        Andrei Yahorau/IBA
To:        pgsql-admin@postgresql.org,
Cc:        Mikalai Keida/IBA@IBA
Date:        13/08/2018 13:16
Subject:        Re: Logical replication monitoring


Hello!

Thank you for your suggestion.
I  afraid this approach is not suitable for me. As a rule my postgresql log  on subscriber side contains a  bunch of the following entries:

ERROR:  terminating logical replication worker due to timeout
00000 LOG:  worker process: logical replication worker for subscription 24578 (PID 6217) exited with exit code 1

How should I handle this situation?
As I understand this is quite normal situation. But why is severity for it  an ERROR ?

I have another assumption. Could you correct me if I am wrong.
I found out in the source code that logical replication worker termination depends on wal_receiver_timeout paramer.
So I propose setting wal_receiver_timeout  to 0.
In this case I think that monitoring of the following views  pg_stat_subscription, pg_publication and pg_stat_replication  is enough.
In case if there  is some problem with connection or with replication pg_stat_replication  will show nothing because wal sender will not be working otherwise it will give some information.
Am I right? Are there any vulnerabilities in this approach ?

Best regards,
Andrei Yahorau




From:        Andrei Yahorau/IBA
To:        pgsql-admin@postgresql.org,
Cc:        Mikalai Keida/IBA@IBA
Date:        10/08/2018 13:05
Subject:        Logical replication monitoring



Hello PostgreSQL Community!

I configured logical replication for PostgreSQL 10.4 on 2 machines, set wal_level to logical, created a publication on master node and created a subscription on standby node according to the PostgreSQL documentation.
Could you please suggest an approach for replication state monitoring.

According to my experience the monitoring of pg_stat_subscription and pg_publication, pg_replication_slots unfortunately is not enough for this aim. Moreover standby database does not prohibit write operations by default and it can lead to some inconsistency between these databases.

For example a chain of queries as
SELECT pg_is_is_recovery(),
SELECT * FROM pg_stat_replication and
SELECT * FROM pg_stat_wal_receiver
provide insight into replication state for hot_standby replication.

So is there a reliable way of replication state monitoring for logical replication?

Best regards,
Andrei Yahorau 


-- 
Achilleas Mantzios
IT DEV Lead
IT DEPT
Dynacom Tankers Mgmt
--------------9EA4338213270828B93D5DF6--