pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedLogging: stderr vs syslog?
9+ messages / 4 participants
[nested] [flat]
* Logging: stderr vs syslog?
@ 2017-08-04 16:26 Don Seiler <don@seiler.us>
2017-08-04 17:11 ` Re: Logging: stderr vs syslog? Scott Mead <scottm@openscg.com>
2017-08-04 19:49 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
0 siblings, 2 replies; 9+ messages in thread
From: Don Seiler @ 2017-08-04 16:26 UTC (permalink / raw)
To: pgsql-admin
I've just inherited a few PostgreSQL DBs, having come from Oracle land. I'm
looking to shore up the logging situation. Right now we use stderr logging
and they get rotated based on size threshold. I'd like for those old logs
to be gzipped so we can keep more on disk rather than current method of
just deleting old logs to free up space. This is mostly on pgsql 9.2 with a
couple of 9.3, but I'm planning to upgrade everything to 9.6.3 when I get
my feet on solid ground.
Couple of question around this:
1. I thought logrotate would be a no-brainer here, but it sounds like I
should then change to use syslog rather than stderr. I've read some caveats
around syslog needing to sync files and potentially slow things down. I'm
wondering if any grizzled production postgres veterans could offer up their
experience.
2. Alternatively I could just keep it going with stderr and have a
separate script run find/gzip on log files beyond a certain mtime
threshold. This would probably be the quickest to implement, but I'd much
rather use logrotate facilities if there are no strong opinions against
using syslog.
Thanks in advance for your time, I'm sure I'll be making a lot of us of
these mailing lists in the not-too-distant future.
Don.
--
Don Seiler
www.seiler.us
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
@ 2017-08-04 17:11 ` Scott Mead <scottm@openscg.com>
2017-08-04 17:40 ` Re: Logging: stderr vs syslog? Korry Douglas <korry.douglas@enterprisedb.com>
1 sibling, 1 reply; 9+ messages in thread
From: Scott Mead @ 2017-08-04 17:11 UTC (permalink / raw)
To: Don Seiler <don@seiler.us>; +Cc: pgsql-admin
On Fri, Aug 4, 2017 at 12:26 PM, Don Seiler <don@seiler.us> wrote:
> I've just inherited a few PostgreSQL DBs, having come from Oracle land.
> I'm looking to shore up the logging situation. Right now we use stderr
> logging and they get rotated based on size threshold. I'd like for those
> old logs to be gzipped so we can keep more on disk rather than current
> method of just deleting old logs to free up space. This is mostly on pgsql
> 9.2 with a couple of 9.3, but I'm planning to upgrade everything to 9.6.3
> when I get my feet on solid ground.
>
> Couple of question around this:
>
> 1. I thought logrotate would be a no-brainer here, but it sounds like
> I should then change to use syslog rather than stderr. I've read some
> caveats around syslog needing to sync files and potentially slow things
> down. I'm wondering if any grizzled production postgres veterans could
> offer up their experience.
> 2. Alternatively I could just keep it going with stderr and have a
> separate script run find/gzip on log files beyond a certain mtime
> threshold. This would probably be the quickest to implement, but I'd much
> rather use logrotate facilities if there are no strong opinions against
> using syslog.
>
>
Personally, I'm a huge fan of syslog + logrotate. Since you're new to PG,
you should have a look at pgBadger (https://github.com/dalibo/pgbadger) to
parse / analyze / summarize the logs for you. You *may* need to set an
appropriate log_line_prefix, but, there's a few examples in the pgBadger
docs.
--Scott
> Thanks in advance for your time, I'm sure I'll be making a lot of us of
> these mailing lists in the not-too-distant future.
>
> Don.
>
> --
> Don Seiler
> www.seiler.us
>
--
--
Scott Mead
Sr. Architect
*OpenSCG <http://openscg.com>*
http://openscg.com
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-04 17:11 ` Re: Logging: stderr vs syslog? Scott Mead <scottm@openscg.com>
@ 2017-08-04 17:40 ` Korry Douglas <korry.douglas@enterprisedb.com>
2017-08-04 18:02 ` Re: Logging: stderr vs syslog? Scott Mead <scottm@openscg.com>
0 siblings, 1 reply; 9+ messages in thread
From: Korry Douglas @ 2017-08-04 17:40 UTC (permalink / raw)
To: Scott Mead <scottm@openscg.com>; +Cc: Don Seiler <don@seiler.us>; pgsql-admin
Just an FYI - Scott is indeed a respected postgresql veteran, but he's
hardly grizzled :-)
-- Korry
On Fri, Aug 4, 2017 at 1:11 PM, Scott Mead <scottm@openscg.com> wrote:
>
>
> On Fri, Aug 4, 2017 at 12:26 PM, Don Seiler <don@seiler.us> wrote:
>
>> I've just inherited a few PostgreSQL DBs, having come from Oracle land.
>> I'm looking to shore up the logging situation. Right now we use stderr
>> logging and they get rotated based on size threshold. I'd like for those
>> old logs to be gzipped so we can keep more on disk rather than current
>> method of just deleting old logs to free up space. This is mostly on pgsql
>> 9.2 with a couple of 9.3, but I'm planning to upgrade everything to 9.6.3
>> when I get my feet on solid ground.
>>
>> Couple of question around this:
>>
>> 1. I thought logrotate would be a no-brainer here, but it sounds like
>> I should then change to use syslog rather than stderr. I've read some
>> caveats around syslog needing to sync files and potentially slow things
>> down. I'm wondering if any grizzled production postgres veterans could
>> offer up their experience.
>> 2. Alternatively I could just keep it going with stderr and have a
>> separate script run find/gzip on log files beyond a certain mtime
>> threshold. This would probably be the quickest to implement, but I'd much
>> rather use logrotate facilities if there are no strong opinions against
>> using syslog.
>>
>>
> Personally, I'm a huge fan of syslog + logrotate. Since you're new to PG,
> you should have a look at pgBadger (https://github.com/dalibo/pgbadger) to
> parse / analyze / summarize the logs for you. You *may* need to set an
> appropriate log_line_prefix, but, there's a few examples in the pgBadger
> docs.
>
> --Scott
>
>
>
>> Thanks in advance for your time, I'm sure I'll be making a lot of us of
>> these mailing lists in the not-too-distant future.
>>
>> Don.
>>
>> --
>> Don Seiler
>> www.seiler.us
>>
>
>
>
> --
> --
> Scott Mead
> Sr. Architect
> *OpenSCG <http://openscg.com>*
> http://openscg.com
>
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-04 17:11 ` Re: Logging: stderr vs syslog? Scott Mead <scottm@openscg.com>
2017-08-04 17:40 ` Re: Logging: stderr vs syslog? Korry Douglas <korry.douglas@enterprisedb.com>
@ 2017-08-04 18:02 ` Scott Mead <scottm@openscg.com>
0 siblings, 0 replies; 9+ messages in thread
From: Scott Mead @ 2017-08-04 18:02 UTC (permalink / raw)
To: Korry Douglas <korry.douglas@enterprisedb.com>; +Cc: Don Seiler <don@seiler.us>; pgsql-admin
On Fri, Aug 4, 2017 at 1:40 PM, Korry Douglas <
korry.douglas@enterprisedb.com> wrote:
> Just an FYI - Scott is indeed a respected postgresql veteran, but he's
> hardly grizzled :-)
>
That just depends on who you talk to :-)
>
> -- Korry
>
> On Fri, Aug 4, 2017 at 1:11 PM, Scott Mead <scottm@openscg.com> wrote:
>
>>
>>
>> On Fri, Aug 4, 2017 at 12:26 PM, Don Seiler <don@seiler.us> wrote:
>>
>>> I've just inherited a few PostgreSQL DBs, having come from Oracle land.
>>> I'm looking to shore up the logging situation. Right now we use stderr
>>> logging and they get rotated based on size threshold. I'd like for those
>>> old logs to be gzipped so we can keep more on disk rather than current
>>> method of just deleting old logs to free up space. This is mostly on pgsql
>>> 9.2 with a couple of 9.3, but I'm planning to upgrade everything to 9.6.3
>>> when I get my feet on solid ground.
>>>
>>> Couple of question around this:
>>>
>>> 1. I thought logrotate would be a no-brainer here, but it sounds
>>> like I should then change to use syslog rather than stderr. I've read some
>>> caveats around syslog needing to sync files and potentially slow things
>>> down. I'm wondering if any grizzled production postgres veterans could
>>> offer up their experience.
>>> 2. Alternatively I could just keep it going with stderr and have a
>>> separate script run find/gzip on log files beyond a certain mtime
>>> threshold. This would probably be the quickest to implement, but I'd much
>>> rather use logrotate facilities if there are no strong opinions against
>>> using syslog.
>>>
>>>
>> Personally, I'm a huge fan of syslog + logrotate. Since you're new to
>> PG, you should have a look at pgBadger (https://github.com/dalibo/pgb
>> adger) to parse / analyze / summarize the logs for you. You *may* need
>> to set an appropriate log_line_prefix, but, there's a few examples in the
>> pgBadger docs.
>>
>> --Scott
>>
>>
>>
>>> Thanks in advance for your time, I'm sure I'll be making a lot of us of
>>> these mailing lists in the not-too-distant future.
>>>
>>> Don.
>>>
>>> --
>>> Don Seiler
>>> www.seiler.us
>>>
>>
>>
>>
>> --
>> --
>> Scott Mead
>> Sr. Architect
>> *OpenSCG <http://openscg.com>*
>> http://openscg.com
>>
>
>
--
--
Scott Mead
Sr. Architect
*OpenSCG <http://openscg.com>*
http://openscg.com
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
@ 2017-08-04 19:49 ` Evan Rempel <erempel@uvic.ca>
2017-08-05 05:22 ` Re: Logging: stderr vs syslog? Don Seiler <don@seiler.us>
1 sibling, 1 reply; 9+ messages in thread
From: Evan Rempel @ 2017-08-04 19:49 UTC (permalink / raw)
To: pgsql-admin
I have been a systems administrator of PostgreSQL since version 7.0. I
am also a primary logging architect for a 4000+ device network
infrastructure and SEIM alerting infrastructure architect.
Syslog is by far the best approach to logging. With the right product it
is way FASTER than stderr and there are a ton of tools to parse,
analyze, view and report on syslog streams.
There is one caveat with PostgreSQL version <= 9.5 and that is that
syslog messages are wrapped at approx 80 characters which makes parsing
and error detection problematic. pgBadger may address this limitation by
unwrapping such logs messages whereas generic log parsing engines do not
have any specialized knowledge about how PostgreSQL lines might be
wrapped. PostgreSQL version <= 9.5 this is a compile time only option
but starting in 9.6 this is a runtime configuration directive.
There are two strong reasons for using syslog.
1. In a well architected logging solution the syslog process on the host
will also send the log messages to a central log server. This means that
if the database server is compromised leading to an untrusted set of log
files, there is a trusted copy of the logs on another server.
2. When running a high availability or clustered database all of the
logs can be aggregated to a central log server which places all of the
logs from all of the database servers into one easy to
read/parse/process location.
I hope this provides some rationale for using syslog.
Evan.
On 08/04/2017 09:26 AM, Don Seiler wrote:
> I've just inherited a few PostgreSQL DBs, having come from Oracle
> land. I'm looking to shore up the logging situation. Right now we use
> stderr logging and they get rotated based on size threshold. I'd like
> for those old logs to be gzipped so we can keep more on disk rather
> than current method of just deleting old logs to free up space. This
> is mostly on pgsql 9.2 with a couple of 9.3, but I'm planning to
> upgrade everything to 9.6.3 when I get my feet on solid ground.
>
> Couple of question around this:
>
> 1. I thought logrotate would be a no-brainer here, but it sounds like
> I should then change to use syslog rather than stderr. I've read
> some caveats around syslog needing to sync files and potentially
> slow things down. I'm wondering if any grizzled production
> postgres veterans could offer up their experience.
> 2. Alternatively I could just keep it going with stderr and have a
> separate script run find/gzip on log files beyond a certain mtime
> threshold. This would probably be the quickest to implement, but
> I'd much rather use logrotate facilities if there are no strong
> opinions against using syslog.
>
> Thanks in advance for your time, I'm sure I'll be making a lot of us
> of these mailing lists in the not-too-distant future.
>
> Don.
>
> --
> Don Seiler
> www.seiler.us <http://www.seiler.us;
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-04 19:49 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
@ 2017-08-05 05:22 ` Don Seiler <don@seiler.us>
2017-08-06 00:31 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
0 siblings, 1 reply; 9+ messages in thread
From: Don Seiler @ 2017-08-05 05:22 UTC (permalink / raw)
To: Evan Rempel <erempel@uvic.ca>; +Cc: pgsql-admin
Thanks everyone for your feedback. I've got a test setup running with
syslog and logrotate and it's working well.
Is there any concern with the syslog file sync where it's suggested to put
a dash before the file name to remove the sync requirement?
Don.
On Aug 4, 2017 2:51 PM, "Evan Rempel" <erempel@uvic.ca> wrote:
> I have been a systems administrator of PostgreSQL since version 7.0. I am
> also a primary logging architect for a 4000+ device network infrastructure
> and SEIM alerting infrastructure architect.
>
> Syslog is by far the best approach to logging. With the right product it
> is way FASTER than stderr and there are a ton of tools to parse, analyze,
> view and report on syslog streams.
>
> There is one caveat with PostgreSQL version <= 9.5 and that is that syslog
> messages are wrapped at approx 80 characters which makes parsing and error
> detection problematic. pgBadger may address this limitation by unwrapping
> such logs messages whereas generic log parsing engines do not have any
> specialized knowledge about how PostgreSQL lines might be wrapped.
> PostgreSQL version <= 9.5 this is a compile time only option but starting
> in 9.6 this is a runtime configuration directive.
>
> There are two strong reasons for using syslog.
>
> 1. In a well architected logging solution the syslog process on the host
> will also send the log messages to a central log server. This means that if
> the database server is compromised leading to an untrusted set of log
> files, there is a trusted copy of the logs on another server.
>
> 2. When running a high availability or clustered database all of the logs
> can be aggregated to a central log server which places all of the logs from
> all of the database servers into one easy to read/parse/process location.
>
> I hope this provides some rationale for using syslog.
>
> Evan.
>
> On 08/04/2017 09:26 AM, Don Seiler wrote:
>
> I've just inherited a few PostgreSQL DBs, having come from Oracle land.
> I'm looking to shore up the logging situation. Right now we use stderr
> logging and they get rotated based on size threshold. I'd like for those
> old logs to be gzipped so we can keep more on disk rather than current
> method of just deleting old logs to free up space. This is mostly on pgsql
> 9.2 with a couple of 9.3, but I'm planning to upgrade everything to 9.6.3
> when I get my feet on solid ground.
>
> Couple of question around this:
>
> 1. I thought logrotate would be a no-brainer here, but it sounds like
> I should then change to use syslog rather than stderr. I've read some
> caveats around syslog needing to sync files and potentially slow things
> down. I'm wondering if any grizzled production postgres veterans could
> offer up their experience.
> 2. Alternatively I could just keep it going with stderr and have a
> separate script run find/gzip on log files beyond a certain mtime
> threshold. This would probably be the quickest to implement, but I'd much
> rather use logrotate facilities if there are no strong opinions against
> using syslog.
>
> Thanks in advance for your time, I'm sure I'll be making a lot of us of
> these mailing lists in the not-too-distant future.
>
> Don.
>
> --
> Don Seiler
> www.seiler.us
>
>
>
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-04 19:49 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
2017-08-05 05:22 ` Re: Logging: stderr vs syslog? Don Seiler <don@seiler.us>
@ 2017-08-06 00:31 ` Evan Rempel <erempel@uvic.ca>
2017-08-07 00:29 ` Re: Logging: stderr vs syslog? Don Seiler <don@seiler.us>
0 siblings, 1 reply; 9+ messages in thread
From: Evan Rempel @ 2017-08-06 00:31 UTC (permalink / raw)
To: ; +Cc: pgsql-admin
Tou are now asking questions that are more appropriate for the specific
syslog product you are using.
rsyslog, syslogd, syslog-ng etc.
In general you need to ask yourself what the ramifications of loosing
the most recent locally stored syslog messages. If you are forwarding
you syslog message to a central syslog server, or perhaps event two
syslog servers then the loss of some local syslog messages in the event
of a local host crash is not very significant.
If you can live with the local loss then adding a "no sync" option to
your syslog configuration is a good idea.
For further details please follow up with your syslog specific support
channel.
Evan.
On 08/04/2017 10:22 PM, Don Seiler wrote:
> Thanks everyone for your feedback. I've got a test setup running with
> syslog and logrotate and it's working well.
>
> Is there any concern with the syslog file sync where it's suggested to
> put a dash before the file name to remove the sync requirement?
>
> Don.
>
> On Aug 4, 2017 2:51 PM, "Evan Rempel" <erempel@uvic.ca
> <mailto:erempel@uvic.ca>> wrote:
>
> I have been a systems administrator of PostgreSQL since version
> 7.0. I am also a primary logging architect for a 4000+ device
> network infrastructure and SEIM alerting infrastructure architect.
>
> Syslog is by far the best approach to logging. With the right
> product it is way FASTER than stderr and there are a ton of tools
> to parse, analyze, view and report on syslog streams.
>
> There is one caveat with PostgreSQL version <= 9.5 and that is
> that syslog messages are wrapped at approx 80 characters which
> makes parsing and error detection problematic. pgBadger may
> address this limitation by unwrapping such logs messages whereas
> generic log parsing engines do not have any specialized knowledge
> about how PostgreSQL lines might be wrapped. PostgreSQL version <=
> 9.5 this is a compile time only option but starting in 9.6 this is
> a runtime configuration directive.
>
> There are two strong reasons for using syslog.
>
> 1. In a well architected logging solution the syslog process on
> the host will also send the log messages to a central log server.
> This means that if the database server is compromised leading to
> an untrusted set of log files, there is a trusted copy of the logs
> on another server.
>
> 2. When running a high availability or clustered database all of
> the logs can be aggregated to a central log server which places
> all of the logs from all of the database servers into one easy to
> read/parse/process location.
>
> I hope this provides some rationale for using syslog.
>
> Evan.
>
> On 08/04/2017 09:26 AM, Don Seiler wrote:
>> I've just inherited a few PostgreSQL DBs, having come from Oracle
>> land. I'm looking to shore up the logging situation. Right now we
>> use stderr logging and they get rotated based on size threshold.
>> I'd like for those old logs to be gzipped so we can keep more on
>> disk rather than current method of just deleting old logs to free
>> up space. This is mostly on pgsql 9.2 with a couple of 9.3, but
>> I'm planning to upgrade everything to 9.6.3 when I get my feet on
>> solid ground.
>>
>> Couple of question around this:
>>
>> 1. I thought logrotate would be a no-brainer here, but it sounds
>> like I should then change to use syslog rather than stderr.
>> I've read some caveats around syslog needing to sync files
>> and potentially slow things down. I'm wondering if any
>> grizzled production postgres veterans could offer up their
>> experience.
>> 2. Alternatively I could just keep it going with stderr and have
>> a separate script run find/gzip on log files beyond a certain
>> mtime threshold. This would probably be the quickest to
>> implement, but I'd much rather use logrotate facilities if
>> there are no strong opinions against using syslog.
>>
>> Thanks in advance for your time, I'm sure I'll be making a lot of
>> us of these mailing lists in the not-too-distant future.
>>
>> Don.
>>
>> --
>> Don Seiler
>> www.seiler.us <http://www.seiler.us;
>
>
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-04 19:49 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
2017-08-05 05:22 ` Re: Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-06 00:31 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
@ 2017-08-07 00:29 ` Don Seiler <don@seiler.us>
2017-08-07 01:35 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
0 siblings, 1 reply; 9+ messages in thread
From: Don Seiler @ 2017-08-07 00:29 UTC (permalink / raw)
To: Evan Rempel <erempel@uvic.ca>; +Cc: pgsql-admin
rsyslog on CentOS7, no central server, just local log files.
I appreciate your suggestion of going to syslog support but I'm more
interested in how those in the pgsql-admin community handle these concerns.
I'm not asking for syslog support.
Don.
On Sat, Aug 5, 2017 at 7:31 PM, Evan Rempel <erempel@uvic.ca> wrote:
> Tou are now asking questions that are more appropriate for the specific
> syslog product you are using.
> rsyslog, syslogd, syslog-ng etc.
>
> In general you need to ask yourself what the ramifications of loosing the
> most recent locally stored syslog messages. If you are forwarding you
> syslog message to a central syslog server, or perhaps event two syslog
> servers then the loss of some local syslog messages in the event of a local
> host crash is not very significant.
>
> If you can live with the local loss then adding a "no sync" option to your
> syslog configuration is a good idea.
>
> For further details please follow up with your syslog specific support
> channel.
>
> Evan.
>
>
> On 08/04/2017 10:22 PM, Don Seiler wrote:
>
> Thanks everyone for your feedback. I've got a test setup running with
> syslog and logrotate and it's working well.
>
> Is there any concern with the syslog file sync where it's suggested to put
> a dash before the file name to remove the sync requirement?
>
> Don.
>
> On Aug 4, 2017 2:51 PM, "Evan Rempel" <erempel@uvic.ca> wrote:
>
>> I have been a systems administrator of PostgreSQL since version 7.0. I am
>> also a primary logging architect for a 4000+ device network infrastructure
>> and SEIM alerting infrastructure architect.
>>
>> Syslog is by far the best approach to logging. With the right product it
>> is way FASTER than stderr and there are a ton of tools to parse, analyze,
>> view and report on syslog streams.
>>
>> There is one caveat with PostgreSQL version <= 9.5 and that is that
>> syslog messages are wrapped at approx 80 characters which makes parsing and
>> error detection problematic. pgBadger may address this limitation by
>> unwrapping such logs messages whereas generic log parsing engines do not
>> have any specialized knowledge about how PostgreSQL lines might be wrapped.
>> PostgreSQL version <= 9.5 this is a compile time only option but starting
>> in 9.6 this is a runtime configuration directive.
>>
>> There are two strong reasons for using syslog.
>>
>> 1. In a well architected logging solution the syslog process on the host
>> will also send the log messages to a central log server. This means that if
>> the database server is compromised leading to an untrusted set of log
>> files, there is a trusted copy of the logs on another server.
>>
>> 2. When running a high availability or clustered database all of the logs
>> can be aggregated to a central log server which places all of the logs from
>> all of the database servers into one easy to read/parse/process location.
>>
>> I hope this provides some rationale for using syslog.
>>
>> Evan.
>>
>> On 08/04/2017 09:26 AM, Don Seiler wrote:
>>
>> I've just inherited a few PostgreSQL DBs, having come from Oracle land.
>> I'm looking to shore up the logging situation. Right now we use stderr
>> logging and they get rotated based on size threshold. I'd like for those
>> old logs to be gzipped so we can keep more on disk rather than current
>> method of just deleting old logs to free up space. This is mostly on pgsql
>> 9.2 with a couple of 9.3, but I'm planning to upgrade everything to 9.6.3
>> when I get my feet on solid ground.
>>
>> Couple of question around this:
>>
>> 1. I thought logrotate would be a no-brainer here, but it sounds like
>> I should then change to use syslog rather than stderr. I've read some
>> caveats around syslog needing to sync files and potentially slow things
>> down. I'm wondering if any grizzled production postgres veterans could
>> offer up their experience.
>> 2. Alternatively I could just keep it going with stderr and have a
>> separate script run find/gzip on log files beyond a certain mtime
>> threshold. This would probably be the quickest to implement, but I'd much
>> rather use logrotate facilities if there are no strong opinions against
>> using syslog.
>>
>> Thanks in advance for your time, I'm sure I'll be making a lot of us of
>> these mailing lists in the not-too-distant future.
>>
>> Don.
>>
>> --
>> Don Seiler
>> www.seiler.us
>>
>>
>>
>
--
Don Seiler
www.seiler.us
^ permalink raw reply [nested|flat] 9+ messages in thread
* Re: Logging: stderr vs syslog?
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-04 19:49 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
2017-08-05 05:22 ` Re: Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-06 00:31 ` Re: Logging: stderr vs syslog? Evan Rempel <erempel@uvic.ca>
2017-08-07 00:29 ` Re: Logging: stderr vs syslog? Don Seiler <don@seiler.us>
@ 2017-08-07 01:35 ` Evan Rempel <erempel@uvic.ca>
0 siblings, 0 replies; 9+ messages in thread
From: Evan Rempel @ 2017-08-07 01:35 UTC (permalink / raw)
To: ; +Cc: pgsql-admin
It all depends on how comfortable you are with loosing some of the log
messages in the event of a server crash or syslog daemon crash.
One of my primary roles is logging with regards to security and
auditing, so I am not comfortable with loosing ANY messages. For me, if
you are not copying log messages to one or more central servers and are
only storing log messages to local log files I would recommend that you
do NOT remove the sync option for syslog.
On the other hand we run some databases that contain only public
information that is transient in nature. Security and auditing are not a
concern in these environments so we could tolerate some message loss if
the server were to crash. The likelihood is low and the loss of some log
lines is not a significant concern. In my opinion disabling the sync
feature in this environment would be acceptable.
You also need to consider that log messages are produced after a program
completes an action, so there is always a small window where the action
is completed but the log message is not yet produced by the application.
If the server crashes at that point you have effectively lost one log
message regardless of the sync setting of your logging infrastructure.
There is no right or wrong answer. It is a judgement call that you will
have to make for your use case.
If any PostgreSQL developers have read this far, perhaps you can add
details on where in the transaction process the log messages are
produced. Are the log messages produced immediately prior to the commit
acknowledgement or after the commit? I think the race condition exists
either way. Either the logs indicate that the action was committed when
it was not, or the commit is completed but the log message was not produced.
I hope I haven't made things more confusing for you.
Evan.
On 08/06/2017 05:29 PM, Don Seiler wrote:
> rsyslog on CentOS7, no central server, just local log files.
>
> I appreciate your suggestion of going to syslog support but I'm more
> interested in how those in the pgsql-admin community handle these
> concerns. I'm not asking for syslog support.
>
> Don.
>
> On Sat, Aug 5, 2017 at 7:31 PM, Evan Rempel <erempel@uvic.ca
> <mailto:erempel@uvic.ca>> wrote:
>
> Tou are now asking questions that are more appropriate for the
> specific syslog product you are using.
> rsyslog, syslogd, syslog-ng etc.
>
> In general you need to ask yourself what the ramifications of
> loosing the most recent locally stored syslog messages. If you are
> forwarding you syslog message to a central syslog server, or
> perhaps event two syslog servers then the loss of some local
> syslog messages in the event of a local host crash is not very
> significant.
>
> If you can live with the local loss then adding a "no sync" option
> to your syslog configuration is a good idea.
>
> For further details please follow up with your syslog specific
> support channel.
>
> Evan.
>
>
> On 08/04/2017 10:22 PM, Don Seiler wrote:
>> Thanks everyone for your feedback. I've got a test setup running
>> with syslog and logrotate and it's working well.
>>
>> Is there any concern with the syslog file sync where it's
>> suggested to put a dash before the file name to remove the sync
>> requirement?
>>
>> Don.
>>
>> On Aug 4, 2017 2:51 PM, "Evan Rempel" <erempel@uvic.ca
>> <mailto:erempel@uvic.ca>> wrote:
>>
>> I have been a systems administrator of PostgreSQL since
>> version 7.0. I am also a primary logging architect for a
>> 4000+ device network infrastructure and SEIM alerting
>> infrastructure architect.
>>
>> Syslog is by far the best approach to logging. With the right
>> product it is way FASTER than stderr and there are a ton of
>> tools to parse, analyze, view and report on syslog streams.
>>
>> There is one caveat with PostgreSQL version <= 9.5 and that
>> is that syslog messages are wrapped at approx 80 characters
>> which makes parsing and error detection problematic. pgBadger
>> may address this limitation by unwrapping such logs messages
>> whereas generic log parsing engines do not have any
>> specialized knowledge about how PostgreSQL lines might be
>> wrapped. PostgreSQL version <= 9.5 this is a compile time
>> only option but starting in 9.6 this is a runtime
>> configuration directive.
>>
>> There are two strong reasons for using syslog.
>>
>> 1. In a well architected logging solution the syslog process
>> on the host will also send the log messages to a central log
>> server. This means that if the database server is compromised
>> leading to an untrusted set of log files, there is a trusted
>> copy of the logs on another server.
>>
>> 2. When running a high availability or clustered database all
>> of the logs can be aggregated to a central log server which
>> places all of the logs from all of the database servers into
>> one easy to read/parse/process location.
>>
>> I hope this provides some rationale for using syslog.
>>
>> Evan.
>>
>> On 08/04/2017 09:26 AM, Don Seiler wrote:
>>> I've just inherited a few PostgreSQL DBs, having come from
>>> Oracle land. I'm looking to shore up the logging situation.
>>> Right now we use stderr logging and they get rotated based
>>> on size threshold. I'd like for those old logs to be gzipped
>>> so we can keep more on disk rather than current method of
>>> just deleting old logs to free up space. This is mostly on
>>> pgsql 9.2 with a couple of 9.3, but I'm planning to upgrade
>>> everything to 9.6.3 when I get my feet on solid ground.
>>>
>>> Couple of question around this:
>>>
>>> 1. I thought logrotate would be a no-brainer here, but it
>>> sounds like I should then change to use syslog rather
>>> than stderr. I've read some caveats around syslog
>>> needing to sync files and potentially slow things down.
>>> I'm wondering if any grizzled production postgres
>>> veterans could offer up their experience.
>>> 2. Alternatively I could just keep it going with stderr and
>>> have a separate script run find/gzip on log files beyond
>>> a certain mtime threshold. This would probably be the
>>> quickest to implement, but I'd much rather use logrotate
>>> facilities if there are no strong opinions against using
>>> syslog.
>>>
>>> Thanks in advance for your time, I'm sure I'll be making a
>>> lot of us of these mailing lists in the not-too-distant future.
>>>
>>> Don.
>>>
>>> --
>>> Don Seiler
>>> www.seiler.us <http://www.seiler.us;
>>
>>
>
>
>
>
> --
> Don Seiler
> www.seiler.us <http://www.seiler.us;
^ permalink raw reply [nested|flat] 9+ messages in thread
end of thread, other threads:[~2017-08-07 01:35 UTC | newest]
Thread overview: 9+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2017-08-04 16:26 Logging: stderr vs syslog? Don Seiler <don@seiler.us>
2017-08-04 17:11 ` Scott Mead <scottm@openscg.com>
2017-08-04 17:40 ` Korry Douglas <korry.douglas@enterprisedb.com>
2017-08-04 18:02 ` Scott Mead <scottm@openscg.com>
2017-08-04 19:49 ` Evan Rempel <erempel@uvic.ca>
2017-08-05 05:22 ` Don Seiler <don@seiler.us>
2017-08-06 00:31 ` Evan Rempel <erempel@uvic.ca>
2017-08-07 00:29 ` Don Seiler <don@seiler.us>
2017-08-07 01:35 ` Evan Rempel <erempel@uvic.ca>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox