pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedBackground fsck
17+ messages / 7 participants
[nested] [flat]
* Background fsck
@ 2011-04-06 22:33 Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 16:16 ` Re: Background fsck Greg Smith <greg@2ndQuadrant.com>
0 siblings, 2 replies; 17+ messages in thread
From: Ireneusz Pluta @ 2011-04-06 22:33 UTC (permalink / raw)
To: pgsql-performance
Hello,
I saw some recommendations from people on the net not to use background fsck when running PostgreSQL
on FreeBSD. As I recall, these opinions were just thoughts of people which they shared with the
community, following their bad experience caused by using background fsck. So, not coming any deeper
with underatanding why not, I use that as a clear recommendation for myself and keep background fsck
turned off on all my machines, regardless how much faster a server could come up after a crash.
But waiting so much time (like now) during foreground fsck of a large data filesystem after unclean
shutdown, makes me to come to this group to ask whether I really need to avoid background fsck on a
PostgreSQL machine? Could I hear your opinions?
Thanks
Irek.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
@ 2011-04-06 22:48 ` Scott Marlowe <scott.marlowe@gmail.com>
2011-04-06 22:59 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
1 sibling, 2 replies; 17+ messages in thread
From: Scott Marlowe @ 2011-04-06 22:48 UTC (permalink / raw)
To: Ireneusz Pluta <ipluta@wp.pl>; +Cc: pgsql-performance
On Wed, Apr 6, 2011 at 4:33 PM, Ireneusz Pluta <ipluta@wp.pl> wrote:
> Hello,
>
> I saw some recommendations from people on the net not to use background fsck
> when running PostgreSQL on FreeBSD. As I recall, these opinions were just
> thoughts of people which they shared with the community, following their bad
> experience caused by using background fsck. So, not coming any deeper with
> underatanding why not, I use that as a clear recommendation for myself and
> keep background fsck turned off on all my machines, regardless how much
> faster a server could come up after a crash.
>
> But waiting so much time (like now) during foreground fsck of a large data
> filesystem after unclean shutdown, makes me to come to this group to ask
> whether I really need to avoid background fsck on a PostgreSQL machine?
> Could I hear your opinions?
Shouldn't a journaling file system just come back up almost immediately?
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
@ 2011-04-06 22:59 ` Ireneusz Pluta <ipluta@wp.pl>
1 sibling, 0 replies; 17+ messages in thread
From: Ireneusz Pluta @ 2011-04-06 22:59 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-performance
But waiting so much time (like now) during foreground fsck of a large data
>> filesystem after unclean shutdown, makes me to come to this group to ask
>> whether I really need to avoid background fsck on a PostgreSQL machine?
>> Could I hear your opinions?
> Shouldn't a journaling file system just come back up almost immediately?
>
it's ufs2 with softupdates in my case. That's why I am asking abot background fsck.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
@ 2011-04-07 13:31 ` Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
1 sibling, 1 reply; 17+ messages in thread
From: Ivan Voras @ 2011-04-07 13:31 UTC (permalink / raw)
To: pgsql-performance
On 07/04/2011 00:48, Scott Marlowe wrote:
> On Wed, Apr 6, 2011 at 4:33 PM, Ireneusz Pluta<ipluta@wp.pl> wrote:
>> Hello,
>>
>> I saw some recommendations from people on the net not to use background fsck
>> when running PostgreSQL on FreeBSD. As I recall, these opinions were just
>> thoughts of people which they shared with the community, following their bad
>> experience caused by using background fsck. So, not coming any deeper with
>> underatanding why not, I use that as a clear recommendation for myself and
>> keep background fsck turned off on all my machines, regardless how much
>> faster a server could come up after a crash.
>>
>> But waiting so much time (like now) during foreground fsck of a large data
>> filesystem after unclean shutdown, makes me to come to this group to ask
>> whether I really need to avoid background fsck on a PostgreSQL machine?
>> Could I hear your opinions?
AFAIK, the reason why background fsck has been discouraged when used
with databases is because it uses disk bandwidth which may be needed by
the application. If you are not IO saturated, then there is no
particular reason why you should avoid it.
> Shouldn't a journaling file system just come back up almost immediately?
It's a tradeoff; UFS does not use journalling (at least not likely in
the version the OP is talking about) but it uses "soft updates". It
brings most of the benefits of journalling, including "instant up" after
a crash without the double-write overheads (for comparison, see some
PostgreSQL benchmarks showing that unjournaled ext2 can be faster than
journaled ext3, since PostgreSQL has its own form of journaling - the
WAL). The downside is that fsck needs to be run occasionally to cleanup
non-critical dangling references on the file system - thus the
"background fsck" mode in FreeBSD.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
@ 2011-04-07 14:01 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
0 siblings, 1 reply; 17+ messages in thread
From: Achilleas Mantzios @ 2011-04-07 14:01 UTC (permalink / raw)
To: pgsql-performance
Στις Thursday 07 April 2011 16:31:50 ο/η Ivan Voras έγραψε:
> On 07/04/2011 00:48, Scott Marlowe wrote:
> > On Wed, Apr 6, 2011 at 4:33 PM, Ireneusz Pluta<ipluta@wp.pl> wrote:
> >> Hello,
> >>
> >> I saw some recommendations from people on the net not to use background fsck
> >> when running PostgreSQL on FreeBSD. As I recall, these opinions were just
> >> thoughts of people which they shared with the community, following their bad
> >> experience caused by using background fsck. So, not coming any deeper with
> >> underatanding why not, I use that as a clear recommendation for myself and
> >> keep background fsck turned off on all my machines, regardless how much
> >> faster a server could come up after a crash.
> >>
> >> But waiting so much time (like now) during foreground fsck of a large data
> >> filesystem after unclean shutdown, makes me to come to this group to ask
> >> whether I really need to avoid background fsck on a PostgreSQL machine?
> >> Could I hear your opinions?
>
> AFAIK, the reason why background fsck has been discouraged when used
> with databases is because it uses disk bandwidth which may be needed by
> the application. If you are not IO saturated, then there is no
> particular reason why you should avoid it.
>
> > Shouldn't a journaling file system just come back up almost immediately?
>
> It's a tradeoff; UFS does not use journalling (at least not likely in
> the version the OP is talking about) but it uses "soft updates". It
> brings most of the benefits of journalling, including "instant up" after
> a crash without the double-write overheads (for comparison, see some
> PostgreSQL benchmarks showing that unjournaled ext2 can be faster than
> journaled ext3, since PostgreSQL has its own form of journaling - the
> WAL). The downside is that fsck needs to be run occasionally to cleanup
> non-critical dangling references on the file system - thus the
> "background fsck" mode in FreeBSD.
>
I agree with Ivan. In the case of background fsck ,there is absolutely no reason to
postpone using postgresql, more than doing the same for any other service in the system.
In anyway, having FreeBSD to fsck, (background or not) should not happen. And the problem
becomes bigger when cheap SATA drives will cheat about their write cache being flushed to the disk.
So in the common case with cheap hardware, it is wise to have a UPS connected and being monitored
by the system.
--
Achilleas Mantzios
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
@ 2011-04-08 05:55 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 10:21 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
0 siblings, 2 replies; 17+ messages in thread
From: Ireneusz Pluta @ 2011-04-08 05:55 UTC (permalink / raw)
To: pgsql-performance
Achilleas Mantzios wrote:
>
> In anyway, having FreeBSD to fsck, (background or not) should not happen. And the problem
> becomes bigger when cheap SATA drives will cheat about their write cache being flushed to the disk.
> So in the common case with cheap hardware, it is wise to have a UPS connected and being monitored
> by the system.
>
It's not lack of UPS. Power issues are taken care of here. It's a buggy 3ware controller which hangs
the machine ocassionally and the only way to have it come back is to power cycle, hard reset is not
enough.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
@ 2011-04-08 06:19 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
1 sibling, 1 reply; 17+ messages in thread
From: Achilleas Mantzios @ 2011-04-08 06:19 UTC (permalink / raw)
To: pgsql-performance
Στις Friday 08 April 2011 08:55:51 ο/η Ireneusz Pluta έγραψε:
> Achilleas Mantzios wrote:
> >
> > In anyway, having FreeBSD to fsck, (background or not) should not happen. And the problem
> > becomes bigger when cheap SATA drives will cheat about their write cache being flushed to the disk.
> > So in the common case with cheap hardware, it is wise to have a UPS connected and being monitored
> > by the system.
> >
>
> It's not lack of UPS. Power issues are taken care of here. It's a buggy 3ware controller which hangs
> the machine ocassionally and the only way to have it come back is to power cycle, hard reset is not
> enough.
What has happened to me (as Greg mentioned) is that repeatedly interrupted background fscks (having the system
crash while background fsck was executing) might result in a seriously damaged fs.
Add to this, the possible overhead by rebuilding software raid (gmirror) at the same time,
and the situation becomes more complicated.
So its better to replace/fix/remove this buggy controller, before anything else.
>
>
--
Achilleas Mantzios
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
@ 2011-04-08 06:22 ` Scott Marlowe <scott.marlowe@gmail.com>
2011-04-08 11:52 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 11:53 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
0 siblings, 2 replies; 17+ messages in thread
From: Scott Marlowe @ 2011-04-08 06:22 UTC (permalink / raw)
To: Achilleas Mantzios <achill@matrix.gatewaynet.com>; +Cc: pgsql-performance
On Fri, Apr 8, 2011 at 12:19 AM, Achilleas Mantzios
<achill@matrix.gatewaynet.com> wrote:
> Óôéò Friday 08 April 2011 08:55:51 ï/ç Ireneusz Pluta Ýãñáøå:
>> Achilleas Mantzios wrote:
>> >
>> > In anyway, having FreeBSD to fsck, (background or not) should not happen. And the problem
>> > becomes bigger when cheap SATA drives will cheat about their write cache being flushed to the disk.
>> > So in the common case with cheap hardware, it is wise to have a UPS connected and being monitored
>> > by the system.
>> >
>>
>> It's not lack of UPS. Power issues are taken care of here. It's a buggy 3ware controller which hangs
>> the machine ocassionally and the only way to have it come back is to power cycle, hard reset is not
>> enough.
>
> What has happened to me (as Greg mentioned) is that repeatedly interrupted background fscks (having the system
> crash while background fsck was executing) might result in a seriously damaged fs.
> Add to this, the possible overhead by rebuilding software raid (gmirror) at the same time,
> and the situation becomes more complicated.
>
> So its better to replace/fix/remove this buggy controller, before anything else.
If I may ask, how often does it crash? And have you tried updating
the firmware of the controller and / or the driver in the OS?
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
@ 2011-04-08 11:52 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 14:32 ` Re: Background fsck Jochen Erwied <jochen@pgsql-performance.erwied.eu>
2011-04-18 21:21 ` Re: Background fsck Pierre C <lists@peufeu.com>
1 sibling, 2 replies; 17+ messages in thread
From: Ireneusz Pluta @ 2011-04-08 11:52 UTC (permalink / raw)
To: pgsql-performance
Scott Marlowe wrote:
> If I may ask, how often does it crash? And have you tried updating
> the firmware of the controller and / or the driver in the OS?
>
It happens once per two or three months, or so, taking the average. The firmware is beta as of
January this year, advised to use by their technical support.
Kinda off topic here, but as asked, take a look at the details:
/c0 Driver Version = 3.60.04.006
/c0 Model = 9650SE-16ML
/c0 Available Memory = 224MB
/c0 Firmware Version = FE9X 4.10.00.016
/c0 Bios Version = BE9X 4.08.00.002
/c0 Boot Loader Version = BL9X 3.08.00.001
There is a newer beta firmware available from LSI support page, but changelog does not indicate
anything which might be related to this problem.
OS is 6.2-RELEASE FreeBSD
The driver also should be the newest for this platform.
What's more, this is already a new controller. It replaced the previous one because of exactly the
same persisting problem. I think tech support people not knowing a solution just buy some time for
them and say "flash this beta firmware maybe it helps" or "replace your hardware".
The controller always hangs with the following:
Send AEN (code, time): 0031h, 04/06/2011 21:56:45
Synchronize host/controller time
(EC:0x31, SK=0x00, ASC=0x00, ASCQ=0x00, SEV=04, Type=0x71)
Assert:0 from Command Task
File:cacheSegMgr.cpp Line:290
and this is usually at the time of IO peaks, when dumps get transferred to another system.
My general plan for now is to migrate all services from this machine to the new ones and refresh it
completely for less critical services. But it is not a task for just a few days so the failures have
their chances to happen. While bearing this, I wanted to check if I could ease my life a little with
background checks.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-08 11:52 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
@ 2011-04-08 14:32 ` Jochen Erwied <jochen@pgsql-performance.erwied.eu>
1 sibling, 0 replies; 17+ messages in thread
From: Jochen Erwied @ 2011-04-08 14:32 UTC (permalink / raw)
To: Ireneusz Pluta <ipluta@wp.pl>; +Cc: pgsql-performance
Friday, April 8, 2011, 1:52:03 PM you wrote:
> Scott Marlowe wrote:
>> If I may ask, how often does it crash? And have you tried updating
>> the firmware of the controller and / or the driver in the OS?
>>
> It happens once per two or three months, or so, taking the average. The firmware is beta as of
> January this year, advised to use by their technical support.
Do you run any software to periodically check the array status?
Or are there any other regular tasks involving 'tw_cli'-calls?
I had this effect while trying to examine the SMART-status of the attached
drives on a 9690-8E, leading to spurious controller resets due to timeouts,
also under high load.
Disabling the task solved the problem, and no further resets occured.
--
Jochen Erwied | home: jochen@erwied.eu +49-208-38800-18, FAX: -19
Sauerbruchstr. 17 | work: joe@mbs-software.de +49-2151-7294-24, FAX: -50
D-45470 Muelheim | mobile: jochen.erwied@vodafone.de +49-173-5404164
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-08 11:52 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
@ 2011-04-18 21:21 ` Pierre C <lists@peufeu.com>
1 sibling, 0 replies; 17+ messages in thread
From: Pierre C @ 2011-04-18 21:21 UTC (permalink / raw)
To: pgsql-performance@postgresql.org, "Ireneusz Pluta" <ipluta@wp.pl>
> What's more, this is already a new controller. It replaced the previous
> one because of exactly the same persisting problem. I think tech support
> people not knowing a solution just buy some time for them and say "flash
> this beta firmware maybe it helps" or "replace your hardware".
We had a problem like this on a server a few years ago on the job... The
machine randomly crashed once a month. XFS coped alright until, one day,
it threw the towel, and the poor maintenance guys needed to run xfsrepair.
Needless to say, the machine crashed again while xfsrepair was running
concurrently on all filesystems. All filesystems were then completely
trashed... That convinced the boss maybe something was wrong and a new box
was rushed in... Then a few tens of terabytes of backup restoration ...
zzzzzz ....
It turned out it was a faulty SCSI cable.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
@ 2011-04-08 11:53 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 12:14 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
1 sibling, 1 reply; 17+ messages in thread
From: Ireneusz Pluta @ 2011-04-08 11:53 UTC (permalink / raw)
To: pgsql-performance
Scott Marlowe wrote:
> If I may ask, how often does it crash? And have you tried updating
> the firmware of the controller and / or the driver in the OS?
>
It happens once per two or three months, or so, taking the average. The firmware is beta as of
January this year, advised to use by their technical support.
Kinda off topic here, but as asked, take a look at the details:
/c0 Driver Version = 3.60.04.006
/c0 Model = 9650SE-16ML
/c0 Available Memory = 224MB
/c0 Firmware Version = FE9X 4.10.00.016
/c0 Bios Version = BE9X 4.08.00.002
/c0 Boot Loader Version = BL9X 3.08.00.001
There is a newer beta firmware available from LSI support page, but changelog does not indicate
anything which might be related to this problem.
OS is 6.2-RELEASE FreeBSD
The driver also should be the newest for this platform.
What's more, this is already a new controller. It replaced the previous one because of exactly the
same persisting problem. I think tech support people not knowing a solution just buy some time for
them and say "flash this beta firmware maybe it helps" or "replace your hardware".
The controller always hangs with the following:
Send AEN (code, time): 0031h, 04/06/2011 21:56:45
Synchronize host/controller time
(EC:0x31, SK=0x00, ASC=0x00, ASCQ=0x00, SEV=04, Type=0x71)
Assert:0 from Command Task
File:cacheSegMgr.cpp Line:290
and this is usually at the time of IO peaks, when dumps get transferred to another system.
My general plan for now is to migrate all services from this machine to the new ones and refresh it
completely for use with less critical services. But it is not a task for just a few days so the
failures have their chances to happen. While bearing this, I wanted to check if I could ease my life
a little with background checks.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-08 11:53 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
@ 2011-04-08 12:14 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 16:12 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
0 siblings, 1 reply; 17+ messages in thread
From: Achilleas Mantzios @ 2011-04-08 12:14 UTC (permalink / raw)
To: pgsql-performance
Στις Friday 08 April 2011 14:53:58 ο/η Ireneusz Pluta έγραψε:
>
> My general plan for now is to migrate all services from this machine to the new ones and refresh it
> completely for use with less critical services. But it is not a task for just a few days so the
That's a pain. Migrating from 7.1 to 8.2 was a pain for me as well.
But OTOH, you should upgrade to FreeBSD 8.2 since it is a production system.
Imagine your 3ware card was ok, but the driver has the problem.
How are you gonna show up in the FreeBSD-* mailing list when you are still on 6.2?
BTW, when you make the final transition to 8.2, DO NOT upgrade in place, make a new system
and migrate the data. Or just upgrade system in place but pkg_deinstall all your ports before the upgrade.
portupgrade will not make it through.
> failures have their chances to happen. While bearing this, I wanted to check if I could ease my life
> a little with background checks.
>
>
>
--
Achilleas Mantzios
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-08 11:53 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 12:14 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
@ 2011-04-08 16:12 ` Ireneusz Pluta <ipluta@wp.pl>
0 siblings, 0 replies; 17+ messages in thread
From: Ireneusz Pluta @ 2011-04-08 16:12 UTC (permalink / raw)
To: pgsql-performance
Achilleas Mantzios wrote:
> How are you gonna show up in the FreeBSD-* mailing list when you are still on 6.2?
Psst! - I came just here. Don't tell them.
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Re: Background fsck Scott Marlowe <scott.marlowe@gmail.com>
2011-04-07 13:31 ` Re: Background fsck Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Re: Background fsck Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
@ 2011-04-08 10:21 ` Ivan Voras <ivoras@freebsd.org>
1 sibling, 0 replies; 17+ messages in thread
From: Ivan Voras @ 2011-04-08 10:21 UTC (permalink / raw)
To: pgsql-performance
On 08/04/2011 07:55, Ireneusz Pluta wrote:
> Achilleas Mantzios wrote:
>>
>> In anyway, having FreeBSD to fsck, (background or not) should not
>> happen. And the problem
>> becomes bigger when cheap SATA drives will cheat about their write
>> cache being flushed to the disk.
>> So in the common case with cheap hardware, it is wise to have a UPS
>> connected and being monitored
>> by the system.
>>
>
> It's not lack of UPS. Power issues are taken care of here. It's a buggy
> 3ware controller which hangs the machine ocassionally and the only way
> to have it come back is to power cycle, hard reset is not enough.
So just to summarize your position: you think it is ok to run a database
on a buggy disk controller but are afraid of problems with normal OS
utilities? :) You are of course free to do whatever you want but it
seems odd :)
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
@ 2011-04-07 16:16 ` Greg Smith <greg@2ndQuadrant.com>
2011-04-08 09:34 ` Re: Background fsck Ireneusz Pluta <ipluta@wp.pl>
1 sibling, 1 reply; 17+ messages in thread
From: Greg Smith @ 2011-04-07 16:16 UTC (permalink / raw)
To: pgsql-performance
On 04/06/2011 06:33 PM, Ireneusz Pluta wrote:
> I saw some recommendations from people on the net not to use
> background fsck when running PostgreSQL on FreeBSD. As I recall, these
> opinions were just thoughts of people which they shared with the
> community, following their bad experience caused by using background fsck.
Presumably you're talking about reports like these two:
http://blog.e-shell.org/266
http://lists.freebsd.org/pipermail/freebsd-current/2007-July/074773.html
> But waiting so much time (like now) during foreground fsck of a large
> data filesystem after unclean shutdown, makes me to come to this group
> to ask whether I really need to avoid background fsck on a PostgreSQL
> machine?
The soft update code used in FreeBSD makes sure that there's no damage
to the filesystem that PostgreSQL can't recover from. Once the WAL is
replayed after a crash, the database is consistent. The main purpose of
the background fsck is to find "orphaned" space, things that the
filesystem incorrectly remembers the state of in regards to whether it
was allocated and used. In theory, there's no reason that can't happen
in the background, concurrent with normal database activity.
In practice, background fsck is such an infrequently used piece of code
that it's developed a bit of a reputation for being buggier than
average. It's really hard to test it, filesystem code is complicated,
and the sort of inconsistent data you get after a hard crash is often
really surprising. I wouldn't be too concerned about the database
integrity, but there is a small risk that background fsck will run into
something unexpected and panic. And that's a problem you're much less
likely to hit using the more stable regular fsck code; thus the
recommendations by some to avoid it.
--
Greg Smith 2ndQuadrant US greg@2ndQuadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
"PostgreSQL 9.0 High Performance": http://www.2ndQuadrant.com/books
^ permalink raw reply [nested|flat] 17+ messages in thread
* Re: Background fsck
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-07 16:16 ` Re: Background fsck Greg Smith <greg@2ndQuadrant.com>
@ 2011-04-08 09:34 ` Ireneusz Pluta <ipluta@wp.pl>
0 siblings, 0 replies; 17+ messages in thread
From: Ireneusz Pluta @ 2011-04-08 09:34 UTC (permalink / raw)
To: pgsql-performance
Greg Smith wrote:
> The soft update code used in FreeBSD makes sure that there's no damage to the filesystem that
> PostgreSQL can't recover from. Once the WAL is replayed after a crash, the database is
> consistent. The main purpose of the background fsck is to find "orphaned" space, things that the
> filesystem incorrectly remembers the state of in regards to whether it was allocated and used. In
> theory, there's no reason that can't happen in the background, concurrent with normal database
> activity.
>
> In practice, background fsck is such an infrequently used piece of code that it's developed a bit
> of a reputation for being buggier than average. It's really hard to test it, filesystem code is
> complicated, and the sort of inconsistent data you get after a hard crash is often really
> surprising. I wouldn't be too concerned about the database integrity, but there is a small risk
> that background fsck will run into something unexpected and panic. And that's a problem you're
> much less likely to hit using the more stable regular fsck code; thus the recommendations by some
> to avoid it.
>
Thank you all for your responses.
Greg, given your opinion, and these few raised issues found on the net, I think I better stay with
background fsck disabled.
What I was primarily concerned about, was long time waiting in front of console, looking at lazy
fsck messages and nervously confirming that disk LEDs are still blinking. It's even harder with
remote KVM, where LED's view is not available. But my personal comfort is not a priority, anyway, so
I let foreground fsck doing its job for as much time as it needs.
As I said in my another response, the problem initially comes from the machine hanging and having to
be manually power cycled. There is already a significant downtinme before the recycle has a chance
to happen. So yet another fourty minutes of fsck does not matter too much from the point of view of
service availability.
fsck runtime duration could be shortened if I used smaller inode density for the filesystem. I think
that makes much sense for a filesystem fully decicated to a postgres data cluster, specifically if I
have not so many but large tables, which I rather do.
The system in question has:
df -hi | grep -E 'base|ifree'
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/da1p3 3.0T 1.7T 1.0T 63% 485k 392M 0% /pg/base
(will I ever have even tens of millions of tables?)
I reserved less inodes in a newer, bigger system:
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/mfid0p8 12T 4.8T 6.0T 45% 217k 49M 0% /pg/base
or even less in yet newer one:
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/mfid0p1 12T 3.6T 7.4T 33% 202k 3.4M 6% /pg/base
(ups, maybe too aggressive here?)
When I forced a power drop on these two other systems, to check how they survive, fsck duration on
them was substantially less.
In the inode density context, let me ask you yet another question. Does tuning it in this way have
any other, good or bad, significant impact on system performance?
Irek.
^ permalink raw reply [nested|flat] 17+ messages in thread
end of thread, other threads:[~2011-04-18 21:21 UTC | newest]
Thread overview: 17+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2011-04-06 22:33 Background fsck Ireneusz Pluta <ipluta@wp.pl>
2011-04-06 22:48 ` Scott Marlowe <scott.marlowe@gmail.com>
2011-04-06 22:59 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-07 13:31 ` Ivan Voras <ivoras@freebsd.org>
2011-04-07 14:01 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 05:55 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 06:19 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 06:22 ` Scott Marlowe <scott.marlowe@gmail.com>
2011-04-08 11:52 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 14:32 ` Jochen Erwied <jochen@pgsql-performance.erwied.eu>
2011-04-18 21:21 ` Pierre C <lists@peufeu.com>
2011-04-08 11:53 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 12:14 ` Achilleas Mantzios <achill@matrix.gatewaynet.com>
2011-04-08 16:12 ` Ireneusz Pluta <ipluta@wp.pl>
2011-04-08 10:21 ` Ivan Voras <ivoras@freebsd.org>
2011-04-07 16:16 ` Greg Smith <greg@2ndQuadrant.com>
2011-04-08 09:34 ` Ireneusz Pluta <ipluta@wp.pl>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox