pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedOut of file-descriptors message
18+ messages / 6 participants
[nested] [flat]
* Out of file-descriptors message
@ 2006-11-29 20:51 John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 18+ messages in thread
From: John Allgood @ 2006-11-29 20:51 UTC (permalink / raw)
To: pgsql -admin <@postgresql.org>; +Cc: Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>
Hello All
We are having an issue and it appears to be with file descriptors.
We are running PostgreSQL 7.4.13 on RHEL 3.0 box. This system is running
Redhat Cluster Suite with each system running multiple postmasters. The
error has occurred on both boxes and we can't find any consistencies on
what is causing the problem. I don't see this message in dmesg or any
other log file. It only appears in this single database. There are other
databases running on the same host but from different postmasters and I
don't see the same message. A couple of things to add is that this has
happened on another machine from a different database. When this
problem has been reoccurring for a period of several days the server
get fenced by the other machine. Don't know if related but seems to
happen at the time when the descriptors error is occurring. Here is a
snippet from the log of one of the databases.
2006-11-29 14:57:51 [3823] LOG: out of file descriptors: Too many open
files; release and retry
2006-11-29 14:58:01 [3823] LOG: out of file descriptors: Too many open
files; release and retry
2006-11-29 15:00:26 [3823] LOG: out of file descriptors: Too many open
files; release and retry
--
**John Allgood**
/*/Senior Systems Administrator/*/
**Turbo Logistics - Division of Ozburn-Hessey Logistics**
**2251 Jesse Jewell Pky. NE**
**Gainesville, GA 30507******
**tel (678) 989-3051 fax (770) 531-7878**
**john@turbocorp.com** <mailto:john@turbocorp.com>
**www.turbocorp.com** <http://www.turbocorp.com;
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
@ 2006-11-29 21:32 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
0 siblings, 1 reply; 18+ messages in thread
From: Tom Lane @ 2006-11-29 21:32 UTC (permalink / raw)
To: John Allgood <john@turbocorp.com>; +Cc: pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>
John Allgood <john@turbocorp.com> writes:
> 2006-11-29 14:57:51 [3823] LOG: out of file descriptors: Too many open
> files; release and retry
Consider reducing PG's max_files_per_process setting. Postgres itself
will usually not have a serious problem when you've run the kernel out
of file descriptors, but everything else on the machine will :-(
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-11-29 22:03 ` John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
0 siblings, 1 reply; 18+ messages in thread
From: John Allgood @ 2006-11-29 22:03 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>
Hey Tom
I assume that if it is the kernel running out of descriptors that I
would get the messages in dmesg. This message only appears in the log
file for that database.
Thanks
Tom Lane wrote:
> John Allgood <john@turbocorp.com> writes:
>
>> 2006-11-29 14:57:51 [3823] LOG: out of file descriptors: Too many open
>> files; release and retry
>>
>
> Consider reducing PG's max_files_per_process setting. Postgres itself
> will usually not have a serious problem when you've run the kernel out
> of file descriptors, but everything else on the machine will :-(
>
> regards, tom lane
>
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
@ 2006-11-29 22:25 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 13:23 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
0 siblings, 2 replies; 18+ messages in thread
From: Alvaro Herrera @ 2006-11-29 22:25 UTC (permalink / raw)
To: John Allgood <jallgood@the-allgoods.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>
John Allgood wrote:
> Hey Tom
>
> I assume that if it is the kernel running out of descriptors that I
> would get the messages in dmesg. This message only appears in the log
> file for that database.
Yeah, the point is that you have the max_files_per_process setting
higher than what the kernel likes. So decrease it, and Postgres will
adjust itself to use less file descriptors by closing and reopening
files as needed.
--
Alvaro Herrera http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 13:23 ` John Allgood <jallgood@the-allgoods.net>
1 sibling, 0 replies; 18+ messages in thread
From: John Allgood @ 2006-11-30 13:23 UTC (permalink / raw)
To: pgsql-admin
Alvaro Herrera wrote:
> John Allgood wrote:
>
>> Hey Tom
>>
>> I assume that if it is the kernel running out of descriptors that I
>> would get the messages in dmesg. This message only appears in the log
>> file for that database.
>>
>
> Yeah, the point is that you have the max_files_per_process setting
> higher than what the kernel likes. So decrease it, and Postgres will
> adjust itself to use less file descriptors by closing and reopening
> files as needed.
>
>
I really can't see that is possible file-max is set to 838860. I believe
it is based on the amount of ram on the machine and we have 8GB. ulimit
-a reports you can have 1024 open files. I thought maybe we were running
out of descriptors for the postgres user but that should not bring down
the machine. Thoughts anyone.
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 14:12 ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:22 ` Re: Out of file-descriptors message Bradley Kieser <brad@kieser.net>
2006-11-30 14:27 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 15:10 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 3 replies; 18+ messages in thread
From: Geoffrey @ 2006-11-30 14:12 UTC (permalink / raw)
To: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Alvaro Herrera wrote:
> John Allgood wrote:
>> Hey Tom
>>
>> I assume that if it is the kernel running out of descriptors that I
>> would get the messages in dmesg. This message only appears in the log
>> file for that database.
>
> Yeah, the point is that you have the max_files_per_process setting
> higher than what the kernel likes. So decrease it, and Postgres will
> adjust itself to use less file descriptors by closing and reopening
> files as needed.
I work with the original poster and wanted to make sure the problem here
is clear. The 'out of file descriptors' message is coming from
Postgresql, not the kernel. Thus, it doesn't make sense to me that the
max_files_per_process setting is too high. I would think we need to
increase it so that Postgresql will stop generating these errors.
--
Until later, Geoffrey
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:22 ` Bradley Kieser <brad@kieser.net>
2006-11-30 14:26 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2 siblings, 1 reply; 18+ messages in thread
From: Bradley Kieser @ 2006-11-30 14:22 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Hmm, not entirely true. You may well only see the error in the process
itself.
Geoffrey wrote:
> Alvaro Herrera wrote:
>> John Allgood wrote:
>>> Hey Tom
>>>
>>> I assume that if it is the kernel running out of descriptors that I
>>> would get the messages in dmesg. This message only appears in the log
>>> file for that database.
>>
>> Yeah, the point is that you have the max_files_per_process setting
>> higher than what the kernel likes. So decrease it, and Postgres will
>> adjust itself to use less file descriptors by closing and reopening
>> files as needed.
>
> I work with the original poster and wanted to make sure the problem
> here is clear. The 'out of file descriptors' message is coming from
> Postgresql, not the kernel. Thus, it doesn't make sense to me that
> the max_files_per_process setting is too high. I would think we need
> to increase it so that Postgresql will stop generating these errors.
>
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:22 ` Re: Out of file-descriptors message Bradley Kieser <brad@kieser.net>
@ 2006-11-30 14:26 ` Geoffrey <esoteric@3times25.net>
0 siblings, 0 replies; 18+ messages in thread
From: Geoffrey @ 2006-11-30 14:26 UTC (permalink / raw)
To: Bradley Kieser <brad@kieser.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Bradley Kieser wrote:
> Hmm, not entirely true. You may well only see the error in the process
> itself.
You're telling me that the kernel could be running out of file
descriptors and I wouldn't see a message regarding this in the kernel
logs? I don't believe that is possible.
>
>
> Geoffrey wrote:
>> Alvaro Herrera wrote:
>>> John Allgood wrote:
>>>> Hey Tom
>>>>
>>>> I assume that if it is the kernel running out of descriptors that I
>>>> would get the messages in dmesg. This message only appears in the log
>>>> file for that database.
>>>
>>> Yeah, the point is that you have the max_files_per_process setting
>>> higher than what the kernel likes. So decrease it, and Postgres will
>>> adjust itself to use less file descriptors by closing and reopening
>>> files as needed.
>>
>> I work with the original poster and wanted to make sure the problem
>> here is clear. The 'out of file descriptors' message is coming from
>> Postgresql, not the kernel. Thus, it doesn't make sense to me that
>> the max_files_per_process setting is too high. I would think we need
>> to increase it so that Postgresql will stop generating these errors.
>>
>
--
Until later, Geoffrey
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:27 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2 siblings, 1 reply; 18+ messages in thread
From: Alvaro Herrera @ 2006-11-30 14:27 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Geoffrey wrote:
> Alvaro Herrera wrote:
> >John Allgood wrote:
> >>Hey Tom
> >>
> >> I assume that if it is the kernel running out of descriptors that I
> >>would get the messages in dmesg. This message only appears in the log
> >>file for that database.
> >
> >Yeah, the point is that you have the max_files_per_process setting
> >higher than what the kernel likes. So decrease it, and Postgres will
> >adjust itself to use less file descriptors by closing and reopening
> >files as needed.
>
> I work with the original poster and wanted to make sure the problem here
> is clear.
Yes, that was understood from the beginning.
> The 'out of file descriptors' message is coming from
> Postgresql, not the kernel. Thus, it doesn't make sense to me that the
> max_files_per_process setting is too high. I would think we need to
> increase it so that Postgresql will stop generating these errors.
No, you need to lower it so that Postgres doesn't _try_ to use as many
file descriptors. Read this again:
> > So decrease it, and Postgres will
> >adjust itself to use less file descriptors by closing and reopening
> >files as needed.
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:27 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 14:46 ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:51 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:52 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
0 siblings, 2 replies; 18+ messages in thread
From: Geoffrey @ 2006-11-30 14:46 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Alvaro Herrera wrote:
> Geoffrey wrote:
>> Alvaro Herrera wrote:
>>> John Allgood wrote:
>>>> Hey Tom
>>>>
>>>> I assume that if it is the kernel running out of descriptors that I
>>>> would get the messages in dmesg. This message only appears in the log
>>>> file for that database.
>>> Yeah, the point is that you have the max_files_per_process setting
>>> higher than what the kernel likes. So decrease it, and Postgres will
>>> adjust itself to use less file descriptors by closing and reopening
>>> files as needed.
>> I work with the original poster and wanted to make sure the problem here
>> is clear.
>
> Yes, that was understood from the beginning.
>
>> The 'out of file descriptors' message is coming from
>> Postgresql, not the kernel. Thus, it doesn't make sense to me that the
>> max_files_per_process setting is too high. I would think we need to
>> increase it so that Postgresql will stop generating these errors.
>
> No, you need to lower it so that Postgres doesn't _try_ to use as many
> file descriptors. Read this again:
>
>>> So decrease it, and Postgres will
>>> adjust itself to use less file descriptors by closing and reopening
>>> files as needed.
Okay, I'm just not getting it. Postgres complains that it is out of
file descriptors. The kernel is not complaining about any such issues.
So I should lower the max_files_per_process
value and this will rid us of the 'out of file descriptors' error?
Is it because the max_files_per_process is greater then the number of
file descriptors that are alloted to Postgres by the kernel?
--
Until later, Geoffrey
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:27 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:51 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:54 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
1 sibling, 1 reply; 18+ messages in thread
From: Alvaro Herrera @ 2006-11-30 14:51 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Geoffrey wrote:
> Okay, I'm just not getting it. Postgres complains that it is out of
> file descriptors. The kernel is not complaining about any such issues.
>
>
> So I should lower the max_files_per_process
> value and this will rid us of the 'out of file descriptors' error?
>
> Is it because the max_files_per_process is greater then the number of
> file descriptors that are alloted to Postgres by the kernel?
Yes. Which is kinda weird, because Postgres actually tests the number
when it starts, so that if you set the number too high, it will decrease
it according to what the kernel allows.
Maybe the test is newer than the version you are running -- what was it,
again?
--
Alvaro Herrera http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:27 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:51 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 14:54 ` Geoffrey <esoteric@3times25.net>
2006-11-30 15:30 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 18+ messages in thread
From: Geoffrey @ 2006-11-30 14:54 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Alvaro Herrera wrote:
> Geoffrey wrote:
>
>> Okay, I'm just not getting it. Postgres complains that it is out of
>> file descriptors. The kernel is not complaining about any such issues.
>>
>>
>> So I should lower the max_files_per_process
>> value and this will rid us of the 'out of file descriptors' error?
>>
>> Is it because the max_files_per_process is greater then the number of
>> file descriptors that are alloted to Postgres by the kernel?
>
> Yes. Which is kinda weird, because Postgres actually tests the number
> when it starts, so that if you set the number too high, it will decrease
> it according to what the kernel allows.
>
> Maybe the test is newer than the version you are running -- what was it,
> again?
>
7.4.13
--
Until later, Geoffrey
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:27 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:51 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:54 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 15:30 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-30 15:40 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
0 siblings, 1 reply; 18+ messages in thread
From: Tom Lane @ 2006-11-30 15:30 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Geoffrey <esoteric@3times25.net> writes:
> Alvaro Herrera wrote:
>> Yes. Which is kinda weird, because Postgres actually tests the number
>> when it starts, so that if you set the number too high, it will decrease
>> it according to what the kernel allows.
>>
>> Maybe the test is newer than the version you are running -- what was it,
>> again?
>>
> 7.4.13
Hmph ... 7.4.13 does have the same set_max_safe_fds() logic as HEAD.
So this shouldn't be happening really, no matter what the relative
values of max_files_per_process and ulimit -n are.
Explanations I can think of:
* ulimit -n decreased after process startup (is this even possible?)
* something is leaking file descriptors that are outside fd.c's control,
such that eventually there are more than NUM_RESERVED_FDS of 'em.
Theory B seems a lot more plausible. It'd be interesting to look at
"lsof" output for one of the backend processes that is emitting this
message, to see if we can identify what's getting leaked.
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:27 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:51 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:54 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 15:30 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-11-30 15:40 ` Geoffrey <esoteric@3times25.net>
0 siblings, 0 replies; 18+ messages in thread
From: Geoffrey @ 2006-11-30 15:40 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Tom Lane wrote:
> Geoffrey <esoteric@3times25.net> writes:
>> Alvaro Herrera wrote:
>>> Yes. Which is kinda weird, because Postgres actually tests the number
>>> when it starts, so that if you set the number too high, it will decrease
>>> it according to what the kernel allows.
>>>
>>> Maybe the test is newer than the version you are running -- what was it,
>>> again?
>>>
>> 7.4.13
>
> Hmph ... 7.4.13 does have the same set_max_safe_fds() logic as HEAD.
> So this shouldn't be happening really, no matter what the relative
> values of max_files_per_process and ulimit -n are.
>
> Explanations I can think of:
>
> * ulimit -n decreased after process startup (is this even possible?)
>
> * something is leaking file descriptors that are outside fd.c's control,
> such that eventually there are more than NUM_RESERVED_FDS of 'em.
>
> Theory B seems a lot more plausible. It'd be interesting to look at
> "lsof" output for one of the backend processes that is emitting this
> message, to see if we can identify what's getting leaked.
We have been looking at a particular issue related to a library we've
built into the backend. It does appear that this library could well be
the culprit. We are researching it further, will post more as we get
more info.
--
Until later, Geoffrey
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 14:27 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:52 ` John Allgood <jallgood@the-allgoods.net>
1 sibling, 0 replies; 18+ messages in thread
From: John Allgood @ 2006-11-30 14:52 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; +Cc: pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Maybe the whole problem is with ulimit. Ulimit sets the max files per
user at the shell and postgres might be exceeding the 1024 limit. What
would increasing this value to 2048 do?
Geoffrey wrote:
> Alvaro Herrera wrote:
>> Geoffrey wrote:
>>> Alvaro Herrera wrote:
>>>> John Allgood wrote:
>>>>> Hey Tom
>>>>>
>>>>> I assume that if it is the kernel running out of descriptors
>>>>> that I
>>>>> would get the messages in dmesg. This message only appears in the log
>>>>> file for that database.
>>>> Yeah, the point is that you have the max_files_per_process setting
>>>> higher than what the kernel likes. So decrease it, and Postgres will
>>>> adjust itself to use less file descriptors by closing and reopening
>>>> files as needed.
>>> I work with the original poster and wanted to make sure the problem
>>> here is clear.
>>
>> Yes, that was understood from the beginning.
>>
>>> The 'out of file descriptors' message is coming from Postgresql, not
>>> the kernel. Thus, it doesn't make sense to me that the
>>> max_files_per_process setting is too high. I would think we need to
>>> increase it so that Postgresql will stop generating these errors.
>>
>> No, you need to lower it so that Postgres doesn't _try_ to use as many
>> file descriptors. Read this again:
>>
>>>> So decrease it, and Postgres will
>>>> adjust itself to use less file descriptors by closing and reopening
>>>> files as needed.
>
> Okay, I'm just not getting it. Postgres complains that it is out of
> file descriptors. The kernel is not complaining about any such issues.
>
> So I should lower the max_files_per_process
> value and this will rid us of the 'out of file descriptors' error?
>
> Is it because the max_files_per_process is greater then the number of
> file descriptors that are alloted to Postgres by the kernel?
>
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 15:10 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-30 15:29 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2 siblings, 1 reply; 18+ messages in thread
From: Tom Lane @ 2006-11-30 15:10 UTC (permalink / raw)
To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Geoffrey <esoteric@3times25.net> writes:
> I work with the original poster and wanted to make sure the problem here
> is clear. The 'out of file descriptors' message is coming from
> Postgresql, not the kernel. Thus, it doesn't make sense to me that the
> max_files_per_process setting is too high. I would think we need to
> increase it so that Postgresql will stop generating these errors.
These are not "errors", they are only log messages. It would be
perfectly safe to ignore them. However, as Alvaro says, if you want
to get rid of them then the way to do that is to *decrease*
max_files_per_process to something less than whatever per-process file
limit the kernel is enforcing.
regards, tom lane
^ permalink raw reply [nested|flat] 18+ messages in thread
* Re: Out of file-descriptors message
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:12 ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
2006-11-30 15:10 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-11-30 15:29 ` Geoffrey <esoteric@3times25.net>
0 siblings, 0 replies; 18+ messages in thread
From: Geoffrey @ 2006-11-30 15:29 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>
Tom Lane wrote:
> Geoffrey <esoteric@3times25.net> writes:
>> I work with the original poster and wanted to make sure the problem here
>> is clear. The 'out of file descriptors' message is coming from
>> Postgresql, not the kernel. Thus, it doesn't make sense to me that the
>> max_files_per_process setting is too high. I would think we need to
>> increase it so that Postgresql will stop generating these errors.
>
> These are not "errors", they are only log messages. It would be
> perfectly safe to ignore them. However, as Alvaro says, if you want
> to get rid of them then the way to do that is to *decrease*
> max_files_per_process to something less than whatever per-process file
> limit the kernel is enforcing.
Understood, but we had one database that was generating these messages
continuously over a period of 4 days. Eventually, the machine became
unresponsive. Still, we did not have any file descriptor errors
generated by the kernel. Thus we are considering whether we should
decrease max_files_per_process or increase the postgres id's 'ulimit -n'
value.
--
Until later, Geoffrey
Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
- Benjamin Franklin
^ permalink raw reply [nested|flat] 18+ messages in thread
* Out of file-descriptors message
@ 2006-11-29 21:04 John Allgood <jallgood@the-allgoods.net>
0 siblings, 0 replies; 18+ messages in thread
From: John Allgood @ 2006-11-29 21:04 UTC (permalink / raw)
To: pgsql -admin <@postgresql.org>
Hello All
We are having an issue and it appears to be with file descriptors.
We are running PostgreSQL 7.4.13 on RHEL 3.0 box. This system is running
Redhat Cluster Suite with each system running multiple postmasters. The
error has occurred on both boxes and we can't find any consistencies on
what is causing the problem. I don't see this message in dmesg or any
other log file. It only appears in this single database. There are other
databases running on the same host but from different postmasters and I
don't see the same message. A couple of things to add is that this has
happened on another machine from a different database. When this
problem has been reoccurring for a period of several days the server
get fenced by the other machine. Don't know if related but seems to
happen at the time when the descriptors error is occurring. Here is a
snippet from the log of one of the databases.
2006-11-29 14:57:51 [3823] LOG: out of file descriptors: Too many open
files; release and retry
2006-11-29 14:58:01 [3823] LOG: out of file descriptors: Too many open
files; release and retry
2006-11-29 15:00:26 [3823] LOG: out of file descriptors: Too many open
files; release and retry
^ permalink raw reply [nested|flat] 18+ messages in thread
end of thread, other threads:[~2006-11-30 15:40 UTC | newest]
Thread overview: 18+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03 ` John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 13:23 ` John Allgood <jallgood@the-allgoods.net>
2006-11-30 14:12 ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:22 ` Bradley Kieser <brad@kieser.net>
2006-11-30 14:26 ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:27 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46 ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:51 ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:54 ` Geoffrey <esoteric@3times25.net>
2006-11-30 15:30 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-30 15:40 ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:52 ` John Allgood <jallgood@the-allgoods.net>
2006-11-30 15:10 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-30 15:29 ` Geoffrey <esoteric@3times25.net>
2006-11-29 21:04 Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox