pg.ddx.io  pgsql-admin@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Out of file-descriptors message
18+ messages / 6 participants
[nested] [flat]

* Out of file-descriptors message
@ 2006-11-29 20:51 John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 18+ messages in thread

From: John Allgood @ 2006-11-29 20:51 UTC (permalink / raw)
  To: pgsql -admin <@postgresql.org>; +Cc: Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>

Hello All

    We are having an issue and it appears to be with file descriptors. 
We are running PostgreSQL 7.4.13 on RHEL 3.0 box. This system is running 
Redhat Cluster Suite with each system running multiple postmasters. The 
error has occurred on both boxes and we can't find any consistencies on 
what is causing the problem. I don't see this message in dmesg or any 
other log file. It only appears in this single database. There are other 
databases running on the same host but from different postmasters and I 
don't see the same message. A couple of things to add is that this has 
happened on another machine from a different database.  When this 
problem has been reoccurring for a period of several days  the server 
get fenced by the other machine. Don't know if related but seems to 
happen at the time when the descriptors error is occurring. Here is a 
snippet from the log of one of the databases.

2006-11-29 14:57:51 [3823] LOG:  out of file descriptors: Too many open 
files; release and retry
2006-11-29 14:58:01 [3823] LOG:  out of file descriptors: Too many open 
files; release and retry
2006-11-29 15:00:26 [3823] LOG:  out of file descriptors: Too many open 
files; release and retry

-- 
**John Allgood**
/*/Senior Systems Administrator/*/
**Turbo Logistics - Division of Ozburn-Hessey Logistics**
**2251 Jesse Jewell Pky. NE**
**Gainesville, GA 30507******
**tel (678) 989-3051  fax (770) 531-7878**
**john@turbocorp.com** <mailto:john@turbocorp.com>
**www.turbocorp.com** <http://www.turbocorp.com;





^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
@ 2006-11-29 21:32 ` Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  0 siblings, 1 reply; 18+ messages in thread

From: Tom Lane @ 2006-11-29 21:32 UTC (permalink / raw)
  To: John Allgood <john@turbocorp.com>; +Cc: pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>

John Allgood <john@turbocorp.com> writes:
> 2006-11-29 14:57:51 [3823] LOG:  out of file descriptors: Too many open 
> files; release and retry

Consider reducing PG's max_files_per_process setting.  Postgres itself
will usually not have a serious problem when you've run the kernel out
of file descriptors, but everything else on the machine will :-(

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-11-29 22:03   ` John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  0 siblings, 1 reply; 18+ messages in thread

From: John Allgood @ 2006-11-29 22:03 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>

Hey Tom

    I assume that if it is the kernel running out of descriptors that I
would get the messages in dmesg. This message only appears in the log
file for that database.

Thanks

Tom Lane wrote:
> John Allgood <john@turbocorp.com> writes:
>   
>> 2006-11-29 14:57:51 [3823] LOG:  out of file descriptors: Too many open 
>> files; release and retry
>>     
>
> Consider reducing PG's max_files_per_process setting.  Postgres itself
> will usually not have a serious problem when you've run the kernel out
> of file descriptors, but everything else on the machine will :-(
>
> 			regards, tom lane
>   



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
@ 2006-11-29 22:25     ` Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 13:23       ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  0 siblings, 2 replies; 18+ messages in thread

From: Alvaro Herrera @ 2006-11-29 22:25 UTC (permalink / raw)
  To: John Allgood <jallgood@the-allgoods.net>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>; Geoffrey <esoteric@3times25.net>

John Allgood wrote:
> Hey Tom
> 
>     I assume that if it is the kernel running out of descriptors that I
> would get the messages in dmesg. This message only appears in the log
> file for that database.

Yeah, the point is that you have the max_files_per_process setting
higher than what the kernel likes.  So decrease it, and Postgres will
adjust itself to use less file descriptors by closing and reopening
files as needed.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
The PostgreSQL Company - Command Prompt, Inc.



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 13:23       ` John Allgood <jallgood@the-allgoods.net>
  1 sibling, 0 replies; 18+ messages in thread

From: John Allgood @ 2006-11-30 13:23 UTC (permalink / raw)
  To: pgsql-admin

Alvaro Herrera wrote:
> John Allgood wrote:
>   
>> Hey Tom
>>
>>     I assume that if it is the kernel running out of descriptors that I
>> would get the messages in dmesg. This message only appears in the log
>> file for that database.
>>     
>
> Yeah, the point is that you have the max_files_per_process setting
> higher than what the kernel likes.  So decrease it, and Postgres will
> adjust itself to use less file descriptors by closing and reopening
> files as needed.
>
>   
I really can't see that is possible file-max is set to 838860. I believe
it is based on the amount of ram on the machine and we have 8GB. ulimit
-a reports you can have 1024 open files. I thought maybe we were running
out of descriptors for the postgres user but that should not bring down
the machine. Thoughts anyone.



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 14:12       ` Geoffrey <esoteric@3times25.net>
  2006-11-30 14:22         ` Re: Out of file-descriptors message Bradley Kieser <brad@kieser.net>
  2006-11-30 14:27         ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 15:10         ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  1 sibling, 3 replies; 18+ messages in thread

From: Geoffrey @ 2006-11-30 14:12 UTC (permalink / raw)
  To: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Alvaro Herrera wrote:
> John Allgood wrote:
>> Hey Tom
>>
>>     I assume that if it is the kernel running out of descriptors that I
>> would get the messages in dmesg. This message only appears in the log
>> file for that database.
> 
> Yeah, the point is that you have the max_files_per_process setting
> higher than what the kernel likes.  So decrease it, and Postgres will
> adjust itself to use less file descriptors by closing and reopening
> files as needed.

I work with the original poster and wanted to make sure the problem here 
is clear.  The 'out of file descriptors' message is coming from 
Postgresql, not the kernel.  Thus, it doesn't make sense to me that the 
max_files_per_process setting is too high.  I would think we need to 
increase it so that Postgresql will stop generating these errors.

-- 
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:22         ` Bradley Kieser <brad@kieser.net>
  2006-11-30 14:26           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2 siblings, 1 reply; 18+ messages in thread

From: Bradley Kieser @ 2006-11-30 14:22 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Hmm, not entirely true. You may well only see the error in the process 
itself.


Geoffrey wrote:
> Alvaro Herrera wrote:
>> John Allgood wrote:
>>> Hey Tom
>>>
>>>     I assume that if it is the kernel running out of descriptors that I
>>> would get the messages in dmesg. This message only appears in the log
>>> file for that database.
>>
>> Yeah, the point is that you have the max_files_per_process setting
>> higher than what the kernel likes.  So decrease it, and Postgres will
>> adjust itself to use less file descriptors by closing and reopening
>> files as needed.
>
> I work with the original poster and wanted to make sure the problem 
> here is clear.  The 'out of file descriptors' message is coming from 
> Postgresql, not the kernel.  Thus, it doesn't make sense to me that 
> the max_files_per_process setting is too high.  I would think we need 
> to increase it so that Postgresql will stop generating these errors.
>



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:22         ` Re: Out of file-descriptors message Bradley Kieser <brad@kieser.net>
@ 2006-11-30 14:26           ` Geoffrey <esoteric@3times25.net>
  0 siblings, 0 replies; 18+ messages in thread

From: Geoffrey @ 2006-11-30 14:26 UTC (permalink / raw)
  To: Bradley Kieser <brad@kieser.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Bradley Kieser wrote:
> Hmm, not entirely true. You may well only see the error in the process 
> itself.

You're telling me that the kernel could be running out of file 
descriptors and I wouldn't see a message regarding this in the kernel 
logs?  I don't believe that is possible.

> 
> 
> Geoffrey wrote:
>> Alvaro Herrera wrote:
>>> John Allgood wrote:
>>>> Hey Tom
>>>>
>>>>     I assume that if it is the kernel running out of descriptors that I
>>>> would get the messages in dmesg. This message only appears in the log
>>>> file for that database.
>>>
>>> Yeah, the point is that you have the max_files_per_process setting
>>> higher than what the kernel likes.  So decrease it, and Postgres will
>>> adjust itself to use less file descriptors by closing and reopening
>>> files as needed.
>>
>> I work with the original poster and wanted to make sure the problem 
>> here is clear.  The 'out of file descriptors' message is coming from 
>> Postgresql, not the kernel.  Thus, it doesn't make sense to me that 
>> the max_files_per_process setting is too high.  I would think we need 
>> to increase it so that Postgresql will stop generating these errors.
>>
> 


-- 
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:27         ` Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:46           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2 siblings, 1 reply; 18+ messages in thread

From: Alvaro Herrera @ 2006-11-30 14:27 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Geoffrey wrote:
> Alvaro Herrera wrote:
> >John Allgood wrote:
> >>Hey Tom
> >>
> >>    I assume that if it is the kernel running out of descriptors that I
> >>would get the messages in dmesg. This message only appears in the log
> >>file for that database.
> >
> >Yeah, the point is that you have the max_files_per_process setting
> >higher than what the kernel likes.  So decrease it, and Postgres will
> >adjust itself to use less file descriptors by closing and reopening
> >files as needed.
> 
> I work with the original poster and wanted to make sure the problem here 
> is clear.

Yes, that was understood from the beginning.

> The 'out of file descriptors' message is coming from 
> Postgresql, not the kernel.  Thus, it doesn't make sense to me that the 
> max_files_per_process setting is too high.  I would think we need to 
> increase it so that Postgresql will stop generating these errors.

No, you need to lower it so that Postgres doesn't _try_ to use as many
file descriptors.  Read this again:

> >                                    So decrease it, and Postgres will
> >adjust itself to use less file descriptors by closing and reopening
> >files as needed.

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:27         ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 14:46           ` Geoffrey <esoteric@3times25.net>
  2006-11-30 14:51             ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:52             ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  0 siblings, 2 replies; 18+ messages in thread

From: Geoffrey @ 2006-11-30 14:46 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Alvaro Herrera wrote:
> Geoffrey wrote:
>> Alvaro Herrera wrote:
>>> John Allgood wrote:
>>>> Hey Tom
>>>>
>>>>    I assume that if it is the kernel running out of descriptors that I
>>>> would get the messages in dmesg. This message only appears in the log
>>>> file for that database.
>>> Yeah, the point is that you have the max_files_per_process setting
>>> higher than what the kernel likes.  So decrease it, and Postgres will
>>> adjust itself to use less file descriptors by closing and reopening
>>> files as needed.
>> I work with the original poster and wanted to make sure the problem here 
>> is clear.
> 
> Yes, that was understood from the beginning.
> 
>> The 'out of file descriptors' message is coming from 
>> Postgresql, not the kernel.  Thus, it doesn't make sense to me that the 
>> max_files_per_process setting is too high.  I would think we need to 
>> increase it so that Postgresql will stop generating these errors.
> 
> No, you need to lower it so that Postgres doesn't _try_ to use as many
> file descriptors.  Read this again:
> 
>>>                                    So decrease it, and Postgres will
>>> adjust itself to use less file descriptors by closing and reopening
>>> files as needed.

Okay, I'm just not getting it.  Postgres complains that it is out of 
file descriptors.  The kernel is not complaining about any such issues. 
 
 
                         So I should lower the max_files_per_process 
value and this will rid us of the 'out of file descriptors' error?

Is it because the max_files_per_process is greater then the number of 
file descriptors that are alloted to Postgres by the kernel?

-- 
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:27         ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:46           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:51             ` Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:54               ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  1 sibling, 1 reply; 18+ messages in thread

From: Alvaro Herrera @ 2006-11-30 14:51 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Geoffrey wrote:

> Okay, I'm just not getting it.  Postgres complains that it is out of 
> file descriptors.  The kernel is not complaining about any such issues. 
> 
> 
>                         So I should lower the max_files_per_process 
> value and this will rid us of the 'out of file descriptors' error?
> 
> Is it because the max_files_per_process is greater then the number of 
> file descriptors that are alloted to Postgres by the kernel?

Yes.  Which is kinda weird, because Postgres actually tests the number
when it starts, so that if you set the number too high, it will decrease
it according to what the kernel allows.

Maybe the test is newer than the version you are running -- what was it,
again?

-- 
Alvaro Herrera                                http://www.CommandPrompt.com/
PostgreSQL Replication, Consulting, Custom Development, 24x7 support



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:27         ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:46           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:51             ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
@ 2006-11-30 14:54               ` Geoffrey <esoteric@3times25.net>
  2006-11-30 15:30                 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  0 siblings, 1 reply; 18+ messages in thread

From: Geoffrey @ 2006-11-30 14:54 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Alvaro Herrera wrote:
> Geoffrey wrote:
> 
>> Okay, I'm just not getting it.  Postgres complains that it is out of 
>> file descriptors.  The kernel is not complaining about any such issues. 
>>
>>
>>                         So I should lower the max_files_per_process 
>> value and this will rid us of the 'out of file descriptors' error?
>>
>> Is it because the max_files_per_process is greater then the number of 
>> file descriptors that are alloted to Postgres by the kernel?
> 
> Yes.  Which is kinda weird, because Postgres actually tests the number
> when it starts, so that if you set the number too high, it will decrease
> it according to what the kernel allows.
> 
> Maybe the test is newer than the version you are running -- what was it,
> again?
> 
7.4.13

-- 
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:27         ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:46           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:51             ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:54               ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 15:30                 ` Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-30 15:40                   ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  0 siblings, 1 reply; 18+ messages in thread

From: Tom Lane @ 2006-11-30 15:30 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Geoffrey <esoteric@3times25.net> writes:
> Alvaro Herrera wrote:
>> Yes.  Which is kinda weird, because Postgres actually tests the number
>> when it starts, so that if you set the number too high, it will decrease
>> it according to what the kernel allows.
>> 
>> Maybe the test is newer than the version you are running -- what was it,
>> again?
>> 
> 7.4.13

Hmph ... 7.4.13 does have the same set_max_safe_fds() logic as HEAD.
So this shouldn't be happening really, no matter what the relative
values of max_files_per_process and ulimit -n are.

Explanations I can think of:

* ulimit -n decreased after process startup (is this even possible?)

* something is leaking file descriptors that are outside fd.c's control,
such that eventually there are more than NUM_RESERVED_FDS of 'em.

Theory B seems a lot more plausible.  It'd be interesting to look at
"lsof" output for one of the backend processes that is emitting this
message, to see if we can identify what's getting leaked.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:27         ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:46           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:51             ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:54               ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 15:30                 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-11-30 15:40                   ` Geoffrey <esoteric@3times25.net>
  0 siblings, 0 replies; 18+ messages in thread

From: Geoffrey @ 2006-11-30 15:40 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Tom Lane wrote:
> Geoffrey <esoteric@3times25.net> writes:
>> Alvaro Herrera wrote:
>>> Yes.  Which is kinda weird, because Postgres actually tests the number
>>> when it starts, so that if you set the number too high, it will decrease
>>> it according to what the kernel allows.
>>>
>>> Maybe the test is newer than the version you are running -- what was it,
>>> again?
>>>
>> 7.4.13
> 
> Hmph ... 7.4.13 does have the same set_max_safe_fds() logic as HEAD.
> So this shouldn't be happening really, no matter what the relative
> values of max_files_per_process and ulimit -n are.
> 
> Explanations I can think of:
> 
> * ulimit -n decreased after process startup (is this even possible?)
> 
> * something is leaking file descriptors that are outside fd.c's control,
> such that eventually there are more than NUM_RESERVED_FDS of 'em.
> 
> Theory B seems a lot more plausible.  It'd be interesting to look at
> "lsof" output for one of the backend processes that is emitting this
> message, to see if we can identify what's getting leaked.

We have been looking at a particular issue related to a library we've 
built into the backend.  It does appear that this library could well be 
the culprit.  We are researching it further, will post more as we get 
more info.

-- 
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 14:27         ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:46           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 14:52             ` John Allgood <jallgood@the-allgoods.net>
  1 sibling, 0 replies; 18+ messages in thread

From: John Allgood @ 2006-11-30 14:52 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; +Cc: pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Maybe the whole problem is with ulimit. Ulimit sets the max files per
user at the shell and postgres might be exceeding the 1024 limit. What
would increasing this value to 2048 do?

Geoffrey wrote:
> Alvaro Herrera wrote:
>> Geoffrey wrote:
>>> Alvaro Herrera wrote:
>>>> John Allgood wrote:
>>>>> Hey Tom
>>>>>
>>>>>    I assume that if it is the kernel running out of descriptors
>>>>> that I
>>>>> would get the messages in dmesg. This message only appears in the log
>>>>> file for that database.
>>>> Yeah, the point is that you have the max_files_per_process setting
>>>> higher than what the kernel likes.  So decrease it, and Postgres will
>>>> adjust itself to use less file descriptors by closing and reopening
>>>> files as needed.
>>> I work with the original poster and wanted to make sure the problem
>>> here is clear.
>>
>> Yes, that was understood from the beginning.
>>
>>> The 'out of file descriptors' message is coming from Postgresql, not
>>> the kernel.  Thus, it doesn't make sense to me that the
>>> max_files_per_process setting is too high.  I would think we need to
>>> increase it so that Postgresql will stop generating these errors.
>>
>> No, you need to lower it so that Postgres doesn't _try_ to use as many
>> file descriptors.  Read this again:
>>
>>>>                                    So decrease it, and Postgres will
>>>> adjust itself to use less file descriptors by closing and reopening
>>>> files as needed.
>
> Okay, I'm just not getting it.  Postgres complains that it is out of
> file descriptors.  The kernel is not complaining about any such issues.
>
>                         So I should lower the max_files_per_process
> value and this will rid us of the 'out of file descriptors' error?
>
> Is it because the max_files_per_process is greater then the number of
> file descriptors that are alloted to Postgres by the kernel?
>



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
@ 2006-11-30 15:10         ` Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-30 15:29           ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2 siblings, 1 reply; 18+ messages in thread

From: Tom Lane @ 2006-11-30 15:10 UTC (permalink / raw)
  To: Geoffrey <esoteric@3times25.net>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Geoffrey <esoteric@3times25.net> writes:
> I work with the original poster and wanted to make sure the problem here 
> is clear.  The 'out of file descriptors' message is coming from 
> Postgresql, not the kernel.  Thus, it doesn't make sense to me that the 
> max_files_per_process setting is too high.  I would think we need to 
> increase it so that Postgresql will stop generating these errors.

These are not "errors", they are only log messages.  It would be
perfectly safe to ignore them.  However, as Alvaro says, if you want
to get rid of them then the way to do that is to *decrease*
max_files_per_process to something less than whatever per-process file
limit the kernel is enforcing.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Re: Out of file-descriptors message
  2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
  2006-11-29 21:32 ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
  2006-11-29 22:03   ` Re: Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>
  2006-11-29 22:25     ` Re: Out of file-descriptors message Alvaro Herrera <alvherre@commandprompt.com>
  2006-11-30 14:12       ` Re: Out of file-descriptors message Geoffrey <esoteric@3times25.net>
  2006-11-30 15:10         ` Re: Out of file-descriptors message Tom Lane <tgl@sss.pgh.pa.us>
@ 2006-11-30 15:29           ` Geoffrey <esoteric@3times25.net>
  0 siblings, 0 replies; 18+ messages in thread

From: Geoffrey @ 2006-11-30 15:29 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: John Allgood <john@turbocorp.com>; pgsql-admin@postgresql.org, Terry Lee Tucker <terry@leetuckert.net>; J. D. Pearson <jpearson@turbocorp.com>

Tom Lane wrote:
> Geoffrey <esoteric@3times25.net> writes:
>> I work with the original poster and wanted to make sure the problem here 
>> is clear.  The 'out of file descriptors' message is coming from 
>> Postgresql, not the kernel.  Thus, it doesn't make sense to me that the 
>> max_files_per_process setting is too high.  I would think we need to 
>> increase it so that Postgresql will stop generating these errors.
> 
> These are not "errors", they are only log messages.  It would be
> perfectly safe to ignore them.  However, as Alvaro says, if you want
> to get rid of them then the way to do that is to *decrease*
> max_files_per_process to something less than whatever per-process file
> limit the kernel is enforcing.

Understood, but we had one database that was generating these messages 
continuously over a period of 4 days.  Eventually, the machine became 
unresponsive.  Still, we did not have any file descriptor errors 
generated by the kernel.  Thus we are considering whether we should 
decrease max_files_per_process or increase the postgres id's 'ulimit -n' 
value.

-- 
Until later, Geoffrey

Those who would give up essential Liberty, to purchase a little
temporary Safety, deserve neither Liberty nor Safety.
  - Benjamin Franklin



^ permalink  raw  reply  [nested|flat] 18+ messages in thread

* Out of file-descriptors message
@ 2006-11-29 21:04 John Allgood <jallgood@the-allgoods.net>
  0 siblings, 0 replies; 18+ messages in thread

From: John Allgood @ 2006-11-29 21:04 UTC (permalink / raw)
  To: pgsql -admin <@postgresql.org>

Hello All

   We are having an issue and it appears to be with file descriptors.
We are running PostgreSQL 7.4.13 on RHEL 3.0 box. This system is running
Redhat Cluster Suite with each system running multiple postmasters. The
error has occurred on both boxes and we can't find any consistencies on
what is causing the problem. I don't see this message in dmesg or any
other log file. It only appears in this single database. There are other
databases running on the same host but from different postmasters and I
don't see the same message. A couple of things to add is that this has
happened on another machine from a different database.  When this
problem has been reoccurring for a period of several days  the server
get fenced by the other machine. Don't know if related but seems to
happen at the time when the descriptors error is occurring. Here is a
snippet from the log of one of the databases.

2006-11-29 14:57:51 [3823] LOG:  out of file descriptors: Too many open
files; release and retry
2006-11-29 14:58:01 [3823] LOG:  out of file descriptors: Too many open
files; release and retry
2006-11-29 15:00:26 [3823] LOG:  out of file descriptors: Too many open
files; release and retry



^ permalink  raw  reply  [nested|flat] 18+ messages in thread


end of thread, other threads:[~2006-11-30 15:40 UTC | newest]

Thread overview: 18+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-11-29 20:51 Out of file-descriptors message John Allgood <john@turbocorp.com>
2006-11-29 21:32 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-29 22:03   ` John Allgood <jallgood@the-allgoods.net>
2006-11-29 22:25     ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 13:23       ` John Allgood <jallgood@the-allgoods.net>
2006-11-30 14:12       ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:22         ` Bradley Kieser <brad@kieser.net>
2006-11-30 14:26           ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:27         ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:46           ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:51             ` Alvaro Herrera <alvherre@commandprompt.com>
2006-11-30 14:54               ` Geoffrey <esoteric@3times25.net>
2006-11-30 15:30                 ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-30 15:40                   ` Geoffrey <esoteric@3times25.net>
2006-11-30 14:52             ` John Allgood <jallgood@the-allgoods.net>
2006-11-30 15:10         ` Tom Lane <tgl@sss.pgh.pa.us>
2006-11-30 15:29           ` Geoffrey <esoteric@3times25.net>
2006-11-29 21:04 Out of file-descriptors message John Allgood <jallgood@the-allgoods.net>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox