From: Irene Barg <ibarg@noao.edu>
To: pgsql-admin@postgresql.org
Subject: postgresql-server-8.4.4-1PGDG.el5 - ERROR: could not write block 503414 of temporary file: No space le,ft on device
Date: Mon, 26 Jul 2010 15:01:44 -0700
Message-ID: <4C4E05C8.9090702@noao.edu> (raw)
System info: Lancelot 1874-T dual Intel Xeon(Nehalem-DP) SATAII 24GB
RAM, CentOS release 5.4 (Final), postgresql-server-8.4.4-1PGDG
hi,
We installed a fresh version of postgresql-server-8.4.4-1 using RPMs
from PGDG. We are seeing the following errors, if the system trys to do
'anything' sort of update, insert, etc.:
> 2010-07-26 11:06:59 MST LOG: database system was shut down at 2010-07-26 10:51:41 MST
> 2010-07-26 11:06:59 MST LOG: autovacuum launcher started
> 2010-07-26 11:06:59 MST LOG: database system is ready to accept connections
> 2010-07-26 11:08:18 MST science_admin metadataFATAL: password authentication failed for user "science_admin"
> 2010-07-26 11:09:33 MST postgres postgresFATAL: password authentication failed for user "postgres"
> 2010-07-26 11:09:55 MST arcsoft arcsoftFATAL: password authentication failed for user "arcsoft"
> 2010-07-26 11:10:00 MST arcsoft arcsoftFATAL: database "arcsoft" does not exist
> 2010-07-26 11:10:43 MST arcsoft postgresLOG: statement: ALTER USER postgres WITH PASSWORD 'nsa_core';
> 2010-07-26 11:11:06 MST postgres postgresLOG: statement: ALTER USER operations WITH PASSWORD 'Mosaic_DHS';
> 2010-07-26 11:11:15 MST postgres postgresLOG: statement: ALTER USER pipeline WITH PASSWORD 'Mosaic_DHS';
> 2010-07-26 11:11:23 MST postgres postgresLOG: statement: ALTER USER science_admin WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:11:31 MST postgres postgresLOG: statement: ALTER USER security_admin WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:11:48 MST postgres postgresLOG: statement: ALTER USER voi WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:15:24 MST system_admin metadataFATAL: password authentication failed for user "system_admin"
> 2010-07-26 11:19:36 MST postgres postgresLOG: statement: ALTER USER system_admin WITH PASSWORD 'nsa_core';
> 2010-07-26 11:21:03 MST postgres postgresLOG: statement: ALTER USER system_admin WITH PASSWORD 'nsa_devel';
> 2010-07-26 11:21:19 MST system_admin system_adminFATAL: database "system_admin" does not exist
> 2010-07-26 11:33:33 MST system_admin metadataERROR: could not write block 503414 of temporary file: No space le
> ft on device
> 2010-07-26 11:33:33 MST system_admin metadataHINT: Perhaps out of disk space?
The postgresql is installed on /var/lib/pgsql, but it has plenty of space::
> -bash-3.2$ df -h
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 16G 13G 1.7G 89% /
> /dev/sda8 635G 979M 601G 1% /data0
> /dev/sda7 2.0G 241M 1.7G 13% /home
> /dev/sda5 7.8G 1.5G 5.9G 20% /u1
> /dev/sda2 16G 184M 15G 2% /var/log
> /dev/sda1 487M 22M 440M 5% /boot
> tmpfs 12G 0 12G 0% /dev/shm
> /dev/md0 2.7T 202M 2.6T 1% /data1
The pg_stat_tmp is hardly growing::
> -bash-3.2$ date
> Mon Jul 26 14:37:38 MST 2010
> -bash-3.2$ pwd
> /var/lib/pgsql/data
> -bash-3.2$ du -ks pg_stat_tmp/*
> 56 pg_stat_tmp/pgstat.stat
> -bash-3.2$ du -ks pg_stat_tmp/*
> 56 pg_stat_tmp/pgstat.stat
In summary:
1) installed new version of postgresql-server-8.4.4-1.
2) restored a binary dump of a database from another system running same
version. The restore succeeded without error.
3) started using the system
We saw this same symptom in an CentOS5.2 machine, but we thought it had
to do with the version of CentOS, so we repeated the steps to a system
with more resources and running CentOS5.4.
We have postgresql-server-8.2.14-1PGDG.rhel5 on other CentOS5.3 systems
and we do not see this error.
I read this post:
> http://archives.postgresql.org/pgsql-performance/2010-04/msg00192.php
I understand that autovacuum is enabled by default, I have not modified
them:
> -bash-3.2$ grep vacuum postgresql.conf
> #vacuum_cost_delay = 0ms # 0-100 milliseconds
> #vacuum_cost_page_hit = 1 # 0-10000 credits
> #vacuum_cost_page_miss = 10 # 0-10000 credits
> #vacuum_cost_page_dirty = 20 # 0-10000 credits
> #vacuum_cost_limit = 200 # 1-10000 credits
> #autovacuum = on # Enable autovacuum subprocess? 'on'
> #log_autovacuum_min_duration = -1 # -1 disables, 0 logs all actions and
> #autovacuum_max_workers = 3 # max number of autovacuum subprocesses
> #autovacuum_naptime = 1min # time between autovacuum runs
> #autovacuum_vacuum_threshold = 50 # min number of row updates before
> # vacuum
> #autovacuum_analyze_threshold = 50 # min number of row updates before
> #autovacuum_vacuum_scale_factor = 0.2 # fraction of table size before vacuum
> #autovacuum_analyze_scale_factor = 0.1 # fraction of table size before analyze
> #autovacuum_freeze_max_age = 200000000 # maximum XID age before forced vacuum
> #autovacuum_vacuum_cost_delay = 20ms # default vacuum cost delay for
> # autovacuum, in milliseconds;
> # -1 means use vacuum_cost_delay
> #autovacuum_vacuum_cost_limit = -1 # default vacuum cost limit for
> # autovacuum, -1 means use
> # vacuum_cost_limit
> #vacuum_freeze_min_age = 50000000
> #vacuum_freeze_table_age = 150000000
Your comments/suggestions are appreciated.
--irene
---------------------------------------------------------------------
Irene Barg Email: ibarg@noao.edu
NOAO/AURA Inc. http://www.noao.edu/noao/staff/ibarg
950 N. Cherry Ave. Voice: 520-318-8273
Tucson, AZ 85726 USA FAX: 520-318-8360
---------------------------------------------------------------------
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-admin@postgresql.org
Cc: ibarg@noao.edu
Subject: Re: postgresql-server-8.4.4-1PGDG.el5 - ERROR: could not write block 503414 of temporary file: No space le,ft on device
In-Reply-To: <4C4E05C8.9090702@noao.edu>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox