pg.ddx.io pgsql-performance@postgresql.org mailing list archive
help / color / mirror / Atom feedMemory issues
14+ messages / 3 participants
[nested] [flat]
* Memory issues
@ 2012-09-24 06:41 Shiran Kleiderman <shirank1@gmail.com>
0 siblings, 0 replies; 14+ messages in thread
From: Shiran Kleiderman @ 2012-09-24 06:41 UTC (permalink / raw)
To: pgsql-performance
Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.
*Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform*
I've changed the main default values under file *postgresql.conf* to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB
When I run the app, after an hour or two, free -m looks like below ans the
crons can't run due to memory loss or similar (i'm new to postgres and db
admin).
Thanks!
free -m, errors:
total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511
total used *free* shared buffers cached
Mem: 17079 16833 *245 *0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511
**errors:
*DBI connect('database=---;host=localhost','postgres',...) failed: could
not fork new process for connection: Cannot allocate memory*
could not fork new process for connection: Cannot allocate memory
and
execute failed: ERROR: out of memory
DETAIL: Failed on request of size 968. [for Statement "
SELECT DISTINCT....
Thank you!
^ permalink raw reply [nested|flat] 14+ messages in thread
* Memory issues
@ 2012-09-24 06:45 Shiran Kleiderman <shirank1@gmail.com>
0 siblings, 2 replies; 14+ messages in thread
From: Shiran Kleiderman @ 2012-09-24 06:45 UTC (permalink / raw)
To: pgsql-general@postgresql.org
Hi,
I'm using and Amazon ec2 instance with the following spec and the
application that I'm running uses a postgres DB 9.1.
The app has 3 main cron jobs.
*Ubuntu 12, High-Memory Extra Large Instance
17.1 GB of memory
6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
420 GB of instance storage
64-bit platform*
I've changed the main default values under file *postgresql.conf* to:
shared_buffers = 4GB
work_mem = 16MB
wal_buffers = 16MB
checkpoint_segments = 32
effective_cache_size = 8GB
When I run the app, after an hour or two, free -m looks like below ans the
crons can't run due to memory loss or similar (i'm new to postgres and db
admin).
Thanks!
free -m, errors:
total used free shared buffers cached
Mem: 17079 13742 3337 0 64 11882
-/+ buffers/cache: 1796 15283
Swap: 511 0 511
total used *free* shared buffers cached
Mem: 17079 16833 *245 *0 42 14583
-/+ buffers/cache: 2207 14871
Swap: 511 0 511
**free above stays low even when nothing is running.
**errors:
*DBI connect('database=---;host=localhost','postgres',...) failed: could
not fork new process for connection: Cannot allocate memory*
could not fork new process for connection: Cannot allocate memory
and
execute failed: ERROR: out of memory
DETAIL: Failed on request of size 968. [for Statement "
SELECT DISTINCT....
Thank you!
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Memory issues
@ 2012-09-26 00:56 Scott Marlowe <scott.marlowe@gmail.com>
parent: Shiran Kleiderman <shirank1@gmail.com>
1 sibling, 1 reply; 14+ messages in thread
From: Scott Marlowe @ 2012-09-26 00:56 UTC (permalink / raw)
To: Shiran Kleiderman <shirank1@gmail.com>; +Cc: pgsql-general@postgresql.org
On Mon, Sep 24, 2012 at 12:45 AM, Shiran Kleiderman <shirank1@gmail.com> wrote:
>
>
> Hi,
> I'm using and Amazon ec2 instance with the following spec and the
> application that I'm running uses a postgres DB 9.1.
> The app has 3 main cron jobs.
>
> Ubuntu 12, High-Memory Extra Large Instance
> 17.1 GB of memory
> 6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
> 420 GB of instance storage
> 64-bit platform
>
> I've changed the main default values under file postgresql.conf to:
> shared_buffers = 4GB
> work_mem = 16MB
> wal_buffers = 16MB
> checkpoint_segments = 32
> effective_cache_size = 8GB
>
> When I run the app, after an hour or two, free -m looks like below ans the
> crons can't run due to memory loss or similar (i'm new to postgres and db
> admin).
> Thanks!
>
> free -m, errors:
>
> total used free shared buffers cached
> Mem: 17079 13742 3337 0 64 11882
> -/+ buffers/cache: 1796 15283
> Swap: 511 0 511
You have 11.8G cached, that's basically free memory on demand.
> total used free shared buffers cached
> Mem: 17079 16833 245 0 42 14583
> -/+ buffers/cache: 2207 14871
> Swap: 511 0 511
Here you have 14.5G cached, again that's free memory so to speak.
I.e. when something needs it it gets allocated.
> **free above stays low even when nothing is running.
>
>
> **errors:
> DBI connect('database=---;host=localhost','postgres',...) failed: could not
> fork new process for connection: Cannot allocate memory
> could not fork new process for connection: Cannot allocate memory
This error is happening in your client process. Maybe it's 32 bit or
something and running out of local memory in its process space? Maybe
memory is so fragmented that no large blocks can get allocated or
something? Either way, your machine has plenty of memory according to
free. BTW, it's pretty common for folks new to unix to mis-read free
and not realize that cached memory + free memory is what's really
available.
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Memory issues
@ 2012-09-26 01:00 Shiran Kleiderman <shirank1@gmail.com>
parent: Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Shiran Kleiderman @ 2012-09-26 01:00 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; pgsql-general@postgresql.org
Hi
Thanks for your answer.
I understood that the server is ok memory wise.
What can I check on the client side or the DB queries?
Thank u.
On Wed, Sep 26, 2012 at 2:56 AM, Scott Marlowe <scott.marlowe@gmail.com>wrote:
> On Mon, Sep 24, 2012 at 12:45 AM, Shiran Kleiderman <shirank1@gmail.com>
> wrote:
> >
> >
> > Hi,
> > I'm using and Amazon ec2 instance with the following spec and the
> > application that I'm running uses a postgres DB 9.1.
> > The app has 3 main cron jobs.
> >
> > Ubuntu 12, High-Memory Extra Large Instance
> > 17.1 GB of memory
> > 6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
> > 420 GB of instance storage
> > 64-bit platform
> >
> > I've changed the main default values under file postgresql.conf to:
> > shared_buffers = 4GB
> > work_mem = 16MB
> > wal_buffers = 16MB
> > checkpoint_segments = 32
> > effective_cache_size = 8GB
> >
> > When I run the app, after an hour or two, free -m looks like below ans
> the
> > crons can't run due to memory loss or similar (i'm new to postgres and db
> > admin).
> > Thanks!
> >
> > free -m, errors:
> >
> > total used free shared buffers cached
> > Mem: 17079 13742 3337 0 64 11882
> > -/+ buffers/cache: 1796 15283
> > Swap: 511 0 511
>
> You have 11.8G cached, that's basically free memory on demand.
>
> > total used free shared buffers cached
> > Mem: 17079 16833 245 0 42 14583
> > -/+ buffers/cache: 2207 14871
> > Swap: 511 0 511
>
> Here you have 14.5G cached, again that's free memory so to speak.
> I.e. when something needs it it gets allocated.
>
> > **free above stays low even when nothing is running.
> >
> >
> > **errors:
> > DBI connect('database=---;host=localhost','postgres',...) failed: could
> not
> > fork new process for connection: Cannot allocate memory
> > could not fork new process for connection: Cannot allocate memory
>
> This error is happening in your client process. Maybe it's 32 bit or
> something and running out of local memory in its process space? Maybe
> memory is so fragmented that no large blocks can get allocated or
> something? Either way, your machine has plenty of memory according to
> free. BTW, it's pretty common for folks new to unix to mis-read free
> and not realize that cached memory + free memory is what's really
> available.
>
--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Memory issues
@ 2012-09-26 16:29 Scott Marlowe <scott.marlowe@gmail.com>
parent: Shiran Kleiderman <shirank1@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Scott Marlowe @ 2012-09-26 16:29 UTC (permalink / raw)
To: Shiran Kleiderman <shirank1@gmail.com>; +Cc: pgsql-general@postgresql.org
On Tue, Sep 25, 2012 at 7:00 PM, Shiran Kleiderman <shirank1@gmail.com> wrote:
>
> Hi
> Thanks for your answer.
> I understood that the server is ok memory wise.
> What can I check on the client side or the DB queries?
Well you're connecting to localhost so I'd expect you to show a memory
issue in free I'm not seeing. Are you really connecting to localhost
or not?
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Memory issues
@ 2012-09-26 16:41 Shiran Kleiderman <shirank1@gmail.com>
parent: Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Shiran Kleiderman @ 2012-09-26 16:41 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; pgsql-general@postgresql.org
Hi
Thanks again.
Right now, this is *free -m and ps aux* and non of the crons can run -
can't allocate memory.
cif@domU-12-31-39-08-06-20:~$ free -m
total used free shared buffers cached
Mem: 17079 12051 5028 0 270 9578
-/+ buffers/cache: 2202 14877
Swap: 511 0 511
cif@domU-12-31-39-08-06-20:~$ ps aux
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1 0.0 0.0 24316 2280 ? Ss Sep24 0:00 /sbin/init
root 2 0.0 0.0 0 0 ? S Sep24 0:00 [kthreadd]
root 3 0.0 0.0 0 0 ? S Sep24 0:00
[ksoftirqd/0]
root 4 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/0:0]
root 5 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/u:0]
root 6 0.0 0.0 0 0 ? S Sep24 0:00
[migration/0]
root 7 0.0 0.0 0 0 ? S Sep24 0:00
[watchdog/0]
root 8 0.0 0.0 0 0 ? S Sep24 0:00
[migration/1]
root 9 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/1:0]
root 10 0.0 0.0 0 0 ? S Sep24 0:01
[ksoftirqd/1]
root 11 0.0 0.0 0 0 ? S Sep24 0:00
[watchdog/1]
root 12 0.0 0.0 0 0 ? S< Sep24 0:00 [cpuset]
root 13 0.0 0.0 0 0 ? S< Sep24 0:00 [khelper]
root 14 0.0 0.0 0 0 ? S Sep24 0:00 [kdevtmpfs]
root 15 0.0 0.0 0 0 ? S< Sep24 0:00 [netns]
root 16 0.0 0.0 0 0 ? S Sep24 0:00
[kworker/u:1]
root 17 0.0 0.0 0 0 ? S Sep24 0:00 [xenwatch]
root 18 0.0 0.0 0 0 ? S Sep24 0:00 [xenbus]
root 19 0.0 0.0 0 0 ? S Sep24 0:00
[sync_supers]
root 20 0.0 0.0 0 0 ? S Sep24 0:00
[bdi-default]
root 21 0.0 0.0 0 0 ? S< Sep24 0:00
[kintegrityd]
root 22 0.0 0.0 0 0 ? S< Sep24 0:00 [kblockd]
root 23 0.0 0.0 0 0 ? S< Sep24 0:00 [ata_sff]
root 24 0.0 0.0 0 0 ? S Sep24 0:00 [khubd]
root 25 0.0 0.0 0 0 ? S< Sep24 0:00 [md]
root 26 0.0 0.0 0 0 ? S Sep24 0:02
[kworker/0:1]
root 28 0.0 0.0 0 0 ? S Sep24 0:00
[khungtaskd]
root 29 0.0 0.0 0 0 ? S Sep24 0:00 [kswapd0]
root 30 0.0 0.0 0 0 ? SN Sep24 0:00 [ksmd]
root 31 0.0 0.0 0 0 ? S Sep24 0:00
[fsnotify_mark]
root 32 0.0 0.0 0 0 ? S Sep24 0:00
[ecryptfs-kthrea]
root 33 0.0 0.0 0 0 ? S< Sep24 0:00 [crypto]
root 41 0.0 0.0 0 0 ? S< Sep24 0:00 [kthrotld]
root 42 0.0 0.0 0 0 ? S Sep24 0:00 [khvcd]
root 43 0.0 0.0 0 0 ? S Sep24 0:01
[kworker/1:1]
root 62 0.0 0.0 0 0 ? S< Sep24 0:00
[devfreq_wq]
root 176 0.0 0.0 0 0 ? S< Sep24 0:00 [kdmflush]
root 187 0.0 0.0 0 0 ? S Sep24 0:01
[jbd2/xvda1-8]
root 188 0.0 0.0 0 0 ? S< Sep24 0:00
[ext4-dio-unwrit]
root 258 0.0 0.0 17224 640 ? S Sep24 0:00
upstart-udev-bridge --daemon
root 265 0.0 0.0 21460 1196 ? Ss Sep24 0:00
/sbin/udevd --daemon
root 328 0.0 0.0 21456 712 ? S Sep24 0:00
/sbin/udevd --daemon
root 329 0.0 0.0 21456 716 ? S Sep24 0:00
/sbin/udevd --daemon
root 389 0.0 0.0 15180 392 ? S Sep24 0:00
upstart-socket-bridge --daemon
root 419 0.0 0.0 7256 1008 ? Ss Sep24 0:00 dhclient3
-e IF_METRIC=100 -pf /var/run/dhclient.eth0.pid -lf
/var/lib/dhcp/dhclient.eth0.leases -1 eth
root 574 0.0 0.0 0 0 ? S Sep24 0:03
[jbd2/dm-0-8]
root 575 0.0 0.0 0 0 ? S< Sep24 0:00
[ext4-dio-unwrit]
root 610 0.0 0.0 49948 2880 ? Ss Sep24 0:00
/usr/sbin/sshd -D
syslog 625 0.0 0.0 253708 1552 ? Sl Sep24 0:11 rsyslogd
-c5
102 630 0.0 0.0 23808 944 ? Ss Sep24 0:00
dbus-daemon --system --fork --activation=upstart
root 687 0.0 0.0 14496 968 tty4 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty4
root 696 0.0 0.0 14496 972 tty5 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty5
root 708 0.0 0.0 14496 968 tty2 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty2
root 710 0.0 0.0 14496 964 tty3 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty3
root 715 0.0 0.0 14496 968 tty6 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty6
root 720 0.0 0.0 4320 660 ? Ss Sep24 0:00 acpid -c
/etc/acpi/events -s /var/run/acpid.socket
root 728 0.0 11.9 2194848 2097324 ? Ss Sep24 0:09
/usr/bin/searchd --nodetach
root 733 0.0 0.0 19104 928 ? Ss Sep24 0:00 cron
daemon 735 0.0 0.0 16900 376 ? Ss Sep24 0:00 atd
bind 739 0.0 0.0 235540 13404 ? Ssl Sep24 0:00
/usr/sbin/named -u bind
mysql 755 0.0 0.2 558104 47940 ? Ssl Sep24 0:34
/usr/sbin/mysqld
whoopsie 790 0.0 0.0 187576 4236 ? Ssl Sep24 0:00 whoopsie
root 924 0.0 0.0 0 0 ? S Sep24 0:00
[flush-252:0]
root 999 0.0 0.0 99400 6496 ? Ss Sep24 0:04
/usr/sbin/apache2 -k start
www-data 1018 0.0 1.0 427080 185684 ? S Sep24 0:10
/usr/sbin/apache2 -k start
www-data 1019 0.0 1.0 427140 185852 ? S Sep24 0:33
/usr/sbin/apache2 -k start
root 1032 0.0 0.1 80220 21276 ? Ss Sep24 0:17 starman
master --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1035 0.0 0.0 14496 968 tty1 Ss+ Sep24 0:00
/sbin/getty -8 38400 tty1
root 1037 0.0 0.1 184400 28532 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1038 0.0 0.1 184444 28592 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1039 0.0 0.1 184132 28040 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1040 0.0 0.1 184408 28600 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
root 1041 0.0 0.1 184444 28588 ? S Sep24 0:00 starman
worker --port 5000 --daemonize -MMoose /usr/local/cif-rest-sphinx/CIF.psgi
www-data 1055 0.0 1.2 469948 225732 ? S Sep24 1:04
/usr/sbin/apache2 -k start
www-data 1056 0.0 1.0 427180 185924 ? S Sep24 0:28
/usr/sbin/apache2 -k start
www-data 3452 0.0 1.0 426964 185624 ? S Sep25 0:28
/usr/sbin/apache2 -k start
www-data 3775 0.0 1.0 426900 185696 ? S Sep25 0:14
/usr/sbin/apache2 -k start
postgres 4717 0.0 0.6 4411584 113372 ? S Sep25 0:01
/usr/lib/postgresql/9.1/bin/postgres -D /mnt/dbstorage/9.1/main -c
config_file=/etc/postgresql/9.1/main
postgres 4720 0.0 0.1 4413628 31392 ? Ss Sep25 0:09 postgres:
writer process
postgres 4721 0.0 0.0 4413636 1808 ? Ss Sep25 0:08 postgres:
wal writer process
postgres 4722 0.0 0.0 4414344 3044 ? Ss Sep25 1:36 postgres:
autovacuum launcher process
postgres 4723 0.0 0.0 94920 1752 ? Ss Sep25 0:41 postgres:
stats collector process
postgres 4738 0.0 7.0 4491580 1229488 ? Ss Sep25 0:58 postgres:
postgres cif 127.0.0.1(56867) idle
postgres 4740 0.0 6.9 4417748 1221100 ? Ss Sep25 0:18 postgres:
postgres cif 127.0.0.1(56869) idle
postgres 4741 0.0 7.0 4425720 1228856 ? Ss Sep25 0:31 postgres:
postgres cif 127.0.0.1(56870) idle
postgres 4742 0.0 6.9 4417376 1220464 ? Ss Sep25 0:08 postgres:
postgres cif 127.0.0.1(56871) idle
postgres 4743 0.0 7.0 4421104 1225328 ? Ss Sep25 0:24 postgres:
postgres cif 127.0.0.1(56872) idle
postgres 4745 0.0 7.0 4421124 1225040 ? Ss Sep25 0:27 postgres:
postgres cif 127.0.0.1(56874) idle
www-data 4746 0.0 1.2 455984 212284 ? S Sep25 0:30
/usr/sbin/apache2 -k start
postgres 4754 0.0 6.9 4417728 1220988 ? Ss Sep25 0:30 postgres:
postgres cif 127.0.0.1(56879) idle
www-data 4755 0.0 0.9 403836 163240 ? S Sep25 0:06
/usr/sbin/apache2 -k start
postgres 4765 0.0 6.9 4482528 1220704 ? Ss Sep25 0:23 postgres:
postgres cif 127.0.0.1(56881) idle
www-data 4995 0.0 0.9 403464 162872 ? S Sep25 0:03
/usr/sbin/apache2 -k start
www-data 4997 0.0 0.3 305460 64996 ? S Sep25 0:00
/usr/sbin/apache2 -k start
postgres 5002 0.0 6.9 4417384 1220172 ? Ss Sep25 0:09 postgres:
postgres cif 127.0.0.1(56895) idle
postgres 5003 0.0 7.1 4417920 1243772 ? Ss Sep25 0:06 postgres:
postgres cif 127.0.0.1(56896) idle
root 5218 0.0 0.0 0 0 ? S Sep25 0:00
[flush-202:1]
root 5820 0.0 0.0 73352 3568 ? Ss 16:37 0:00 sshd:
ubuntu [priv]
ubuntu 5950 0.0 0.0 73352 1676 ? S 16:37 0:00 sshd:
ubuntu@pts/0
ubuntu 5952 0.6 0.0 25872 8432 pts/0 Ss 16:37 0:00 -bash
root 6048 0.0 0.0 41896 1752 pts/0 S 16:38 0:00 sudo su -
cif
cif 6049 0.0 0.0 39516 1388 pts/0 S 16:38 0:00 su - cif
cif 6050 0.8 0.0 25912 8472 pts/0 S 16:38 0:00 -su
cif 6161 0.0 0.0 16872 1272 pts/0 R+ 16:38 0:00 ps aux
On Wed, Sep 26, 2012 at 6:29 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:
> On Tue, Sep 25, 2012 at 7:00 PM, Shiran Kleiderman <shirank1@gmail.com>
> wrote:
> >
> > Hi
> > Thanks for your answer.
> > I understood that the server is ok memory wise.
> > What can I check on the client side or the DB queries?
>
> Well you're connecting to localhost so I'd expect you to show a memory
> issue in free I'm not seeing. Are you really connecting to localhost
> or not?
>
--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [GENERAL] Memory issues
@ 2012-09-26 20:55 Scott Marlowe <scott.marlowe@gmail.com>
parent: Shiran Kleiderman <shirank1@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Scott Marlowe @ 2012-09-26 20:55 UTC (permalink / raw)
To: Shiran Kleiderman <shirank1@gmail.com>; +Cc: pgsql-performance
On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com> wrote:
> Hi
> Thanks again.
> Right now, this is free -m and ps aux and non of the crons can run - can't
> allocate memory.
OK, so is the machine you're running free -m on the same as the one
running postgresql and the same one you're running cron jobs on and
the same one you're running apache on?
Also please don't remove the cc for the list, others might have an
insight I'd miss.
> cif@domU-12-31-39-08-06-20:~$ free -m
> total used free shared buffers cached
> Mem: 17079 12051 5028 0 270 9578
> -/+ buffers/cache: 2202 14877
> Swap: 511 0 511
>
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [GENERAL] Memory issues
@ 2012-09-26 21:00 Shiran Kleiderman <shirank1@gmail.com>
parent: Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Shiran Kleiderman @ 2012-09-26 21:00 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-performance
Hi
Yes, same machine.
Thanks for your help.
On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:
> On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com>
> wrote:
> > Hi
> > Thanks again.
> > Right now, this is free -m and ps aux and non of the crons can run -
> can't
> > allocate memory.
>
> OK, so is the machine you're running free -m on the same as the one
> running postgresql and the same one you're running cron jobs on and
> the same one you're running apache on?
>
> Also please don't remove the cc for the list, others might have an
> insight I'd miss.
>
> > cif@domU-12-31-39-08-06-20:~$ free -m
> > total used free shared buffers cached
> > Mem: 17079 12051 5028 0 270 9578
> > -/+ buffers/cache: 2202 14877
> > Swap: 511 0 511
> >
>
--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [GENERAL] Memory issues
@ 2012-09-26 21:11 Scott Marlowe <scott.marlowe@gmail.com>
parent: Shiran Kleiderman <shirank1@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Scott Marlowe @ 2012-09-26 21:11 UTC (permalink / raw)
To: Shiran Kleiderman <shirank1@gmail.com>; +Cc: pgsql-performance
OK then I'm lost. It's got to either be a bug in how amazon ec2
instances work or severely fragmented memory because you've got a TON
of kernel cache available.
On Wed, Sep 26, 2012 at 3:00 PM, Shiran Kleiderman <shirank1@gmail.com> wrote:
> Hi
> Yes, same machine.
>
> Thanks for your help.
>
>
> On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <scott.marlowe@gmail.com>
> wrote:
>>
>> On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com>
>> wrote:
>> > Hi
>> > Thanks again.
>> > Right now, this is free -m and ps aux and non of the crons can run -
>> > can't
>> > allocate memory.
>>
>> OK, so is the machine you're running free -m on the same as the one
>> running postgresql and the same one you're running cron jobs on and
>> the same one you're running apache on?
>>
>> Also please don't remove the cc for the list, others might have an
>> insight I'd miss.
>>
>> > cif@domU-12-31-39-08-06-20:~$ free -m
>> > total used free shared buffers
>> > cached
>> > Mem: 17079 12051 5028 0 270
>> > 9578
>> > -/+ buffers/cache: 2202 14877
>> > Swap: 511 0 511
>> >
>
>
>
>
> --
> Best,
> Shiran Kleiderman
> +972 - 542380838
> Skype - shirank1
>
--
To understand recursion, one must first understand recursion.
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [GENERAL] Memory issues
@ 2012-09-26 21:36 Shiran Kleiderman <shirank1@gmail.com>
parent: Scott Marlowe <scott.marlowe@gmail.com>
0 siblings, 1 reply; 14+ messages in thread
From: Shiran Kleiderman @ 2012-09-26 21:36 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-performance
Hi
I contact amazon with this issue.
What can I check for the fragmented memory issue?
Thanks (:
On Wed, Sep 26, 2012 at 11:11 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:
> OK then I'm lost. It's got to either be a bug in how amazon ec2
> instances work or severely fragmented memory because you've got a TON
> of kernel cache available.
>
> On Wed, Sep 26, 2012 at 3:00 PM, Shiran Kleiderman <shirank1@gmail.com>
> wrote:
> > Hi
> > Yes, same machine.
> >
> > Thanks for your help.
> >
> >
> > On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <scott.marlowe@gmail.com
> >
> > wrote:
> >>
> >> On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <shirank1@gmail.com
> >
> >> wrote:
> >> > Hi
> >> > Thanks again.
> >> > Right now, this is free -m and ps aux and non of the crons can run -
> >> > can't
> >> > allocate memory.
> >>
> >> OK, so is the machine you're running free -m on the same as the one
> >> running postgresql and the same one you're running cron jobs on and
> >> the same one you're running apache on?
> >>
> >> Also please don't remove the cc for the list, others might have an
> >> insight I'd miss.
> >>
> >> > cif@domU-12-31-39-08-06-20:~$ free -m
> >> > total used free shared buffers
> >> > cached
> >> > Mem: 17079 12051 5028 0 270
> >> > 9578
> >> > -/+ buffers/cache: 2202 14877
> >> > Swap: 511 0 511
> >> >
> >
> >
> >
> >
> > --
> > Best,
> > Shiran Kleiderman
> > +972 - 542380838
> > Skype - shirank1
> >
>
>
>
> --
> To understand recursion, one must first understand recursion.
>
--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: [GENERAL] Memory issues
@ 2012-09-26 22:01 Shiran Kleiderman <shirank1@gmail.com>
parent: Shiran Kleiderman <shirank1@gmail.com>
0 siblings, 0 replies; 14+ messages in thread
From: Shiran Kleiderman @ 2012-09-26 22:01 UTC (permalink / raw)
To: Scott Marlowe <scott.marlowe@gmail.com>; +Cc: pgsql-performance
Hi
Another thing that may help,
When I restart the postgres db, then I have a little bit of "grace" time
(the memory free field is also lifted a bit).
I can run the crons and then after an hour or two the status returns to
regular... "out of memory" errors.
Thanks!
On Wed, Sep 26, 2012 at 11:36 PM, Shiran Kleiderman <shirank1@gmail.com>wrote:
> Hi
> I contact amazon with this issue.
> What can I check for the fragmented memory issue?
>
> Thanks (:
>
>
> On Wed, Sep 26, 2012 at 11:11 PM, Scott Marlowe <scott.marlowe@gmail.com>wrote:
>
>> OK then I'm lost. It's got to either be a bug in how amazon ec2
>> instances work or severely fragmented memory because you've got a TON
>> of kernel cache available.
>>
>> On Wed, Sep 26, 2012 at 3:00 PM, Shiran Kleiderman <shirank1@gmail.com>
>> wrote:
>> > Hi
>> > Yes, same machine.
>> >
>> > Thanks for your help.
>> >
>> >
>> > On Wed, Sep 26, 2012 at 10:55 PM, Scott Marlowe <
>> scott.marlowe@gmail.com>
>> > wrote:
>> >>
>> >> On Wed, Sep 26, 2012 at 10:41 AM, Shiran Kleiderman <
>> shirank1@gmail.com>
>> >> wrote:
>> >> > Hi
>> >> > Thanks again.
>> >> > Right now, this is free -m and ps aux and non of the crons can run -
>> >> > can't
>> >> > allocate memory.
>> >>
>> >> OK, so is the machine you're running free -m on the same as the one
>> >> running postgresql and the same one you're running cron jobs on and
>> >> the same one you're running apache on?
>> >>
>> >> Also please don't remove the cc for the list, others might have an
>> >> insight I'd miss.
>> >>
>> >> > cif@domU-12-31-39-08-06-20:~$ free -m
>> >> > total used free shared buffers
>> >> > cached
>> >> > Mem: 17079 12051 5028 0 270
>> >> > 9578
>> >> > -/+ buffers/cache: 2202 14877
>> >> > Swap: 511 0 511
>> >> >
>> >
>> >
>> >
>> >
>> > --
>> > Best,
>> > Shiran Kleiderman
>> > +972 - 542380838
>> > Skype - shirank1
>> >
>>
>>
>>
>> --
>> To understand recursion, one must first understand recursion.
>>
>
>
>
> --
> Best,
> Shiran Kleiderman
> +972 - 542380838
> Skype - shirank1
>
>
--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Memory issues
@ 2012-09-27 06:59 Andres Freund <andres@2ndquadrant.com>
parent: Shiran Kleiderman <shirank1@gmail.com>
1 sibling, 1 reply; 14+ messages in thread
From: Andres Freund @ 2012-09-27 06:59 UTC (permalink / raw)
To: pgsql-general@postgresql.org; +Cc: Shiran Kleiderman <shirank1@gmail.com>
On Monday, September 24, 2012 08:45:06 AM Shiran Kleiderman wrote:
> Hi,
> I'm using and Amazon ec2 instance with the following spec and the
> application that I'm running uses a postgres DB 9.1.
> The app has 3 main cron jobs.
>
> *Ubuntu 12, High-Memory Extra Large Instance
> 17.1 GB of memory
> 6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
> 420 GB of instance storage
> 64-bit platform*
>
> I've changed the main default values under file *postgresql.conf* to:
> shared_buffers = 4GB
> work_mem = 16MB
> wal_buffers = 16MB
> checkpoint_segments = 32
> effective_cache_size = 8GB
>
> When I run the app, after an hour or two, free -m looks like below ans the
> crons can't run due to memory loss or similar (i'm new to postgres and db
> admin).
> Thanks!
>
> free -m, errors:
>
> total used free shared buffers cached
> Mem: 17079 13742 3337 0 64 11882
> -/+ buffers/cache: 1796 15283
> Swap: 511 0 511
>
> total used *free* shared buffers cached
> Mem: 17079 16833 *245 *0 42 14583
> -/+ buffers/cache: 2207 14871
> Swap: 511 0 511
>
> **free above stays low even when nothing is running.
>
> **errors:
> *DBI connect('database=---;host=localhost','postgres',...) failed: could
> not fork new process for connection: Cannot allocate memory*
> could not fork new process for connection: Cannot allocate memory
>
> and
> execute failed: ERROR: out of memory
> DETAIL: Failed on request of size 968. [for Statement "
> SELECT DISTINCT....
could you show cat /proc/meminfo?
Greetings,
Andres
--
Andres Freund http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Memory issues
@ 2012-10-15 00:45 Shiran Kleiderman <shirank1@gmail.com>
parent: Andres Freund <andres@2ndquadrant.com>
0 siblings, 1 reply; 14+ messages in thread
From: Shiran Kleiderman @ 2012-10-15 00:45 UTC (permalink / raw)
To: Andres Freund <andres@2ndquadrant.com>; +Cc: pgsql-general@postgresql.org
Hi
I've returned the memory configs to the default, erased data from my db and
am testing the system again.
This is the output of *cat /proc/meminfo*
Thanks
root@ip-10-194-167-240:~# cat /proc/meminfo
MemTotal: 7629508 kB
MemFree: 170368 kB
Buffers: 10272 kB
Cached: 6220848 kB
SwapCached: 0 kB
Active: 3249748 kB
Inactive: 3936960 kB
Active(anon): 971336 kB
Inactive(anon): 2103844 kB
Active(file): 2278412 kB
Inactive(file): 1833116 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 524284 kB
SwapFree: 522716 kB
Dirty: 83068 kB
Writeback: 3080 kB
AnonPages: 955856 kB
Mapped: 2132564 kB
Shmem: 2119424 kB
Slab: 157200 kB
SReclaimable: 144488 kB
SUnreclaim: 12712 kB
KernelStack: 1184 kB
PageTables: 21092 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4339036 kB
Committed_AS: 3637424 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 26152 kB
VmallocChunk: 34359710052 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7872512 kB
DirectMap2M: 0 kB
On Thu, Sep 27, 2012 at 8:59 AM, Andres Freund <andres@2ndquadrant.com>wrote:
> On Monday, September 24, 2012 08:45:06 AM Shiran Kleiderman wrote:
> > Hi,
> > I'm using and Amazon ec2 instance with the following spec and the
> > application that I'm running uses a postgres DB 9.1.
> > The app has 3 main cron jobs.
> >
> > *Ubuntu 12, High-Memory Extra Large Instance
> > 17.1 GB of memory
> > 6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
> > 420 GB of instance storage
> > 64-bit platform*
> >
> > I've changed the main default values under file *postgresql.conf* to:
> > shared_buffers = 4GB
> > work_mem = 16MB
> > wal_buffers = 16MB
> > checkpoint_segments = 32
> > effective_cache_size = 8GB
> >
> > When I run the app, after an hour or two, free -m looks like below ans
> the
> > crons can't run due to memory loss or similar (i'm new to postgres and db
> > admin).
> > Thanks!
> >
> > free -m, errors:
> >
> > total used free shared buffers cached
> > Mem: 17079 13742 3337 0 64 11882
> > -/+ buffers/cache: 1796 15283
> > Swap: 511 0 511
> >
> > total used *free* shared buffers cached
> > Mem: 17079 16833 *245 *0 42 14583
> > -/+ buffers/cache: 2207 14871
> > Swap: 511 0 511
> >
> > **free above stays low even when nothing is running.
> >
> > **errors:
> > *DBI connect('database=---;host=localhost','postgres',...) failed: could
> > not fork new process for connection: Cannot allocate memory*
> > could not fork new process for connection: Cannot allocate memory
> >
> > and
> > execute failed: ERROR: out of memory
> > DETAIL: Failed on request of size 968. [for Statement "
> > SELECT DISTINCT....
> could you show cat /proc/meminfo?
>
> Greetings,
>
> Andres
> --
> Andres Freund http://www.2ndQuadrant.com/
> PostgreSQL Development, 24x7 Support, Training & Services
>
--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1
^ permalink raw reply [nested|flat] 14+ messages in thread
* Re: Memory issues
@ 2012-10-15 02:36 Shiran Kleiderman <shirank1@gmail.com>
parent: Shiran Kleiderman <shirank1@gmail.com>
0 siblings, 0 replies; 14+ messages in thread
From: Shiran Kleiderman @ 2012-10-15 02:36 UTC (permalink / raw)
To: Andres Freund <andres@2ndquadrant.com>; +Cc: pgsql-general@postgresql.org
Hi
This is the output of meminfo when the system is under some stress.
Thanks
cif@ip-10-194-167-240:/tmp$ cat /proc/meminfo
MemTotal: 7629508 kB
MemFree: 37820 kB
Buffers: 2108 kB
Cached: 5500200 kB
SwapCached: 332 kB
Active: 4172020 kB
Inactive: 3166244 kB
Active(anon): 1864040 kB
Inactive(anon): 1568760 kB
Active(file): 2307980 kB
Inactive(file): 1597484 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 524284 kB
SwapFree: 0 kB
Dirty: 23336 kB
Writeback: 0 kB
AnonPages: 1835716 kB
Mapped: 1610460 kB
Shmem: 1596916 kB
Slab: 136168 kB
SReclaimable: 123820 kB
SUnreclaim: 12348 kB
KernelStack: 1176 kB
PageTables: 23148 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 4339036 kB
Committed_AS: 4517524 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 26152 kB
VmallocChunk: 34359710052 kB
HardwareCorrupted: 0 kB
AnonHugePages: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
DirectMap4k: 7872512 kB
DirectMap2M: 0 kB
On Mon, Oct 15, 2012 at 2:45 AM, Shiran Kleiderman <shirank1@gmail.com>wrote:
> Hi
> I've returned the memory configs to the default, erased data from my db
> and am testing the system again.
>
> This is the output of *cat /proc/meminfo*
> Thanks
>
> root@ip-10-194-167-240:~# cat /proc/meminfo
> MemTotal: 7629508 kB
> MemFree: 170368 kB
> Buffers: 10272 kB
> Cached: 6220848 kB
> SwapCached: 0 kB
> Active: 3249748 kB
> Inactive: 3936960 kB
> Active(anon): 971336 kB
> Inactive(anon): 2103844 kB
> Active(file): 2278412 kB
> Inactive(file): 1833116 kB
> Unevictable: 0 kB
> Mlocked: 0 kB
> SwapTotal: 524284 kB
> SwapFree: 522716 kB
> Dirty: 83068 kB
> Writeback: 3080 kB
> AnonPages: 955856 kB
> Mapped: 2132564 kB
> Shmem: 2119424 kB
> Slab: 157200 kB
> SReclaimable: 144488 kB
> SUnreclaim: 12712 kB
> KernelStack: 1184 kB
> PageTables: 21092 kB
> NFS_Unstable: 0 kB
> Bounce: 0 kB
> WritebackTmp: 0 kB
> CommitLimit: 4339036 kB
> Committed_AS: 3637424 kB
> VmallocTotal: 34359738367 kB
> VmallocUsed: 26152 kB
> VmallocChunk: 34359710052 kB
> HardwareCorrupted: 0 kB
> AnonHugePages: 0 kB
> HugePages_Total: 0
> HugePages_Free: 0
> HugePages_Rsvd: 0
> HugePages_Surp: 0
> Hugepagesize: 2048 kB
> DirectMap4k: 7872512 kB
> DirectMap2M: 0 kB
>
>
> On Thu, Sep 27, 2012 at 8:59 AM, Andres Freund <andres@2ndquadrant.com>wrote:
>
>> On Monday, September 24, 2012 08:45:06 AM Shiran Kleiderman wrote:
>> > Hi,
>> > I'm using and Amazon ec2 instance with the following spec and the
>> > application that I'm running uses a postgres DB 9.1.
>> > The app has 3 main cron jobs.
>> >
>> > *Ubuntu 12, High-Memory Extra Large Instance
>> > 17.1 GB of memory
>> > 6.5 EC2 Compute Units (2 virtual cores with 3.25 EC2 Compute Units each)
>> > 420 GB of instance storage
>> > 64-bit platform*
>> >
>> > I've changed the main default values under file *postgresql.conf* to:
>> > shared_buffers = 4GB
>> > work_mem = 16MB
>> > wal_buffers = 16MB
>> > checkpoint_segments = 32
>> > effective_cache_size = 8GB
>> >
>> > When I run the app, after an hour or two, free -m looks like below ans
>> the
>> > crons can't run due to memory loss or similar (i'm new to postgres and
>> db
>> > admin).
>> > Thanks!
>> >
>> > free -m, errors:
>> >
>> > total used free shared buffers cached
>> > Mem: 17079 13742 3337 0 64 11882
>> > -/+ buffers/cache: 1796 15283
>> > Swap: 511 0 511
>> >
>> > total used *free* shared buffers cached
>> > Mem: 17079 16833 *245 *0 42 14583
>> > -/+ buffers/cache: 2207 14871
>> > Swap: 511 0 511
>> >
>> > **free above stays low even when nothing is running.
>> >
>> > **errors:
>> > *DBI connect('database=---;host=localhost','postgres',...) failed: could
>> > not fork new process for connection: Cannot allocate memory*
>> > could not fork new process for connection: Cannot allocate memory
>> >
>> > and
>> > execute failed: ERROR: out of memory
>> > DETAIL: Failed on request of size 968. [for Statement "
>> > SELECT DISTINCT....
>> could you show cat /proc/meminfo?
>>
>> Greetings,
>>
>> Andres
>> --
>> Andres Freund http://www.2ndQuadrant.com/
>> PostgreSQL Development, 24x7 Support, Training & Services
>>
>
>
>
> --
> Best,
> Shiran Kleiderman
> +972 - 542380838
> Skype - shirank1
>
>
--
Best,
Shiran Kleiderman
+972 - 542380838
Skype - shirank1
^ permalink raw reply [nested|flat] 14+ messages in thread
end of thread, other threads:[~2012-10-15 02:36 UTC | newest]
Thread overview: 14+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2012-09-24 06:41 Memory issues Shiran Kleiderman <shirank1@gmail.com>
2012-09-24 06:45 Memory issues Shiran Kleiderman <shirank1@gmail.com>
2012-09-26 00:56 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-09-26 01:00 ` Shiran Kleiderman <shirank1@gmail.com>
2012-09-26 16:29 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-09-26 16:41 ` Shiran Kleiderman <shirank1@gmail.com>
2012-09-26 20:55 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-09-26 21:00 ` Shiran Kleiderman <shirank1@gmail.com>
2012-09-26 21:11 ` Scott Marlowe <scott.marlowe@gmail.com>
2012-09-26 21:36 ` Shiran Kleiderman <shirank1@gmail.com>
2012-09-26 22:01 ` Shiran Kleiderman <shirank1@gmail.com>
2012-09-27 06:59 ` Andres Freund <andres@2ndquadrant.com>
2012-10-15 00:45 ` Shiran Kleiderman <shirank1@gmail.com>
2012-10-15 02:36 ` Shiran Kleiderman <shirank1@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox