pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedwork_mem
22+ messages / 10 participants
[nested] [flat]
* work_mem
@ 2021-03-31 14:04 Campbell, Lance <lance@illinois.edu>
0 siblings, 2 replies; 22+ messages in thread
From: Campbell, Lance @ 2021-03-31 14:04 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
PostgreSQL 12
Is there a query that will tell us at any given time what amount of PostgreSQL memory is being used for work_mem?
Thanks,
LANCE CAMPBELL<https://directory.illinois.edu/person/lance;
Web Services<https://webtools.illinois.edu/;
Public Affairs<https://publicaffairs.illinois.edu/;
[/var/folders/wp/1f6l7hw95y718z976kgnl5f9kr5rtc/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/signature_logo.png]<http://illinois.edu/;
Under the Illinois Freedom of Information Act any written communication to or from university employees regarding university business is a public record and may be subject to public disclosure.
Attachments:
[image/png] image001.png (2.5K, ../../A2AA6D08-9506-4B41-9738-4D5FF8E2E011@illinois.edu/3-image001.png)
download | view image
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-03-31 14:17 Bruce Momjian <bruce@momjian.us>
parent: Campbell, Lance <lance@illinois.edu>
1 sibling, 0 replies; 22+ messages in thread
From: Bruce Momjian @ 2021-03-31 14:17 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; +Cc: pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
On Wed, Mar 31, 2021 at 02:04:07PM +0000, Campbell, Lance wrote:
> PostgreSQL 12
>
>
>
> Is there a query that will tell us at any given time what amount of PostgreSQL
> memory is being used for work_mem?
Well, you can look at the process memory usage via 'ps', but I don't
know a way to see current work_mem allocation. SHOW work_mem does show
the current setting, but that isn't the allocated amount.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-03-31 15:22 Holger Jakobs <holger@jakobs.com>
parent: Campbell, Lance <lance@illinois.edu>
1 sibling, 2 replies; 22+ messages in thread
From: Holger Jakobs @ 2021-03-31 15:22 UTC (permalink / raw)
To: pgsql-admin@lists.postgresql.org
You can show any or all settings with the command SHOW.
show work_mem;
show all;
How many times the amount of work_mem has been allocated currently cannot be shown this way. Any sort operation running currently allocates it. So it is not identical with the number of current queries, because queries can allocate it several times.
Am 31. März 2021 16:04:07 MESZ schrieb "Campbell, Lance" <lance@illinois.edu>:
>PostgreSQL 12
>
>Is there a query that will tell us at any given time what amount of
>PostgreSQL memory is being used for work_mem?
>
>Thanks,
>
>LANCE CAMPBELL<https://directory.illinois.edu/person/lance;
>
>Web Services<https://webtools.illinois.edu/;
>Public Affairs<https://publicaffairs.illinois.edu/;
>
>
>[/var/folders/wp/1f6l7hw95y718z976kgnl5f9kr5rtc/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/signature_logo.png]<http://illinois.edu/;
>
>Under the Illinois Freedom of Information Act any written communication
>to or from university employees regarding university business is a
>public record and may be subject to public disclosure.
--
Holger Jakobs, Bergisch Gladbach
+49 178 9759012
- sent from mobile, therefore short -
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-01 07:43 Sachin Kumar <sachinkumaras@gmail.com>
parent: Holger Jakobs <holger@jakobs.com>
1 sibling, 0 replies; 22+ messages in thread
From: Sachin Kumar @ 2021-04-01 07:43 UTC (permalink / raw)
To: Holger Jakobs <holger@jakobs.com>; +Cc: pgsql-admin@lists.postgresql.org
Hi Mr. Holger,
what is the setting that needs to be changed in the PostgreSQL 13
configuration to use it more efficiently?
I had just updated work_mem TO '256MB' my PC RAM is 32 GB; anything else we
need to do to make POSTGRESQL performance better.
Please advise in detail it will help all PostgreSQL users.
Thanks in advance
Sachin Kumar
On Wed, Mar 31, 2021 at 8:52 PM Holger Jakobs <holger@jakobs.com> wrote:
> You can show any or all settings with the command SHOW.
>
> show work_mem;
> show all;
>
> How many times the amount of work_mem has been allocated currently cannot
> be shown this way. Any sort operation running currently allocates it. So it
> is not identical with the number of current queries, because queries can
> allocate it several times.
>
>
> Am 31. März 2021 16:04:07 MESZ schrieb "Campbell, Lance" <
> lance@illinois.edu>:
>>
>> PostgreSQL 12
>>
>>
>>
>> Is there a query that will tell us at any given time what amount of
>> PostgreSQL memory is being used for work_mem?
>>
>>
>>
>> Thanks,
>>
>>
>>
>> *LANCE CAMPBELL <https://directory.illinois.edu/person/lance>*
>>
>>
>>
>> Web Services <https://webtools.illinois.edu/;
>>
>> Public Affairs <https://publicaffairs.illinois.edu/;
>>
>>
>>
>>
>>
>> [image:
>> /var/folders/wp/1f6l7hw95y718z976kgnl5f9kr5rtc/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/signature_logo.png]
>> <http://illinois.edu/;
>>
>>
>>
>> *Under the Illinois Freedom of Information Act any written communication
>> to or from university employees regarding university business is a public
>> record and may be subject to public disclosure.*
>>
>>
>>
>
> --
> Holger Jakobs, Bergisch Gladbach
> +49 178 9759012
> - sent from mobile, therefore short -
>
--
Best Regards,
Sachin Kumar
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 12:37 SASIKUMAR Devaraj <sashikumard@yahoo.com>
parent: Holger Jakobs <holger@jakobs.com>
1 sibling, 1 reply; 22+ messages in thread
From: SASIKUMAR Devaraj @ 2021-04-02 12:37 UTC (permalink / raw)
To: holger@jakobs.com, Holger Jakobs <holger@jakobs.com>; pgsql-admin@lists.postgresql.org
Hi All
As soon as client session is established work_mem will be allocated or only when sort happens for that particular session? Please advise how the internal behavior? This will help me to configure my database memory with high connections
Regards
Sasi
Sent from Yahoo Mail on Android
On Wed, Mar 31, 2021 at 8:52 PM, Holger Jakobs<holger@jakobs.com> wrote: #yiv2643575322 #yiv2643575322 -- _filtered {} _filtered {} _filtered {}#yiv2643575322 #yiv2643575322 p.yiv2643575322MsoNormal, #yiv2643575322 li.yiv2643575322MsoNormal, #yiv2643575322 div.yiv2643575322MsoNormal {margin:0in;font-size:12.0pt;font-family:sans-serif;}#yiv2643575322 span.yiv2643575322EmailStyle17 {font-family:sans-serif;color:windowtext;}#yiv2643575322 .yiv2643575322MsoChpDefault {font-size:12.0pt;font-family:sans-serif;} _filtered {}#yiv2643575322 div.yiv2643575322WordSection1 {}#yiv2643575322 You can show any or all settings with the command SHOW.
show work_mem;
show all;
How many times the amount of work_mem has been allocated currently cannot be shown this way. Any sort operation running currently allocates it. So it is not identical with the number of current queries, because queries can allocate it several times.
Am 31. März 2021 16:04:07 MESZ schrieb "Campbell, Lance" <lance@illinois.edu>:
PostgreSQL 12
Is there a query that will tell us at any given time what amount of PostgreSQL memory is being used for work_mem?
Thanks,
LANCE CAMPBELL
Web Services
Public Affairs
Under the Illinois Freedom of Information Act any written communication to or from university employees regarding university business is a public record and may be subject to public disclosure.
--
Holger Jakobs, Bergisch Gladbach
+49 178 9759012
- sent from mobile, therefore short -
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 12:39 MichaelDBA <MichaelDBA@sqlexec.com>
parent: SASIKUMAR Devaraj <sashikumard@yahoo.com>
0 siblings, 1 reply; 22+ messages in thread
From: MichaelDBA @ 2021-04-02 12:39 UTC (permalink / raw)
To: SASIKUMAR Devaraj <sashikumard@yahoo.com>; +Cc: holger@jakobs.com; pgsql-admin@lists.postgresql.org
Memory is allocated dynamically per internal work_mem buffer requests.
SASIKUMAR Devaraj wrote on 4/2/2021 8:37 AM:
> Hi All
>
> As soon as client session is established work_mem will be allocated or
> only when sort happens for that particular session? Please advise how
> the internal behavior? This will help me to configure my database
> memory with high connections
>
> Regards
>
> Sasi
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__Android...;
>
> On Wed, Mar 31, 2021 at 8:52 PM, Holger Jakobs
> <holger@jakobs.com> wrote:
> You can show any or all settings with the command SHOW.
>
> show work_mem;
> show all;
>
> How many times the amount of work_mem has been allocated currently
> cannot be shown this way. Any sort operation running currently
> allocates it. So it is not identical with the number of current
> queries, because queries can allocate it several times.
>
>
> Am 31. März 2021 16:04:07 MESZ schrieb "Campbell, Lance"
> <lance@illinois.edu>:
>
> PostgreSQL 12
>
> Is there a query that will tell us at any given time what
> amount of PostgreSQL memory is being used for work_mem?
>
> Thanks,
>
> *LANCE CAMPBELL <https://directory.illinois.edu/person/lance>*
>
> Web Services <https://webtools.illinois.edu/;
>
> Public Affairs <https://publicaffairs.illinois.edu/;
>
> /var/folders/wp/1f6l7hw95y718z976kgnl5f9kr5rtc/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/signature_logo.png
> <http://illinois.edu/;
>
> /Under the Illinois Freedom of Information Act any written
> communication to or from university employees regarding
> university business is a public record and may be subject to
> public disclosure./
>
>
> --
> Holger Jakobs, Bergisch Gladbach
> +49 178 9759012
> - sent from mobile, therefore short -
>
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 12:45 SASIKUMAR Devaraj <sashikumard@yahoo.com>
parent: MichaelDBA <MichaelDBA@sqlexec.com>
0 siblings, 1 reply; 22+ messages in thread
From: SASIKUMAR Devaraj @ 2021-04-02 12:45 UTC (permalink / raw)
To: MichaelDBA@sqlexec.com, MichaelDBA <MichaelDBA@sqlexec.com>; +Cc: holger@jakobs.com; pgsql-admin@lists.postgresql.org
Thanks Michael
For example work_mem is 4M and if I had 300 connections connected to dB the total memory requirement is 1.2 Gb. So as per my understanding this 1.2 gb is not allocated as soon as 300 connections established, but it may vary from 0 to 1.2gb as per operations from client. Please confirm
Regards
Sasi
Sent from Yahoo Mail on Android
On Fri, Apr 2, 2021 at 6:09 PM, MichaelDBA<MichaelDBA@sqlexec.com> wrote: Memory is allocated dynamically per internal work_mem buffer requests.
SASIKUMAR Devaraj wrote on 4/2/2021 8:37 AM:
Hi All
As soon as client session is established work_mem will be allocated or only when sort happens for that particular session? Please advise how the internal behavior? This will help me to configure my database memory with high connections
Regards
Sasi
Sent from Yahoo Mail on Android
On Wed, Mar 31, 2021 at 8:52 PM, Holger Jakobs<holger@jakobs.com> wrote: #yiv8729720470 -- filtered {}#yiv8729720470 filtered {}#yiv8729720470 filtered {}#yiv8729720470 p.yiv8729720470MsoNormal, #yiv8729720470 li.yiv8729720470MsoNormal, #yiv8729720470 div.yiv8729720470MsoNormal {margin:0in;font-size:12.0pt;font-family:sans-serif;}#yiv8729720470 span.yiv8729720470EmailStyle17 {font-family:sans-serif;color:windowtext;}#yiv8729720470 .yiv8729720470MsoChpDefault {font-size:12.0pt;font-family:sans-serif;}#yiv8729720470 filtered {}#yiv8729720470 div.yiv8729720470WordSection1 {}#yiv8729720470 You can show any or all settings with the command SHOW.
show work_mem;
show all;
How many times the amount of work_mem has been allocated currently cannot be shown this way. Any sort operation running currently allocates it. So it is not identical with the number of current queries, because queries can allocate it several times.
Am 31. März 2021 16:04:07 MESZ schrieb "Campbell, Lance" <lance@illinois.edu>:
PostgreSQL 12
Is there a query that will tell us at any given time what amount of PostgreSQL memory is being used for work_mem?
Thanks,
LANCE CAMPBELL
Web Services
Public Affairs
Under the Illinois Freedom of Information Act any written communication to or from university employees regarding university business is a public record and may be subject to public disclosure.
--
Holger Jakobs, Bergisch Gladbach
+49 178 9759012
- sent from mobile, therefore short -
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 12:49 MichaelDBA <MichaelDBA@sqlexec.com>
parent: SASIKUMAR Devaraj <sashikumard@yahoo.com>
0 siblings, 1 reply; 22+ messages in thread
From: MichaelDBA @ 2021-04-02 12:49 UTC (permalink / raw)
To: SASIKUMAR Devaraj <sashikumard@yahoo.com>; +Cc: holger@jakobs.com; pgsql-admin@lists.postgresql.org
That is a common misconception. It is not one work_mem buffer per SQL,
but one work_mem buffer per required operation within that SQL. So you
can have manner work_mem buffers per SQL statement!
Right from the official docs:
work_mem sets the maximum amount of memory to be used by a query
operation (such as a sort or hash table) before writing to temporary
disk files. If this value is specified without units, it is taken as
kilobytes...Note that for a complex query, several sort or hash
operations might be running in parallel; each operation will be allowed
to use as much memory as this value specifies before it starts to write
data into temporary files. Also, several running sessions could be doing
such operations concurrently. Therefore, the total memory used could be
many times the value of|work_mem|; it is necessary to keep this fact in
mind when choosing the value. Sort operations are used for|ORDER
BY|,|DISTINCT|, and merge joins. Hash tables are used in hash joins,
hash-based aggregation, and hash-based processing of|IN|subqueries.
Regards,
Michael Vitale
SASIKUMAR Devaraj wrote on 4/2/2021 8:45 AM:
> Thanks Michael
>
> For example work_mem is 4M and if I had 300 connections connected to
> dB the total memory requirement is 1.2 Gb.
> So as per my understanding this 1.2 gb is not allocated as soon as
> 300 connections established, but it may vary from 0 to 1.2gb as per
> operations from client. Please confirm
>
> Regards
>
> Sasi
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__Android...;
>
> On Fri, Apr 2, 2021 at 6:09 PM, MichaelDBA
> <MichaelDBA@sqlexec.com> wrote:
> Memory is allocated dynamically per internal work_mem buffer
> requests.
>
> SASIKUMAR Devaraj wrote on 4/2/2021 8:37 AM:
> Hi All
>
> As soon as client session is established work_mem will be
> allocated or only when sort happens for that particular session?
> Please advise how the internal behavior? This will help me to
> configure my database memory with high connections
>
> Regards
>
> Sasi
>
> Sent from Yahoo Mail on Android
> <https://go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrowth_AndroidEmailSig__Android...;
>
> On Wed, Mar 31, 2021 at 8:52 PM, Holger Jakobs
> <holger@jakobs.com> <mailto:holger@jakobs.com> wrote:
> You can show any or all settings with the command SHOW.
>
> show work_mem;
> show all;
>
> How many times the amount of work_mem has been allocated
> currently cannot be shown this way. Any sort operation running
> currently allocates it. So it is not identical with the number
> of current queries, because queries can allocate it several times.
>
>
> Am 31. März 2021 16:04:07 MESZ schrieb "Campbell, Lance"
> <lance@illinois.edu> <mailto:lance@illinois.edu>:
>
> PostgreSQL 12
>
> Is there a query that will tell us at any given time what
> amount of PostgreSQL memory is being used for work_mem?
>
> Thanks,
>
> *LANCE CAMPBELL <https://directory.illinois.edu/person/lance>*
>
> Web Services <https://webtools.illinois.edu/;
>
> Public Affairs <https://publicaffairs.illinois.edu/;
>
> /var/folders/wp/1f6l7hw95y718z976kgnl5f9kr5rtc/T/com.microsoft.Outlook/WebArchiveCopyPasteTempFiles/signature_logo.png
> <http://illinois.edu/;
>
> /Under the Illinois Freedom of Information Act any written
> communication to or from university employees regarding
> university business is a public record and may be subject
> to public disclosure./
>
>
> --
> Holger Jakobs, Bergisch Gladbach
> +49 178 9759012
> - sent from mobile, therefore short -
>
>
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 13:31 Campbell, Lance <lance@illinois.edu>
parent: MichaelDBA <MichaelDBA@sqlexec.com>
0 siblings, 1 reply; 22+ messages in thread
From: Campbell, Lance @ 2021-04-02 13:31 UTC (permalink / raw)
To: MichaelDBA <MichaelDBA@sqlexec.com>; SASIKUMAR Devaraj <sashikumard@yahoo.com>; +Cc: holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
It feels like there needs to be work_mem and work_mem_stack_size. When work memory is needed a process “pops” a token off of a stack. When it is done processing it “puts” the token back on the stack. If the stack is empty then don’t allocate memory just write to disk for work_mem.
This does two key things:
1) It allows for a real world understanding of how much memory is really needed on a day to day basis. You can track how often a stack is empty. You can also look at the number of temp files to see when work exceeds the work_mem allocation. There is no “art” to setting these values. You can use logical analysis to make choices.
2) This also prevents out of memory issues. You are protecting yourself from extreme loads.
Lance
From: MichaelDBA <MichaelDBA@sqlexec.com>
Date: Friday, April 2, 2021 at 7:50 AM
To: SASIKUMAR Devaraj <sashikumard@yahoo.com>
Cc: holger@jakobs.com <holger@jakobs.com>, pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Subject: Re: work_mem
That is a common misconception. It is not one work_mem buffer per SQL, but one work_mem buffer per required operation within that SQL. So you can have manner work_mem buffers per SQL statement!
Right from the official docs:
work_mem sets the maximum amount of memory to be used by a query operation (such as a sort or hash table) before writing to temporary disk files. If this value is specified without units, it is taken as kilobytes... Note that for a complex query, several sort or hash operations might be running in parallel; each operation will be allowed to use as much memory as this value specifies before it starts to write data into temporary files. Also, several running sessions could be doing such operations concurrently. Therefore, the total memory used could be many times the value of work_mem; it is necessary to keep this fact in mind when choosing the value. Sort operations are used for ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash joins, hash-based aggregation, and hash-based processing of IN subqueries.
Regards,
Michael Vitale
SASIKUMAR Devaraj wrote on 4/2/2021 8:45 AM:
Thanks Michael
For example work_mem is 4M and if I had 300 connections connected to dB the total memory requirement is 1.2 Gb.
So as per my understanding this 1.2 gb is not allocated as soon as 300 connections established, but it may vary from 0 to 1.2gb as per operations from client. Please confirm
Regards
Sasi
Sent from Yahoo Mail on Android<https://urldefense.com/v3/__https:/go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrow...;
On Fri, Apr 2, 2021 at 6:09 PM, MichaelDBA
<MichaelDBA@sqlexec.com><mailto:MichaelDBA@sqlexec.com> wrote:
Memory is allocated dynamically per internal work_mem buffer requests.
SASIKUMAR Devaraj wrote on 4/2/2021 8:37 AM:
Hi All
As soon as client session is established work_mem will be allocated or only when sort happens for that particular session? Please advise how the internal behavior? This will help me to configure my database memory with high connections
Regards
Sasi
Sent from Yahoo Mail on Android<https://urldefense.com/v3/__https:/go.onelink.me/107872968?pid=InProduct&c=Global_Internal_YGrow...;
On Wed, Mar 31, 2021 at 8:52 PM, Holger Jakobs
<holger@jakobs.com><mailto:holger@jakobs.com> wrote:
You can show any or all settings with the command SHOW.
show work_mem;
show all;
How many times the amount of work_mem has been allocated currently cannot be shown this way. Any sort operation running currently allocates it. So it is not identical with the number of current queries, because queries can allocate it several times.
Am 31. März 2021 16:04:07 MESZ schrieb "Campbell, Lance" <lance@illinois.edu><mailto:lance@illinois.edu>:
PostgreSQL 12
Is there a query that will tell us at any given time what amount of PostgreSQL memory is being used for work_mem?
Thanks,
LANCE CAMPBELL<https://directory.illinois.edu/person/lance;
Web Services<https://webtools.illinois.edu/;
Public Affairs<https://publicaffairs.illinois.edu/;
Error! Filename not specified.<http://illinois.edu/;
Under the Illinois Freedom of Information Act any written communication to or from university employees regarding university business is a public record and may be subject to public disclosure.
--
Holger Jakobs, Bergisch Gladbach
+49 178 9759012
- sent from mobile, therefore short -
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 14:59 Laurenz Albe <laurenz.albe@cybertec.at>
parent: Campbell, Lance <lance@illinois.edu>
0 siblings, 1 reply; 22+ messages in thread
From: Laurenz Albe @ 2021-04-02 14:59 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; MichaelDBA <MichaelDBA@sqlexec.com>; SASIKUMAR Devaraj <sashikumard@yahoo.com>; +Cc: holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
On Fri, 2021-04-02 at 13:31 +0000, Campbell, Lance wrote:
> It feels like there needs to be work_mem and work_mem_stack_size. When work memory is
> needed a process “pops” a token off of a stack. When it is done processing it “puts”
> the token back on the stack. If the stack is empty then don’t allocate memory just
> write to disk for work_mem.
>
> This does two key things:
>
> 1) It allows for a real world understanding of how much memory is really needed on a
> day to day basis. You can track how often a stack is empty. You can also look at the
> number of temp files to see when work exceeds the work_mem allocation. There is no
> “art” to setting these values. You can use logical analysis to make choices.
>
> 2) This also prevents out of memory issues. You are protecting yourself from extreme loads.
If I get you right, you want another memory limit per session.
I see the point, but then we wouldn't need "work_mem" any more, right?
What is the point of limiting the memory per plan node if we have an
overall limit?
In practice, I have never had trouble with "work_mem". I usually follow
my rule of thumb: max_connections * work_mem + shared_buffers < RAM
While some backend may need more, many will need less. Only bitmaps, hashes
and sorts are memory hungry.
Yours,
Laurenz Albe
--
Cybertec | https://www.cybertec-postgresql.com
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 15:07 Bruce Momjian <bruce@momjian.us>
parent: Laurenz Albe <laurenz.albe@cybertec.at>
0 siblings, 1 reply; 22+ messages in thread
From: Bruce Momjian @ 2021-04-02 15:07 UTC (permalink / raw)
To: Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: Campbell, Lance <lance@illinois.edu>; MichaelDBA <MichaelDBA@sqlexec.com>; SASIKUMAR Devaraj <sashikumard@yahoo.com>; holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
On Fri, Apr 2, 2021 at 04:59:16PM +0200, Laurenz Albe wrote:
> On Fri, 2021-04-02 at 13:31 +0000, Campbell, Lance wrote:
> > It feels like there needs to be work_mem and work_mem_stack_size. When work memory is
> > needed a process “pops” a token off of a stack. When it is done processing it “puts”
> > the token back on the stack. If the stack is empty then don’t allocate memory just
> > write to disk for work_mem.
> >
> > This does two key things:
> >
> > 1) It allows for a real world understanding of how much memory is really needed on a
> > day to day basis. You can track how often a stack is empty. You can also look at the
> > number of temp files to see when work exceeds the work_mem allocation. There is no
> > “art” to setting these values. You can use logical analysis to make choices.
> >
> > 2) This also prevents out of memory issues. You are protecting yourself from extreme loads.
>
> If I get you right, you want another memory limit per session.
>
> I see the point, but then we wouldn't need "work_mem" any more, right?
> What is the point of limiting the memory per plan node if we have an
> overall limit?
>
> In practice, I have never had trouble with "work_mem". I usually follow
> my rule of thumb: max_connections * work_mem + shared_buffers < RAM
>
> While some backend may need more, many will need less. Only bitmaps, hashes
> and sorts are memory hungry.
This blog entry discusses how work_mem might be improved:
https://momjian.us/main/blogs/pgblog/2018.html#December_10_2018
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 15:25 Campbell, Lance <lance@illinois.edu>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 22+ messages in thread
From: Campbell, Lance @ 2021-04-02 15:25 UTC (permalink / raw)
To: Bruce Momjian <bruce@momjian.us>; Laurenz Albe <laurenz.albe@cybertec.at>; +Cc: MichaelDBA <MichaelDBA@sqlexec.com>; SASIKUMAR Devaraj <sashikumard@yahoo.com>; holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Thanks for sharing this thread. My suggestion of having a work_mem_stack_size is the same concept mentioned in this thread regarding having a work_mem_pool. I prefer this later term rather than the one I was using. When the work mem pool is exhausted PostgreSQL just uses temp files for work_mem. With current statics for temp files and with a new stats on a work mem pool usage a user could fine tune memory much more precisely. It would leave the “art of memory tuning” behind. The other added benefit is that people would have a better understanding of how work_mem is used by naturally having to explain what a work_mem_pool is and when it is drawn on. There are probably a lot of PostgreSQL instance that would run faster just by having the confidence to increase the size of work_mem. I am sure many instances have this value set to low.
Lance
From: Bruce Momjian <bruce@momjian.us>
Date: Friday, April 2, 2021 at 10:07 AM
To: Laurenz Albe <laurenz.albe@cybertec.at>
Cc: Campbell, Lance <lance@illinois.edu>, MichaelDBA <MichaelDBA@sqlexec.com>, SASIKUMAR Devaraj <sashikumard@yahoo.com>, holger@jakobs.com <holger@jakobs.com>, pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Subject: Re: work_mem
On Fri, Apr 2, 2021 at 04:59:16PM +0200, Laurenz Albe wrote:
> On Fri, 2021-04-02 at 13:31 +0000, Campbell, Lance wrote:
> > It feels like there needs to be work_mem and work_mem_stack_size. When work memory is
> > needed a process “pops” a token off of a stack. When it is done processing it “puts”
> > the token back on the stack. If the stack is empty then don’t allocate memory just
> > write to disk for work_mem.
> >
> > This does two key things:
> >
> > 1) It allows for a real world understanding of how much memory is really needed on a
> > day to day basis. You can track how often a stack is empty. You can also look at the
> > number of temp files to see when work exceeds the work_mem allocation. There is no
> > “art” to setting these values. You can use logical analysis to make choices.
> >
> > 2) This also prevents out of memory issues. You are protecting yourself from extreme loads.
>
> If I get you right, you want another memory limit per session.
>
> I see the point, but then we wouldn't need "work_mem" any more, right?
> What is the point of limiting the memory per plan node if we have an
> overall limit?
>
> In practice, I have never had trouble with "work_mem". I usually follow
> my rule of thumb: max_connections * work_mem + shared_buffers < RAM
>
> While some backend may need more, many will need less. Only bitmaps, hashes
> and sorts are memory hungry.
This blog entry discusses how work_mem might be improved:
https://urldefense.com/v3/__https://momjian.us/main/blogs/pgblog/2018.html*December_10_2018__;Iw!!DZ...;
--
Bruce Momjian <bruce@momjian.us> https://urldefense.com/v3/__https://momjian.us__;!!DZ3fjg!ubflo-s4huK2u6qJsCnFu_At1slzkmzzjnkK5vqMOM...;
EDB https://urldefense.com/v3/__https://enterprisedb.com__;!!DZ3fjg!ubflo-s4huK2u6qJsCnFu_At1slzkmzzjnkK...;
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-02 15:34 Bruce Momjian <bruce@momjian.us>
parent: Campbell, Lance <lance@illinois.edu>
0 siblings, 1 reply; 22+ messages in thread
From: Bruce Momjian @ 2021-04-02 15:34 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; +Cc: Laurenz Albe <laurenz.albe@cybertec.at>; MichaelDBA <MichaelDBA@sqlexec.com>; SASIKUMAR Devaraj <sashikumard@yahoo.com>; holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
On Fri, Apr 2, 2021 at 03:25:04PM +0000, Campbell, Lance wrote:
> Thanks for sharing this thread. My suggestion of having a work_mem_stack_size
> is the same concept mentioned in this thread regarding having a work_mem_pool.
> I prefer this later term rather than the one I was using. When the work mem
> pool is exhausted PostgreSQL just uses temp files for work_mem. With current
> statics for temp files and with a new stats on a work mem pool usage a user
> could fine tune memory much more precisely. It would leave the “art of memory
> tuning” behind. The other added benefit is that people would have a better
> understanding of how work_mem is used by naturally having to explain what a
> work_mem_pool is and when it is drawn on. There are probably a lot of
> PostgreSQL instance that would run faster just by having the confidence to
> increase the size of work_mem. I am sure many instances have this value set to
> low.
Uh, did you read the blog before it, referenced in that blog entry:
https://momjian.us/main/blogs/pgblog/2018.html#December_7_2018
Even if we have a pool, it is still complex to configure memory, but it
might help.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-03 05:23 SASIKUMAR Devaraj <sashikumard@yahoo.com>
parent: Bruce Momjian <bruce@momjian.us>
0 siblings, 1 reply; 22+ messages in thread
From: SASIKUMAR Devaraj @ 2021-04-03 05:23 UTC (permalink / raw)
To: Campbell, Lance <lance@illinois.edu>; Bruce Momjian <bruce@momjian.us>; +Cc: Laurenz Albe <laurenz.albe@cybertec.at>; MichaelDBA <michaeldba@sqlexec.com>; holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Hi All
One basic doubt.
Whether work_mem is allocated for each clients connected to Database or for against the Sort Operations(Queries) in database? Please clarify
Case 1: work_mem=4mTotal Sessions connected=100Total=400M
Or Case 2work_mem=4mTotal Sort operations at DB now=50Total work_mem=200M
Please clarify at high level whether Case 1 is true or Case 2 is true?
Regards
Sasi On Friday, 2 April, 2021, 09:04:56 pm IST, Bruce Momjian <bruce@momjian.us> wrote:
On Fri, Apr 2, 2021 at 03:25:04PM +0000, Campbell, Lance wrote:
> Thanks for sharing this thread. My suggestion of having a work_mem_stack_size
> is the same concept mentioned in this thread regarding having a work_mem_pool.
> I prefer this later term rather than the one I was using. When the work mem
> pool is exhausted PostgreSQL just uses temp files for work_mem. With current
> statics for temp files and with a new stats on a work mem pool usage a user
> could fine tune memory much more precisely. It would leave the “art of memory
> tuning” behind. The other added benefit is that people would have a better
> understanding of how work_mem is used by naturally having to explain what a
> work_mem_pool is and when it is drawn on. There are probably a lot of
> PostgreSQL instance that would run faster just by having the confidence to
> increase the size of work_mem. I am sure many instances have this value set to
> low.
Uh, did you read the blog before it, referenced in that blog entry:
https://momjian.us/main/blogs/pgblog/2018.html#December_7_2018
Even if we have a pool, it is still complex to configure memory, but it
might help.
--
Bruce Momjian <bruce@momjian.us> https://momjian.us
EDB https://enterprisedb.com
If only the physical world exists, free will is an illusion.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-03 05:44 David G. Johnston <david.g.johnston@gmail.com>
parent: SASIKUMAR Devaraj <sashikumard@yahoo.com>
0 siblings, 1 reply; 22+ messages in thread
From: David G. Johnston @ 2021-04-03 05:44 UTC (permalink / raw)
To: SASIKUMAR Devaraj <sashikumard@yahoo.com>; +Cc: Campbell, Lance <lance@illinois.edu>; Bruce Momjian <bruce@momjian.us>; Laurenz Albe <laurenz.albe@cybertec.at>; MichaelDBA <michaeldba@sqlexec.com>; holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
On Friday, April 2, 2021, SASIKUMAR Devaraj <sashikumard@yahoo.com> wrote:
>
> Case 1:
> work_mem=4m
> Total Sessions connected=100
> Total=400M
>
> Or Case 2
> work_mem=4m
> Total Sort operations at DB now=50
> Total work_mem=200M
>
> Please clarify at high level whether Case 1 is true or Case 2 is true?
>
Case 2
David J.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: work_mem
@ 2021-04-03 06:15 SASIKUMAR Devaraj <sashikumard@yahoo.com>
parent: David G. Johnston <david.g.johnston@gmail.com>
0 siblings, 0 replies; 22+ messages in thread
From: SASIKUMAR Devaraj @ 2021-04-03 06:15 UTC (permalink / raw)
To: David G. Johnston <david.g.johnston@gmail.com>; +Cc: Campbell, Lance <lance@illinois.edu>; Bruce Momjian <bruce@momjian.us>; Laurenz Albe <laurenz.albe@cybertec.at>; MichaelDBA <michaeldba@sqlexec.com>; holger@jakobs.com <holger@jakobs.com>; pgsql-admin@lists.postgresql.org <pgsql-admin@lists.postgresql.org>
Excellent. Thank you very much for the clarification. Much Appreciated.
Regards
Sasi
On Saturday, 3 April, 2021, 11:14:36 am IST, David G. Johnston <david.g.johnston@gmail.com> wrote:
On Friday, April 2, 2021, SASIKUMAR Devaraj <sashikumard@yahoo.com> wrote:
Case 1:
work_mem=4mTotal Sessions connected=100Total=400M
Or Case 2work_mem=4mTotal Sort operations at DB now=50Total work_mem=200M
Please clarify at high level whether Case 1 is true or Case 2 is true?
Case 2
David J.
^ permalink raw reply [nested|flat] 22+ messages in thread
* Work_mem
@ 2024-04-08 06:50 Rajesh Kumar <rajeshkumar.dba09@gmail.com>
0 siblings, 2 replies; 22+ messages in thread
From: Rajesh Kumar @ 2024-04-08 06:50 UTC (permalink / raw)
To: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Hi team,
In a cluster, I have around 15dbs..out of which in 3 dbs creates of
temp_files which means I belive work_mem allocation is not sufficient for
those dbs.
Can I resolve this by increasing overall work_mem (set to 25MB now)?
Or by setting work_mem to users consuming those 3 dbs?
If I may want to increase how to decide on how much to increase?
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Work_mem
@ 2024-04-08 09:49 Laurenz Albe <laurenz.albe@cybertec.at>
parent: Rajesh Kumar <rajeshkumar.dba09@gmail.com>
1 sibling, 0 replies; 22+ messages in thread
From: Laurenz Albe @ 2024-04-08 09:49 UTC (permalink / raw)
To: Rajesh Kumar <rajeshkumar.dba09@gmail.com>; Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Mon, 2024-04-08 at 12:20 +0530, Rajesh Kumar wrote:
> In a cluster, I have around 15dbs..out of which in 3 dbs creates of temp_files
> which means I belive work_mem allocation is not sufficient for those dbs.
>
> Can I resolve this by increasing overall work_mem (set to 25MB now)?
>
> Or by setting work_mem to users consuming those 3 dbs?
Both would help.
> If I may want to increase how to decide on how much to increase?
You increase it ontil the performance is good, and frequently running
queries no longer need to create temporary files.
But you don't increase too much, otherwise you will run out of memory.
This is a question of trial and error, and it impossible to give you
exact numbers (even if we knew something about your system, which we don't).
Yours,
Laurenz Albe
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Work_mem
@ 2024-04-08 17:27 Jeff Janes <jeff.janes@gmail.com>
parent: Rajesh Kumar <rajeshkumar.dba09@gmail.com>
1 sibling, 1 reply; 22+ messages in thread
From: Jeff Janes @ 2024-04-08 17:27 UTC (permalink / raw)
To: Rajesh Kumar <rajeshkumar.dba09@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Mon, Apr 8, 2024 at 2:50 AM Rajesh Kumar <rajeshkumar.dba09@gmail.com>
wrote:
> Hi team,
>
>
> In a cluster, I have around 15dbs..out of which in 3 dbs creates of
> temp_files which means I belive work_mem allocation is not sufficient for
> those dbs.
>
That is a questionable conclusion. Using temp files is not a disaster. It
is not using them which is more likely to be a disaster, when too large
a setting of work_mem leads to swapping/paging. When the system switches to
using temp files, it also switches to using algorithms which are well
suited to them.
Indeed on modern systems and modern versions of PostgreSQL, switching to
temp files can often be faster than using very large work_mem even when the
large work_mem doesn't lead to swapping/paging. I think this is
because main RAM is slow compared to the various forms of CPU cache, and
the temp-file algorithms are often far friendlier to CPU cache than the
random-access algorithms are. And on modern systems, the temp files likely
never even reach disk; they are just transferred to and from main RAM, and
in cache friendly ways if your cache supports some kind of read-ahead.
Although these improvements are not easy to predict and so not easy to tune
for.
You should figure out which queries cause those temp files, and if those
queries are timing sensitive then test those queries with various settings
of work_mem. Keep in mind that you don't want to set it so high that you
exhaust memory, so you must keep in mind how many instances of work_mem
might be in use, system wide, at the same time.
Can I resolve this by increasing overall work_mem (set to 25MB now)?
>
That is like asking "how long is a piece of string?". Identify the queries
that cause the "problem".
Cheers,
Jeff
>
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Work_mem
@ 2024-04-08 17:56 Rajesh Kumar <rajeshkumar.dba09@gmail.com>
parent: Jeff Janes <jeff.janes@gmail.com>
0 siblings, 1 reply; 22+ messages in thread
From: Rajesh Kumar @ 2024-04-08 17:56 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
I dont see any long running queries under pg_stat_activity or
pg_stat_statements.
On Mon, 8 Apr 2024 at 22:57, Jeff Janes <jeff.janes@gmail.com> wrote:
> On Mon, Apr 8, 2024 at 2:50 AM Rajesh Kumar <rajeshkumar.dba09@gmail.com>
> wrote:
>
>> Hi team,
>>
>>
>> In a cluster, I have around 15dbs..out of which in 3 dbs creates of
>> temp_files which means I belive work_mem allocation is not sufficient for
>> those dbs.
>>
>
> That is a questionable conclusion. Using temp files is not a disaster.
> It is not using them which is more likely to be a disaster, when too large
> a setting of work_mem leads to swapping/paging. When the system switches to
> using temp files, it also switches to using algorithms which are well
> suited to them.
>
> Indeed on modern systems and modern versions of PostgreSQL, switching to
> temp files can often be faster than using very large work_mem even when the
> large work_mem doesn't lead to swapping/paging. I think this is
> because main RAM is slow compared to the various forms of CPU cache, and
> the temp-file algorithms are often far friendlier to CPU cache than the
> random-access algorithms are. And on modern systems, the temp files likely
> never even reach disk; they are just transferred to and from main RAM, and
> in cache friendly ways if your cache supports some kind of read-ahead.
> Although these improvements are not easy to predict and so not easy to tune
> for.
>
> You should figure out which queries cause those temp files, and if those
> queries are timing sensitive then test those queries with various settings
> of work_mem. Keep in mind that you don't want to set it so high that you
> exhaust memory, so you must keep in mind how many instances of work_mem
> might be in use, system wide, at the same time.
>
> Can I resolve this by increasing overall work_mem (set to 25MB now)?
>>
>
> That is like asking "how long is a piece of string?". Identify the
> queries that cause the "problem".
>
> Cheers,
>
> Jeff
>
>>
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Work_mem
@ 2024-04-10 22:21 Jeff Janes <jeff.janes@gmail.com>
parent: Rajesh Kumar <rajeshkumar.dba09@gmail.com>
0 siblings, 1 reply; 22+ messages in thread
From: Jeff Janes @ 2024-04-10 22:21 UTC (permalink / raw)
To: Rajesh Kumar <rajeshkumar.dba09@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
On Mon, Apr 8, 2024 at 1:57 PM Rajesh Kumar <rajeshkumar.dba09@gmail.com>
wrote:
> I dont see any long running queries under pg_stat_activity or
> pg_stat_statements.
>
>>
>> With pg_stat_activity, you would need to catch them "in the act", but
there are no columns there which describe temp file usage anyway.
pg_stat_statements has the columns "temp_blks_read" and
"temp_blks_written", in all supported versions, so you should be able to
spot the queries using temp files there. Unless maybe your
pg_stat_statements.max setting is too low and those queries are forced out.
Or you could set log_temp_files = 0. That would log all statements using
temp files into the PostgreSQL server log.
Cheers,
Jeff
^ permalink raw reply [nested|flat] 22+ messages in thread
* Re: Work_mem
@ 2024-04-11 05:07 Rajesh Kumar <rajeshkumar.dba09@gmail.com>
parent: Jeff Janes <jeff.janes@gmail.com>
0 siblings, 0 replies; 22+ messages in thread
From: Rajesh Kumar @ 2024-04-11 05:07 UTC (permalink / raw)
To: Jeff Janes <jeff.janes@gmail.com>; +Cc: Pgsql-admin <pgsql-admin@lists.postgresql.org>
Thanks, I'll look into it.
On Thu, 11 Apr 2024, 03:51 Jeff Janes, <jeff.janes@gmail.com> wrote:
>
>
> On Mon, Apr 8, 2024 at 1:57 PM Rajesh Kumar <rajeshkumar.dba09@gmail.com>
> wrote:
>
>> I dont see any long running queries under pg_stat_activity or
>> pg_stat_statements.
>>
>>>
>>> With pg_stat_activity, you would need to catch them "in the act", but
> there are no columns there which describe temp file usage anyway.
>
> pg_stat_statements has the columns "temp_blks_read" and
> "temp_blks_written", in all supported versions, so you should be able to
> spot the queries using temp files there. Unless maybe your
> pg_stat_statements.max setting is too low and those queries are forced out.
>
> Or you could set log_temp_files = 0. That would log all statements using
> temp files into the PostgreSQL server log.
>
> Cheers,
>
> Jeff
>
^ permalink raw reply [nested|flat] 22+ messages in thread
end of thread, other threads:[~2024-04-11 05:07 UTC | newest]
Thread overview: 22+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2021-03-31 14:04 work_mem Campbell, Lance <lance@illinois.edu>
2021-03-31 14:17 ` Bruce Momjian <bruce@momjian.us>
2021-03-31 15:22 ` Holger Jakobs <holger@jakobs.com>
2021-04-01 07:43 ` Sachin Kumar <sachinkumaras@gmail.com>
2021-04-02 12:37 ` SASIKUMAR Devaraj <sashikumard@yahoo.com>
2021-04-02 12:39 ` MichaelDBA <MichaelDBA@sqlexec.com>
2021-04-02 12:45 ` SASIKUMAR Devaraj <sashikumard@yahoo.com>
2021-04-02 12:49 ` MichaelDBA <MichaelDBA@sqlexec.com>
2021-04-02 13:31 ` Campbell, Lance <lance@illinois.edu>
2021-04-02 14:59 ` Laurenz Albe <laurenz.albe@cybertec.at>
2021-04-02 15:07 ` Bruce Momjian <bruce@momjian.us>
2021-04-02 15:25 ` Campbell, Lance <lance@illinois.edu>
2021-04-02 15:34 ` Bruce Momjian <bruce@momjian.us>
2021-04-03 05:23 ` SASIKUMAR Devaraj <sashikumard@yahoo.com>
2021-04-03 05:44 ` David G. Johnston <david.g.johnston@gmail.com>
2021-04-03 06:15 ` SASIKUMAR Devaraj <sashikumard@yahoo.com>
2024-04-08 06:50 Work_mem Rajesh Kumar <rajeshkumar.dba09@gmail.com>
2024-04-08 09:49 ` Re: Work_mem Laurenz Albe <laurenz.albe@cybertec.at>
2024-04-08 17:27 ` Re: Work_mem Jeff Janes <jeff.janes@gmail.com>
2024-04-08 17:56 ` Re: Work_mem Rajesh Kumar <rajeshkumar.dba09@gmail.com>
2024-04-10 22:21 ` Re: Work_mem Jeff Janes <jeff.janes@gmail.com>
2024-04-11 05:07 ` Re: Work_mem Rajesh Kumar <rajeshkumar.dba09@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox