Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sVtGh-00CBP6-Rq for pgsql-admin@arkaria.postgresql.org; Mon, 22 Jul 2024 13:40:00 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sVtGf-002AUA-Sz for pgsql-admin@arkaria.postgresql.org; Mon, 22 Jul 2024 13:39:58 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sVtGf-002ASi-63 for pgsql-admin@lists.postgresql.org; Mon, 22 Jul 2024 13:39:57 +0000 Received: from mail2.pscs.co.uk ([178.159.10.131] helo=mail.pscs.co.uk) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sVtGb-000rpd-V4 for pgsql-admin@lists.postgresql.org; Mon, 22 Jul 2024 13:39:56 +0000 Authentication-Results: mail.pscs.co.uk; spf=none; auth=pass (cram-md5) smtp.auth=pscs Received: from lmail.pscs.co.uk ([192.168.120.1]) by mail.pscs.co.uk ([192.168.120.185] running VPOP3) with ESMTPSA (TLSv1.3 TLS_AES_256_GCM_SHA384) for ; Mon, 22 Jul 2024 14:39:49 +0100 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=pscs.co.uk; q=dns/txt; s=lmail; h=Content-Type:Message-ID:Date:MIME-Version:Subject:To:References:From :In-Reply-To:Cc:Content-Transfer-Encoding:Reply-to:Sender; t=1721655249; x=1722260049; bh=HVy6Mk1o8JEWuacUevjimWjXAbvw06aY5b8+S0k6fb4=; b=Hu2xJSN3ZaexZpI1RWWVBSlZPvEGWs/Y9iJA5yuUaEjoPE1ZWBseIRwTDVSFbsH8NRlkmk+Q tSt9Y7vs+UzMZ4zB3Ix7wHIE8/G0e0HbZ9+Swq6iGIDNwy3ztiLoc8ive2TuaesFa1mQ3KsDdp 0OLoAXQa0j2Tu9mfE9A+ytZQM= Authentication-Results: lmail.pscs.co.uk; spf=none; auth=pass (cram-md5) smtp.auth=paul Received: from [192.168.57.71] ([217.155.111.120] (217-155-111-120.dsl.in-addr.zen.co.uk)) by lmail.pscs.co.uk ([192.168.120.70] running VPOP3) with ESMTPSA (TLSv1.3 TLS_AES_256_GCM_SHA384) for ; Mon, 22 Jul 2024 14:34:08 +0100 Content-Type: multipart/alternative; boundary="------------kP3J1atsfzah6HMFGBg01l4s" Message-ID: Date: Mon, 22 Jul 2024 14:34:09 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: small temp files To: pgsql-admin@lists.postgresql.org References: <7A0C9A69-5632-4CFC-B156-53FAAAB33E26@elevated-dev.com> <8E03B792-C389-456D-913D-4F2B0EBB903E@elevated-dev.com> Content-Language: en-GB From: Paul Smith* In-Reply-To: <8E03B792-C389-456D-913D-4F2B0EBB903E@elevated-dev.com> X-Authenticated-Sender: paul X-Server: VPOP3 Enterprise V8.5 - Registered X-Organisation: Paul Smith Computer Services X-VPOP3Tester: 12 345 X-Authenticated-Sender: pscs List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multi-part message in MIME format. --------------kP3J1atsfzah6HMFGBg01l4s Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit On 22/07/2024 14:28, Scott Ribe wrote: > I understand those things--my question is why, with work_mem set to 128MB, I would see tiny temp files (7452 is common, as is 102, and I've seen as small as 51). > From the manual: "Note that a complex query might perform several sort and hash operations at the same time, with each operation generally being allowed to use as much memory as this value specifies before it starts to write data into temporary files. Also, several running sessions could be doing such operations concurrently. Therefore, the total memory used could be many times the value of|work_mem|; it is necessary to keep this fact in mind when choosing the value. Sort operations are used for|ORDER BY|,|DISTINCT|, and merge joins. Hash tables are used in hash joins, hash-based aggregation, memoize nodes and hash-based processing of|IN|subqueries." So, if it's doing lots of joins, there may be lots of bits of temporary data which together add up to more than work_mem. AIUI PostgreSQL doesn't necessarily shove all the temporary data for one query into one file, it may have multiple smaller files for the data for a single query. --------------kP3J1atsfzah6HMFGBg01l4s Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit
On 22/07/2024 14:28, Scott Ribe wrote:
I understand those things--my question is why, with work_mem set to 128MB, I would see tiny temp files (7452 is common, as is 102, and I've seen as small as 51).
From the manual:

"Note that a complex query might perform several sort and hash operations at the same time, with each operation generally being allowed to use as much memory as this value specifies before it starts to write data into temporary files. Also, several running sessions could be doing such operations concurrently. Therefore, the total memory used could be many times the value of work_mem; it is necessary to keep this fact in mind when choosing the value. Sort operations are used for ORDER BY, DISTINCT, and merge joins. Hash tables are used in hash joins, hash-based aggregation, memoize nodes and hash-based processing of IN subqueries."

So, if it's doing lots of joins, there may be lots of bits of temporary data which together add up to more than work_mem. AIUI PostgreSQL doesn't necessarily shove all the temporary data for one query into one file, it may have multiple smaller files for the data for a single query.


--------------kP3J1atsfzah6HMFGBg01l4s--