Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vnJqG-0054Ih-0k for pgsql-hackers@arkaria.postgresql.org; Tue, 03 Feb 2026 17:05:32 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.96) (envelope-from ) id 1vnJqF-006Rf9-0u for pgsql-hackers@arkaria.postgresql.org; Tue, 03 Feb 2026 17:05:31 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.96) (envelope-from ) id 1vnJqE-006Rez-2w for pgsql-hackers@lists.postgresql.org; Tue, 03 Feb 2026 17:05:30 +0000 Received: from mail-dl1-x1244.google.com ([2607:f8b0:4864:20::1244]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.98.2) (envelope-from ) id 1vnJqC-00000000rzT-0zMB for pgsql-hackers@postgresql.org; Tue, 03 Feb 2026 17:05:30 +0000 Received: by mail-dl1-x1244.google.com with SMTP id a92af1059eb24-1233bc1117fso31513c88.0 for ; Tue, 03 Feb 2026 09:05:28 -0800 (PST) ARC-Seal: i=1; a=rsa-sha256; t=1770138326; cv=none; d=google.com; s=arc-20240605; b=f8r6C3nraHsfPt9O4urHs9hUDFJpLDuNR7RIF/zr+XHcBCkVB4g7KRzrbef9u3jmj6 06efnJHSuJmceli6T4+ckSk4X0lZZngE3M9Nq0bNeGHeomlmBrmXVsLRFUA+eun6dm1Q eqTiCfYvIX0/yMiX+SzR8L5hz2uHUUXX/4eFM+xEj+YfdO79dSDqTXyyIoHMADuyE7lN AZKcBxHshH6RKapYQtIu2hozOnBlQpfzAzSXtGMWolduBYdEaW1wk2z+k8lAyvdIMung SpRdW14hv+iI0OQwqJNBzdvaWp1D23S5CIHX/Og8VMFczy+CRvhH2EIs0h3M+c1wIIdm r7OA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20240605; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:dkim-signature; bh=Dv13JlBZQ1InunQ8SVgo1ZCQxcE15mTMJ48Wp9dWIm0=; fh=d7N9tkE5MIR/ZWdt6PaCMcrc7bcDddEA0b7qVNbWvAU=; b=OqLni62iNoZZ7Mx9TRb1t5Z4m4W7cO1foHmC6BjJFecZwu3Xu11xEE8gXsAB2l7qxL XvmaLkQig3Z/WTO4r4ff2yq8nvu2LfBUKhlIMl4ejBhFy3CYhWFTTR9YDQ8SwI4e7biX vdGIdYtiLM9I96SLwsk9KYdeuIzrrcD2QJckrPGFGFJBasne02ZETCpq9dUCBd+133hr QODmCTFi8a2FW6AiX6tdLrhdOOQjDUfkqMfZ8gDo/bKhB5BLJ/laD/xtBD/X0yACg4tl +xVCb342n779JvoczMXThcEzf54y4XBGvtoJuKhlmaEKXk1ubQqoTozD0icQnfVQlby+ m9Ow==; darn=postgresql.org ARC-Authentication-Results: i=1; mx.google.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1770138326; x=1770743126; darn=postgresql.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=Dv13JlBZQ1InunQ8SVgo1ZCQxcE15mTMJ48Wp9dWIm0=; b=Xej23v3+yNiP2gozBmR+rETRtUqTOtgg9dvZAv+FnIEzsXDwcKZWqfjIFj/DFEyBcf HKF6tQAHDDcl4scjMh0Uxwrs9cfCA+9ldDQS4p/vTZwqhDGYao1Cs5vjpvjg8Ug8KOZ2 n1AX+uotscmfCwpVqXLtDIfQqNqXiZF/+SgEU4Mp0QrqH0fg1MA6TNOBPh6EgYtujcWm ytG0EvDkpw6YgtnOLMNLXG08OeqvF24YA1v+fQQWiC8NmQlE2yzjpZMinLW5EKY5YSEh +9AMq7jvuJ02hLgW3Mb6tbNrgO45XKzWOjeoVvlltXx4t06bqPwpy0z2Br7yGFW92W4T WXfw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1770138326; x=1770743126; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-gg:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=Dv13JlBZQ1InunQ8SVgo1ZCQxcE15mTMJ48Wp9dWIm0=; b=WIA9OWOubooxuWRX2S3LQWdmHBS0pZBTFEAMx43RviQgTVq43Dpe9HxMpQupOZO1mP 9O8MvNyiJ6Kpp4+tqksP4cg4o5T3t3WpWYimv5JcmAd24JsvZ/aVbfpowm5tYbaXDHyp 5i6vviBbWiQU47fTRI2SUpmhdg6PGzsFe3ZQhHcqAyGfRQCdiYzOoghh0qe4oEmX5Sm8 Xc0OEyeNn6rXj9YnnDzgUk/OtuNU2VswvGhlaadYQTXBnRxk8tot3QQTerFoaDqoYpqB 15SEGSF9dhq59LCK5+0AtlUoaVZR0ec8zDEeWSpHUguxpQEZi7+OyI5DRuUw2dY+EOHX 6oTA== X-Gm-Message-State: AOJu0Yzki/FLdKfFaHrlacLUp8qbwEPJ7mDQHUxHj5LmklL5dx3Z5M2X e+kc5g0H7cOCawuAr0e460KwEDTFXNP3WpXOAaD1RbEKiSIyy2fEFH4GNCIIom4AMbxApM90oLO ecg7gPGJ5g8pho8KmJbR+J4PkaZgGe7Q= X-Gm-Gg: AZuq6aJDKjLB+M4Rs1YpG+jhDX61Gd7r8fgQlqMy8l/ha6LZg26ULahOOOZTlZFUn9A NBi0xOYYFYAtBL/zaXS0ls65xmRgZnKhmVoDZmImYgKPDG/6TVDvmxPvyRNSV21diqt4KKy69Il JR9pCeUE6FOO4liGutYNZX+LScgREUYSWdvPc1ftwL5EikYlznjBLJVXvgJ/H05ThqoKjzfYrXY tbn4iD4yz9VfbDWwP6FcBPS4wY5wBQrLjmGWZIjym8xFuy1AhszcnfFw8g+7a7PizYW9A== X-Received: by 2002:a05:7300:641d:b0:2b8:209d:598d with SMTP id 5a478bee46e88-2b832e676a3mr38966eec.11.1770138326081; Tue, 03 Feb 2026 09:05:26 -0800 (PST) MIME-Version: 1.0 References: <7bcc0420-d457-4af5-a459-a4e5d929a665@vondra.me> <7gbzkoxrpqvj2sgxof4pqirz7pf6wb5vizak3mxrz2zpvrf3pa@lvnwdsy3em56> In-Reply-To: From: lakshmi Date: Tue, 3 Feb 2026 22:37:58 +0530 X-Gm-Features: AZwV_QgDykXp7s_i3TsZ0ZR2frUD-1UDevnztSjK-HBIljtLC739jIilzFIkGqk Message-ID: Subject: Re: Proposal: Adding compression of temporary files To: Filip Janus Cc: pgsql-hackers , zsolt.parragi@percona.com Content-Type: multipart/alternative; boundary="0000000000002123080649ee7270" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --0000000000002123080649ee7270 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi all, I=E2=80=99ve applied the latest two patches on current 19devel and wanted t= o share some testing results. The patches apply cleanly, build and install without issues, and the server starts normally. I verified that the new temp_file_compression GUC works as expected and accepts the documented values (no, lz4, pglz), while invalid values are correctly rejected. For testing, I forced temp file usage by running parallel hash joins with a small work_mem. I ran the same query with temp_file_compression set to no, lz4, and pglz. In all cases, temp files were created and used (hash join spilling to disk), query results were identical, and I did not see any crashes or read/write errors. The temp read/write counters were very similar across all three modes. This seems expected for hash join spills, since they use fixed-size fileset chunks, so compression doesn=E2=80=99t necessarily reduce the number of tem= p blocks written. Execution time was comparable across modes, with no regressions observed. I also ran make check, which passed. Based on this testing, everything looks good to me, and the observed behavior matches the documentation clarification about workload-dependent effects of temp file compression. Thanks for working on this I=E2=80=99m happy to test further or try additi= onal workloads if that would be useful. Best regards, Lakshmi On Sun, Jan 25, 2026 at 5:27=E2=80=AFPM Filip Janus wro= te: > Fixed spacing in the patch. > > -Filip- > > > p=C3=A1 23. 1. 2026 v 17:40 odes=C3=ADlatel Filip Janus napsal: > >> Hi all, >> Thanks for the feedback and the provided patch. >> I've addressed your findings and proposals. Lakshmi's documentation patc= h >> was incorporated. >> >> -Filip- >> >> >> st 21. 1. 2026 v 7:30 odes=C3=ADlatel lakshmi n= apsal: >> >>> HI all, >>> While testing the temp file compression patch,noticed that the new >>> temp_file_compression GUC isn't documented yet.I put together a small d= ocs >>> patch to add a short description and clarify that the effect of compres= sion >>> depends on the workload(for example ,hash join spills may not show visi= ble >>> size reduction due to fixed_size chunks). >>> >>> patch is attached.Happy to adjust the wording if needed. >>> thanks, >>> lakshmi >>> >>> On Tue, Jan 20, 2026 at 4:21=E2=80=AFPM lakshmi wrote: >>> >>>> Hi Filip, >>>> >>>> I tested both patches on current master using git am -3 .They apply >>>> cleanly,build fine,and the temp_file _compression GUC works as expecte= d. >>>> Query results are unchanged. >>>> >>>> For hash join spill test,temp files were created as expected,but the >>>> logged size were same for no,lz4,and pglz,which seems consistent with >>>> fixed-size fileset chunking.It might be helpful to briefly note this i= n the >>>> documentation to avoid confusion. >>>> >>>> Thanks for working on this . >>>> best regards, >>>> lakshmi >>>> >>>> On Tue, Jan 20, 2026 at 4:10=E2=80=AFAM Zsolt Parragi < >>>> zsolt.parragi@percona.com> wrote: >>>> >>>>> Hello! >>>>> >>>>> I tried to review the code. It compiled, the test suite passed. >>>>> >>>>> I noticed two typos: >>>>> >>>>> buffile.c:77 - "Disaled" >>>>> buffile.c:133 - "mathods" >>>>> >>>>> And a few other small findings: >>>>> >>>>> buffile.h:35 and buffile.c:63 - same constants defined first as an >>>>> Enum and then as #defines - code builds properly without the defines. >>>>> >>>>> buffile.c:121 - compress_tempfile is defined, set to false at :167, >>>>> but never used otherwise >>>>> >>>>> guc_tables.c:470 - the comment says that pglz isn't supported yet, bu= t >>>>> we have a value for it, and I see support for it in the code >>>>> >>>>> buffile.c:659: (and at other places) if USE_LZ4 is undefined, the >>>>> codepath doesn't do anything. I think these ifdefs should follow how >>>>> other compression code works, such as wal compression where there's a= n >>>>> #else path with elog(ERROR, ...) >>>>> Similarly, maybe there should be an explicit TEMP_NONE_COMPRESSION >>>>> branch that does nothing, and the default branch should be an error? >>>>> >>>>> buffile.c:265: If seek isn't supported/limited, shouldn't there be at >>>>> least an assertion about it in BufFileSeek? And tell isn't mentioned, >>>>> but it seems to me that tell also doesn't work properly. >>>>> >>>> --0000000000002123080649ee7270 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable

Hi all,

I=E2=80=99ve applied the latest two patches on current 1= 9devel and wanted to share some testing results.

The patches apply c= leanly, build and install without issues, and the server starts normally. I= verified that the new temp_file_compression GUC works as expe= cted and accepts the documented values (no, lz4, = pglz), while invalid values are correctly rejected.

For= testing, I forced temp file usage by running parallel hash joins with a sm= all work_mem. I ran the same query with temp_file_compre= ssion set to no, lz4, and pglz. In all cases, temp files were created and used (hash join spilling to di= sk), query results were identical, and I did not see any crashes or read/wr= ite errors.

The temp read/write counters were very similar across al= l three modes. This seems expected for hash join spills, since they use fix= ed-size fileset chunks, so compression doesn=E2=80=99t necessarily reduce t= he number of temp blocks written. Execution time was comparable across mode= s, with no regressions observed.

I also ran make check, = which passed.

Based on this testing, everything looks good to me, an= d the observed behavior matches the documentation clarification about workl= oad-dependent effects of temp file compression.

Thanks for working o= n this=C2=A0 I=E2=80=99m happy to test further or try additional workloads = if that would be useful.

Best regards,
Lakshmi

=

On Sun, Jan 25, 2026 at 5:27=E2=80=AFPM Filip Janus &= lt;fjanus@redhat.com> wrote:
<= div>Fixed spacing in the patch.

=C2=A0 =C2=A0 -Filip-


p=C3=A1 23. 1. 2026= v=C2=A017:40 odes=C3=ADlatel Filip Janus <fjanus@redhat.com> napsal:
Hi all,= =C2=A0
Thanks for the feedback and the provided patch.
= I've addressed=C2=A0your findings and proposals.=C2=A0Lakshmi's doc= umentation patch was incorporated.=C2=A0 =C2=A0

=
=C2=A0 =C2=A0 -Filip-


st = 21. 1. 2026 v=C2=A07:30 odes=C3=ADlatel lakshmi <lakshmigcdac@gmail.com> napsal:=
HI all,
While testing the temp file compression patch,noticed that the= new temp_file_compression GUC isn't documented yet.I put together a sm= all docs patch to add a short description and clarify that the effect=C2=A0= of compression depends on the workload(for example ,hash join spills may no= t show visible size reduction due to fixed_size chunks).

patch is at= tached.Happy to adjust the wording if needed.
thanks,
lakshmi
On Tue, = Jan 20, 2026 at 4:21=E2=80=AFPM lakshmi <lakshmigcdac@gmail.com> wrote:
Hi Fili= p,

I tested both patches on current master using git am -3 .They app= ly cleanly,build fine,and the temp_file _compression GUC works as expected.=
Query results are unchanged.

For hash join spill test,temp files= were created as expected,but the logged size were same for no,lz4,and pglz= ,which seems consistent with fixed-size fileset chunking.It might be helpfu= l to briefly note this in the documentation to avoid confusion.

Than= ks for working on this .
best regards,
lakshmi

On Tue, Jan 20, 2026 at= 4:10=E2=80=AFAM Zsolt Parragi <zsolt.parragi@percona.com> wrote:
Hello!

I tried to review the code. It compiled, the test suite passed.

I noticed two typos:

buffile.c:77 - "Disaled"
buffile.c:133 - "mathods"

And a few other small findings:

buffile.h:35 and buffile.c:63 - same constants defined first as an
Enum and then as #defines - code builds properly without the defines.

buffile.c:121 - compress_tempfile is defined, set to false at :167,
but never used otherwise

guc_tables.c:470 - the comment says that pglz isn't supported yet, but<= br> we have a value for it, and I see support for it in the code

buffile.c:659: (and at other places) if USE_LZ4 is undefined, the
codepath doesn't do anything. I think these ifdefs should follow how other compression code works, such as wal compression where there's an<= br> #else path with elog(ERROR, ...)
Similarly, maybe there should be an explicit TEMP_NONE_COMPRESSION
branch that does nothing, and the default branch should be an error?

buffile.c:265: If seek isn't supported/limited, shouldn't there be = at
least an assertion about it in BufFileSeek? And tell isn't mentioned, but it seems to me that tell also doesn't work properly.
--0000000000002123080649ee7270--