pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedpglz performance
70+ messages / 13 participants
[nested] [flat]
Commits from this thread:
2019-11-28 c60e520f6e Use memcpy instead of a byte loop in pglz_decompress
* pglz performance
@ 2019-05-13 02:45 Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 70+ messages in thread
From: Andrey Borodin @ 2019-05-13 02:45 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi hackers!
I was reviewing Paul Ramsey's TOAST patch[0] and noticed that there is a big room for improvement in performance of pglz compression and decompression.
With Vladimir we started to investigate ways to boost byte copying and eventually created test suit[1] to investigate performance of compression and decompression.
This is and extension with single function test_pglz() which performs tests for different:
1. Data payloads
2. Compression implementations
3. Decompression implementations
Currently we test mostly decompression improvements against two WALs and one data file taken from pgbench-generated database. Any suggestion on more relevant data payloads are very welcome.
My laptop tests show that our decompression implementation [2] can be from 15% to 50% faster.
Also I've noted that compression is extremely slow, ~30 times slower than decompression. I believe we can do something about it.
We focus only on boosting existing codec without any considerations of other compression algorithms.
Any comments are much appreciated.
Most important questions are:
1. What are relevant data sets?
2. What are relevant CPUs? I have only XEON-based servers and few laptops\desktops with intel CPUs
3. If compression is 30 times slower, should we better focus on compression instead of decompression?
Best regards, Andrey Borodin.
[0] https://www.postgresql.org/message-id/flat/CANP8%2BjKcGj-JYzEawS%2BCUZnfeGKq4T5LswcswMP4GUHeZEP1ag%4...
[1] https://github.com/x4m/test_pglz
[2] https://www.postgresql.org/message-id/C2D8E5D5-3E83-469B-8751-1C7877C2A5F2%40yandex-team.ru
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-05-13 07:14 Michael Paquier <michael@paquier.xyz>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 2 replies; 70+ messages in thread
From: Michael Paquier @ 2019-05-13 07:14 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Mon, May 13, 2019 at 07:45:59AM +0500, Andrey Borodin wrote:
> I was reviewing Paul Ramsey's TOAST patch[0] and noticed that there
> is a big room for improvement in performance of pglz compression and
> decompression.
Yes, I believe so too. pglz is a huge CPU-consumer when it comes to
compilation compared to more modern algos like lz4.
> With Vladimir we started to investigate ways to boost byte copying
> and eventually created test suit[1] to investigate performance of
> compression and decompression. This is and extension with single
> function test_pglz() which performs tests for different:
> 1. Data payloads
> 2. Compression implementations
> 3. Decompression implementations
Cool. I got something rather similar in my wallet of plugins:
https://github.com/michaelpq/pg_plugins/tree/master/compress_test
This is something I worked on mainly for FPW compression in WAL.
> Currently we test mostly decompression improvements against two WALs
> and one data file taken from pgbench-generated database. Any
> suggestion on more relevant data payloads are very welcome.
Text strings made of random data and variable length? For any test of
this kind I think that it is good to focus on the performance of the
low-level calls, even going as far as a simple C wrapper on top of the
pglz APIs to test only the performance and not have extra PG-related
overhead like palloc() which can be a barrier. Focusing on strings of
lengths of 1kB up to 16kB may be an idea of size, and it is important
to keep the same uncompressed strings for performance comparison.
> My laptop tests show that our decompression implementation [2] can
> be from 15% to 50% faster. Also I've noted that compression is
> extremely slow, ~30 times slower than decompression. I believe we
> can do something about it.
That's nice.
> We focus only on boosting existing codec without any considerations
> of other compression algorithms.
There is this as well. A couple of algorithms have a license
compatible with Postgres, but it may be more simple to just improve
pglz. A 10%~20% improvement is something worth doing.
> Most important questions are:
> 1. What are relevant data sets?
> 2. What are relevant CPUs? I have only XEON-based servers and few
> laptops\desktops with intel CPUs
> 3. If compression is 30 times slower, should we better focus on
> compression instead of decompression?
Decompression can matter a lot for mostly-read workloads and
compression can become a bottleneck for heavy-insert loads, so
improving compression or decompression should be two separate
problems, not two problems linked. Any improvement in one or the
other, or even both, is nice to have.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-05-15 10:06 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-05-15 10:06 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
> 13 мая 2019 г., в 12:14, Michael Paquier <michael@paquier.xyz> написал(а):
>
>> Currently we test mostly decompression improvements against two WALs
>> and one data file taken from pgbench-generated database. Any
>> suggestion on more relevant data payloads are very welcome.
>
> Text strings made of random data and variable length?
Like text corpus?
> For any test of
> this kind I think that it is good to focus on the performance of the
> low-level calls, even going as far as a simple C wrapper on top of the
> pglz APIs to test only the performance and not have extra PG-related
> overhead like palloc() which can be a barrier.
Our test_pglz extension is measuring only time of real compression, doing warmup run, all allocations are done before measurement.
> Focusing on strings of
> lengths of 1kB up to 16kB may be an idea of size, and it is important
> to keep the same uncompressed strings for performance comparison.
We intentionally avoid using generated data, thus keep test files committed into git repo.
Also we check that decompressed data matches source of compression. All tests are done 5 times.
We use PG extension only for simplicity of deployment of benchmarks to our PG clusters.
Here are some test results.
Currently we test on 4 payloads:
1. WAL from cluster initialization
2. 2 WALs from pgbench pgbench -i -s 10
3. data file taken from pgbench -i -s 10
We use these decompressors:
1. pglz_decompress_vanilla - taken from PG source code
2. pglz_decompress_hacked - use sliced memcpy to imitate byte-by-byte pglz decompression
3. pglz_decompress_hacked4, pglz_decompress_hacked8, pglz_decompress_hackedX - use memcpy if match is no less than X bytes. We need to determine best X, if this approach is used.
I used three platforms:
1. Server XEONE5-2660 SM/SYS1027RN3RF/10S2.5/1U/2P (2*INTEL XEON E5-2660/16*DDR3ECCREG/10*SAS-2.5) Under Ubuntu 14, PG 9.6.
2. Desktop Intel(R) Core(TM) i7-6700 CPU @ 3.40GHz Ubuntu 18, PG 12devel
3. Laptop MB Pro 15 2015 2.2 GHz Core i7 (I7-4770HQ) MacOS, PG 12devel
Owners of AMD and ARM devices are welcome.
Server results (less is better):
NOTICE: 00000: Time to decompress one byte in ns:
NOTICE: 00000: Payload 000000010000000000000001
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.647235
NOTICE: 00000: Decompressor pglz_decompress_hacked4 result 0.671029
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.699949
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.739586
NOTICE: 00000: Decompressor pglz_decompress_hacked32 result 0.787926
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 1.147282
NOTICE: 00000: Payload 000000010000000000000006
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.201774
NOTICE: 00000: Decompressor pglz_decompress_hacked4 result 0.211859
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.212610
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.214601
NOTICE: 00000: Decompressor pglz_decompress_hacked32 result 0.221813
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.706005
NOTICE: 00000: Payload 000000010000000000000008
NOTICE: 00000: Decompressor pglz_decompress_hacked result 1.370132
NOTICE: 00000: Decompressor pglz_decompress_hacked4 result 1.388991
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 1.388502
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 1.529455
NOTICE: 00000: Decompressor pglz_decompress_hacked32 result 1.520813
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 1.433527
NOTICE: 00000: Payload 16398
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.606943
NOTICE: 00000: Decompressor pglz_decompress_hacked4 result 0.623044
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.624118
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.620987
NOTICE: 00000: Decompressor pglz_decompress_hacked32 result 0.621183
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 1.365318
Comment: pglz_decompress_hacked is unconditionally optimal. On most of cases it is 2x better than current implementation.
On 000000010000000000000008 it is only marginally better. pglz_decompress_hacked8 is few percents worse than pglz_decompress_hacked.
Desktop results:
NOTICE: Time to decompress one byte in ns:
NOTICE: Payload 000000010000000000000001
NOTICE: Decompressor pglz_decompress_hacked result 0.396454
NOTICE: Decompressor pglz_decompress_hacked4 result 0.429249
NOTICE: Decompressor pglz_decompress_hacked8 result 0.436413
NOTICE: Decompressor pglz_decompress_hacked16 result 0.478077
NOTICE: Decompressor pglz_decompress_hacked32 result 0.491488
NOTICE: Decompressor pglz_decompress_vanilla result 0.695527
NOTICE: Payload 000000010000000000000006
NOTICE: Decompressor pglz_decompress_hacked result 0.110710
NOTICE: Decompressor pglz_decompress_hacked4 result 0.115669
NOTICE: Decompressor pglz_decompress_hacked8 result 0.127637
NOTICE: Decompressor pglz_decompress_hacked16 result 0.120544
NOTICE: Decompressor pglz_decompress_hacked32 result 0.117981
NOTICE: Decompressor pglz_decompress_vanilla result 0.399446
NOTICE: Payload 000000010000000000000008
NOTICE: Decompressor pglz_decompress_hacked result 0.647402
NOTICE: Decompressor pglz_decompress_hacked4 result 0.691891
NOTICE: Decompressor pglz_decompress_hacked8 result 0.693834
NOTICE: Decompressor pglz_decompress_hacked16 result 0.776815
NOTICE: Decompressor pglz_decompress_hacked32 result 0.777960
NOTICE: Decompressor pglz_decompress_vanilla result 0.721192
NOTICE: Payload 16398
NOTICE: Decompressor pglz_decompress_hacked result 0.337654
NOTICE: Decompressor pglz_decompress_hacked4 result 0.355452
NOTICE: Decompressor pglz_decompress_hacked8 result 0.351224
NOTICE: Decompressor pglz_decompress_hacked16 result 0.362548
NOTICE: Decompressor pglz_decompress_hacked32 result 0.356456
NOTICE: Decompressor pglz_decompress_vanilla result 0.837042
Comment: identical to Server results.
Laptop results:
NOTICE: Time to decompress one byte in ns:
NOTICE: Payload 000000010000000000000001
NOTICE: Decompressor pglz_decompress_hacked result 0.661469
NOTICE: Decompressor pglz_decompress_hacked4 result 0.638366
NOTICE: Decompressor pglz_decompress_hacked8 result 0.664377
NOTICE: Decompressor pglz_decompress_hacked16 result 0.696135
NOTICE: Decompressor pglz_decompress_hacked32 result 0.634825
NOTICE: Decompressor pglz_decompress_vanilla result 0.676560
NOTICE: Payload 000000010000000000000006
NOTICE: Decompressor pglz_decompress_hacked result 0.213921
NOTICE: Decompressor pglz_decompress_hacked4 result 0.224864
NOTICE: Decompressor pglz_decompress_hacked8 result 0.229394
NOTICE: Decompressor pglz_decompress_hacked16 result 0.218141
NOTICE: Decompressor pglz_decompress_hacked32 result 0.220954
NOTICE: Decompressor pglz_decompress_vanilla result 0.242412
NOTICE: Payload 000000010000000000000008
NOTICE: Decompressor pglz_decompress_hacked result 1.053417
NOTICE: Decompressor pglz_decompress_hacked4 result 1.063704
NOTICE: Decompressor pglz_decompress_hacked8 result 1.007211
NOTICE: Decompressor pglz_decompress_hacked16 result 1.145089
NOTICE: Decompressor pglz_decompress_hacked32 result 1.079702
NOTICE: Decompressor pglz_decompress_vanilla result 1.051557
NOTICE: Payload 16398
NOTICE: Decompressor pglz_decompress_hacked result 0.251690
NOTICE: Decompressor pglz_decompress_hacked4 result 0.268125
NOTICE: Decompressor pglz_decompress_hacked8 result 0.269248
NOTICE: Decompressor pglz_decompress_hacked16 result 0.277880
NOTICE: Decompressor pglz_decompress_hacked32 result 0.270290
NOTICE: Decompressor pglz_decompress_vanilla result 0.705652
Comment: decompress time on WAL segments is statistically indistinguishable between hacked and original versions. Hacked decompression of data file is 2x faster.
We are going to try these tests on cascade lake processors too.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-05-16 17:13 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 70+ messages in thread
From: Andrey Borodin @ 2019-05-16 17:13 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
> 15 мая 2019 г., в 15:06, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> Owners of AMD and ARM devices are welcome.
Yandex hardware RND guys gave me ARM server and Power9 server. They are looking for AMD and some new Intel boxes.
Meanwhile I made some enhancements to test suit:
1. I've added Shakespeare payload: concatenation of works of this prominent poet.
2. For each payload compute "sliced time" - time to decompress payload if it was sliced by 2Kb pieces or 8Kb pieces.
3. For each decompressor we compute "score": (sum of time to decompress each payload, each payload sliced by 2Kb and 8Kb) * 5 times
I've attached full test logs, meanwhile here's results for different platforms.
Intel Server
NOTICE: 00000: Decompressor pglz_decompress_hacked result 10.346763
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 11.192078
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 11.957727
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 14.262256
ARM Server
NOTICE: Decompressor pglz_decompress_hacked result 12.966668
NOTICE: Decompressor pglz_decompress_hacked8 result 13.004935
NOTICE: Decompressor pglz_decompress_hacked16 result 13.043015
NOTICE: Decompressor pglz_decompress_vanilla result 18.239242
Power9 Server
NOTICE: Decompressor pglz_decompress_hacked result 10.992974
NOTICE: Decompressor pglz_decompress_hacked8 result 11.747443
NOTICE: Decompressor pglz_decompress_hacked16 result 11.026342
NOTICE: Decompressor pglz_decompress_vanilla result 16.375315
Intel laptop
NOTICE: Decompressor pglz_decompress_hacked result 9.445808
NOTICE: Decompressor pglz_decompress_hacked8 result 9.105360
NOTICE: Decompressor pglz_decompress_hacked16 result 9.621833
NOTICE: Decompressor pglz_decompress_vanilla result 10.661968
From these results pglz_decompress_hacked looks best.
Best regards, Andrey Borodin.
Attachments:
pglz_benchmarks.txt (text/plain, 24.4K) download
Intel Server
pgload01f/postgres M # select test_pglz();
NOTICE: 00000: Time to decompress one byte in ns:
LOCATION: test_pglz, test_pglz.c:249
NOTICE: 00000: Payload 000000010000000000000001
LOCATION: test_pglz, test_pglz.c:252
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.142841
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.136788
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.145459
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.207186
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Payload 000000010000000000000001 sliced by 2Kb
LOCATION: test_pglz, test_pglz.c:259
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.747311
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.780535
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.826859
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 1.074147
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Payload 000000010000000000000001 sliced by 8Kb
LOCATION: test_pglz, test_pglz.c:266
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.680256
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.746620
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.759602
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 1.030290
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Payload 000000010000000000000006
LOCATION: test_pglz, test_pglz.c:252
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.040214
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.042281
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.042684
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.123906
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Payload 000000010000000000000006 sliced by 2Kb
LOCATION: test_pglz, test_pglz.c:259
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.368333
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.367808
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.367856
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.742352
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Payload 000000010000000000000006 sliced by 8Kb
LOCATION: test_pglz, test_pglz.c:266
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.276363
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.281572
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.282431
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.680208
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Payload 000000010000000000000008
LOCATION: test_pglz, test_pglz.c:252
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.272968
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.262132
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.290072
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.259988
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Payload 000000010000000000000008 sliced by 2Kb
LOCATION: test_pglz, test_pglz.c:259
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.990641
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 1.017916
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 1.100183
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.968778
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Payload 000000010000000000000008 sliced by 8Kb
LOCATION: test_pglz, test_pglz.c:266
NOTICE: 00000: Decompressor pglz_decompress_hacked result 1.341081
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 1.349056
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 1.449156
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 1.323843
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Payload 16398
LOCATION: test_pglz, test_pglz.c:252
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.120069
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.120604
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.122100
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.262950
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Payload 16398 sliced by 2Kb
LOCATION: test_pglz, test_pglz.c:259
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.417111
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.420177
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.421620
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 1.050393
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Payload 16398 sliced by 8Kb
LOCATION: test_pglz, test_pglz.c:266
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.332931
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.342865
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.343195
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.994908
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Payload shakespeare.txt
LOCATION: test_pglz, test_pglz.c:252
NOTICE: 00000: Decompressor pglz_decompress_hacked result 0.326101
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 0.397510
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 0.439144
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 0.426892
LOCATION: test_pglz, test_pglz.c:255
NOTICE: 00000: Payload shakespeare.txt sliced by 2Kb
LOCATION: test_pglz, test_pglz.c:259
NOTICE: 00000: Decompressor pglz_decompress_hacked result 2.285314
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 2.589329
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 2.808271
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 2.685089
LOCATION: test_pglz, test_pglz.c:262
NOTICE: 00000: Payload shakespeare.txt sliced by 8Kb
LOCATION: test_pglz, test_pglz.c:266
NOTICE: 00000: Decompressor pglz_decompress_hacked result 2.005230
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 2.336886
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 2.559097
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 2.431326
LOCATION: test_pglz, test_pglz.c:269
NOTICE: 00000:
Decompressor score (summ of all times):
LOCATION: test_pglz, test_pglz.c:274
NOTICE: 00000: Decompressor pglz_decompress_hacked result 10.346763
LOCATION: test_pglz, test_pglz.c:277
NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 11.192078
LOCATION: test_pglz, test_pglz.c:277
NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 11.957727
LOCATION: test_pglz, test_pglz.c:277
NOTICE: 00000: Decompressor pglz_decompress_vanilla result 14.262256
LOCATION: test_pglz, test_pglz.c:277
test_pglz
-----------
(1 row)
Time: 199436.688 ms (03:19.437)
root@pgload01f ~ # lscpu
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 32
On-line CPU(s) list: 0-31
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 45
Stepping: 7
CPU MHz: 1571.710
BogoMIPS: 4402.02
Virtualization: VT-x
L1d cache: 32K
L1i cache: 32K
L2 cache: 256K
L3 cache: 20480K
NUMA node0 CPU(s): 0-7,16-23
NUMA node1 CPU(s): 8-15,24-31
ARM Server
NOTICE: Time to decompress one byte in ns:
NOTICE: Payload 000000010000000000000001
NOTICE: Decompressor pglz_decompress_hacked result 0.172689
NOTICE: Decompressor pglz_decompress_hacked8 result 0.161536
NOTICE: Decompressor pglz_decompress_hacked16 result 0.156565
NOTICE: Decompressor pglz_decompress_vanilla result 0.240550
NOTICE: Payload 000000010000000000000001 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.922251
NOTICE: Decompressor pglz_decompress_hacked8 result 0.971508
NOTICE: Decompressor pglz_decompress_hacked16 result 0.914824
NOTICE: Decompressor pglz_decompress_vanilla result 1.361024
NOTICE: Payload 000000010000000000000001 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.851023
NOTICE: Decompressor pglz_decompress_hacked8 result 0.872946
NOTICE: Decompressor pglz_decompress_hacked16 result 0.834894
NOTICE: Decompressor pglz_decompress_vanilla result 1.272988
NOTICE: Payload 000000010000000000000006
NOTICE: Decompressor pglz_decompress_hacked result 0.058796
NOTICE: Decompressor pglz_decompress_hacked8 result 0.059826
NOTICE: Decompressor pglz_decompress_hacked16 result 0.059526
NOTICE: Decompressor pglz_decompress_vanilla result 0.155032
NOTICE: Payload 000000010000000000000006 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.490797
NOTICE: Decompressor pglz_decompress_hacked8 result 0.492752
NOTICE: Decompressor pglz_decompress_hacked16 result 0.491714
NOTICE: Decompressor pglz_decompress_vanilla result 1.041389
NOTICE: Payload 000000010000000000000006 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.391507
NOTICE: Decompressor pglz_decompress_hacked8 result 0.394464
NOTICE: Decompressor pglz_decompress_hacked16 result 0.394011
NOTICE: Decompressor pglz_decompress_vanilla result 0.917017
NOTICE: Payload 000000010000000000000008
NOTICE: Decompressor pglz_decompress_hacked result 0.255346
NOTICE: Decompressor pglz_decompress_hacked8 result 0.280497
NOTICE: Decompressor pglz_decompress_hacked16 result 0.283840
NOTICE: Decompressor pglz_decompress_vanilla result 0.279341
NOTICE: Payload 000000010000000000000008 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 1.085055
NOTICE: Decompressor pglz_decompress_hacked8 result 1.177943
NOTICE: Decompressor pglz_decompress_hacked16 result 1.239061
NOTICE: Decompressor pglz_decompress_vanilla result 1.214778
NOTICE: Payload 000000010000000000000008 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 1.338935
NOTICE: Decompressor pglz_decompress_hacked8 result 1.484537
NOTICE: Decompressor pglz_decompress_hacked16 result 1.522040
NOTICE: Decompressor pglz_decompress_vanilla result 1.463115
NOTICE: Payload 16398
NOTICE: Decompressor pglz_decompress_hacked result 0.124223
NOTICE: Decompressor pglz_decompress_hacked8 result 0.128246
NOTICE: Decompressor pglz_decompress_hacked16 result 0.126049
NOTICE: Decompressor pglz_decompress_vanilla result 0.427221
NOTICE: Payload 16398 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.640287
NOTICE: Decompressor pglz_decompress_hacked8 result 0.651233
NOTICE: Decompressor pglz_decompress_hacked16 result 0.644506
NOTICE: Decompressor pglz_decompress_vanilla result 2.082510
NOTICE: Payload 16398 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.506928
NOTICE: Decompressor pglz_decompress_hacked8 result 0.524692
NOTICE: Decompressor pglz_decompress_hacked16 result 0.520154
NOTICE: Decompressor pglz_decompress_vanilla result 1.982870
NOTICE: Payload shakespeare.txt
NOTICE: Decompressor pglz_decompress_hacked result 0.497708
NOTICE: Decompressor pglz_decompress_hacked8 result 0.442775
NOTICE: Decompressor pglz_decompress_hacked16 result 0.439889
NOTICE: Decompressor pglz_decompress_vanilla result 0.443202
NOTICE: Payload shakespeare.txt sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 2.912550
NOTICE: Decompressor pglz_decompress_hacked8 result 2.801293
NOTICE: Decompressor pglz_decompress_hacked16 result 2.841162
NOTICE: Decompressor pglz_decompress_vanilla result 2.797973
NOTICE: Payload shakespeare.txt sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 2.718573
NOTICE: Decompressor pglz_decompress_hacked8 result 2.560689
NOTICE: Decompressor pglz_decompress_hacked16 result 2.574780
NOTICE: Decompressor pglz_decompress_vanilla result 2.560231
NOTICE:
Decompressor score (summ of all times):
NOTICE: Decompressor pglz_decompress_hacked result 12.966668
NOTICE: Decompressor pglz_decompress_hacked8 result 13.004935
NOTICE: Decompressor pglz_decompress_hacked16 result 13.043015
NOTICE: Decompressor pglz_decompress_vanilla result 18.239242
test_pglz
-----------
(1 row)
x4mmm@thunderx2-gb:~$ lscpu
Architecture: aarch64
Byte Order: Little Endian
CPU(s): 128
On-line CPU(s) list: 0-127
Thread(s) per core: 2
Core(s) per socket: 32
Socket(s): 2
NUMA node(s): 2
NUMA node0 CPU(s): 0-63
NUMA node1 CPU(s): 64-127
Power9 Server
NOTICE: Time to decompress one byte in ns:
NOTICE: Payload 000000010000000000000001
NOTICE: Decompressor pglz_decompress_hacked result 0.143988
NOTICE: Decompressor pglz_decompress_hacked8 result 0.153475
NOTICE: Decompressor pglz_decompress_hacked16 result 0.144823
NOTICE: Decompressor pglz_decompress_vanilla result 0.231304
NOTICE: Payload 000000010000000000000001 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.827038
NOTICE: Decompressor pglz_decompress_hacked8 result 0.873506
NOTICE: Decompressor pglz_decompress_hacked16 result 0.838363
NOTICE: Decompressor pglz_decompress_vanilla result 1.285446
NOTICE: Payload 000000010000000000000001 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.749230
NOTICE: Decompressor pglz_decompress_hacked8 result 0.802684
NOTICE: Decompressor pglz_decompress_hacked16 result 0.763428
NOTICE: Decompressor pglz_decompress_vanilla result 1.206660
NOTICE: Payload 000000010000000000000006
NOTICE: Decompressor pglz_decompress_hacked result 0.043397
NOTICE: Decompressor pglz_decompress_hacked8 result 0.049469
NOTICE: Decompressor pglz_decompress_hacked16 result 0.049090
NOTICE: Decompressor pglz_decompress_vanilla result 0.148702
NOTICE: Payload 000000010000000000000006 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.402653
NOTICE: Decompressor pglz_decompress_hacked8 result 0.442338
NOTICE: Decompressor pglz_decompress_hacked16 result 0.421298
NOTICE: Decompressor pglz_decompress_vanilla result 0.994933
NOTICE: Payload 000000010000000000000006 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.305188
NOTICE: Decompressor pglz_decompress_hacked8 result 0.335515
NOTICE: Decompressor pglz_decompress_hacked16 result 0.326991
NOTICE: Decompressor pglz_decompress_vanilla result 0.865102
NOTICE: Payload 000000010000000000000008
NOTICE: Decompressor pglz_decompress_hacked result 0.281062
NOTICE: Decompressor pglz_decompress_hacked8 result 0.279365
NOTICE: Decompressor pglz_decompress_hacked16 result 0.205989
NOTICE: Decompressor pglz_decompress_vanilla result 0.238497
NOTICE: Payload 000000010000000000000008 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 1.060545
NOTICE: Decompressor pglz_decompress_hacked8 result 1.144373
NOTICE: Decompressor pglz_decompress_hacked16 result 0.918257
NOTICE: Decompressor pglz_decompress_vanilla result 0.927365
NOTICE: Payload 000000010000000000000008 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 1.361299
NOTICE: Decompressor pglz_decompress_hacked8 result 1.463842
NOTICE: Decompressor pglz_decompress_hacked16 result 1.105940
NOTICE: Decompressor pglz_decompress_vanilla result 1.229286
NOTICE: Payload 16398
NOTICE: Decompressor pglz_decompress_hacked result 0.063550
NOTICE: Decompressor pglz_decompress_hacked8 result 0.062770
NOTICE: Decompressor pglz_decompress_hacked16 result 0.059881
NOTICE: Decompressor pglz_decompress_vanilla result 0.338539
NOTICE: Payload 16398 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.406879
NOTICE: Decompressor pglz_decompress_hacked8 result 0.410176
NOTICE: Decompressor pglz_decompress_hacked16 result 0.397264
NOTICE: Decompressor pglz_decompress_vanilla result 1.803832
NOTICE: Payload 16398 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.329523
NOTICE: Decompressor pglz_decompress_hacked8 result 0.324563
NOTICE: Decompressor pglz_decompress_hacked16 result 0.314079
NOTICE: Decompressor pglz_decompress_vanilla result 1.717139
NOTICE: Payload shakespeare.txt
NOTICE: Decompressor pglz_decompress_hacked result 0.395271
NOTICE: Decompressor pglz_decompress_hacked8 result 0.433101
NOTICE: Decompressor pglz_decompress_hacked16 result 0.437657
NOTICE: Decompressor pglz_decompress_vanilla result 0.430828
NOTICE: Payload shakespeare.txt sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 2.395521
NOTICE: Decompressor pglz_decompress_hacked8 result 2.559994
NOTICE: Decompressor pglz_decompress_hacked16 result 2.593140
NOTICE: Decompressor pglz_decompress_vanilla result 2.551323
NOTICE: Payload shakespeare.txt sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 2.227830
NOTICE: Decompressor pglz_decompress_hacked8 result 2.412272
NOTICE: Decompressor pglz_decompress_hacked16 result 2.450142
NOTICE: Decompressor pglz_decompress_vanilla result 2.406361
NOTICE:
Decompressor score (summ of all times):
NOTICE: Decompressor pglz_decompress_hacked result 10.992974
NOTICE: Decompressor pglz_decompress_hacked8 result 11.747443
NOTICE: Decompressor pglz_decompress_hacked16 result 11.026342
NOTICE: Decompressor pglz_decompress_vanilla result 16.375315
test_pglz
x4mmm@power:~$ lscpu
Architecture: ppc64le
Byte Order: Little Endian
CPU(s): 144
On-line CPU(s) list: 0-143
Thread(s) per core: 4
Core(s) per socket: 18
Socket(s): 2
NUMA node(s): 2
Model: 2.2 (pvr 004e 1202)
Model name: POWER9, altivec supported
CPU max MHz: 3800.0000
CPU min MHz: 2166.0000
L1d cache: 32K
L1i cache: 32K
L2 cache: 512K
L3 cache: 10240K
NUMA node0 CPU(s): 0-71
NUMA node8 CPU(s): 72-143
MacBook Pro15 2015
NOTICE: Time to decompress one byte in ns:
NOTICE: Payload 000000010000000000000001
NOTICE: Decompressor pglz_decompress_hacked result 0.128627
NOTICE: Decompressor pglz_decompress_hacked8 result 0.127258
NOTICE: Decompressor pglz_decompress_hacked16 result 0.126970
NOTICE: Decompressor pglz_decompress_vanilla result 0.137920
NOTICE: Payload 000000010000000000000001 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.653553
NOTICE: Decompressor pglz_decompress_hacked8 result 0.657177
NOTICE: Decompressor pglz_decompress_hacked16 result 0.664270
NOTICE: Decompressor pglz_decompress_vanilla result 0.741029
NOTICE: Payload 000000010000000000000001 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.609875
NOTICE: Decompressor pglz_decompress_hacked8 result 0.603628
NOTICE: Decompressor pglz_decompress_hacked16 result 0.609636
NOTICE: Decompressor pglz_decompress_vanilla result 0.685763
NOTICE: Payload 000000010000000000000006
NOTICE: Decompressor pglz_decompress_hacked result 0.043464
NOTICE: Decompressor pglz_decompress_hacked8 result 0.048900
NOTICE: Decompressor pglz_decompress_hacked16 result 0.045280
NOTICE: Decompressor pglz_decompress_vanilla result 0.048118
NOTICE: Payload 000000010000000000000006 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.289261
NOTICE: Decompressor pglz_decompress_hacked8 result 0.312209
NOTICE: Decompressor pglz_decompress_hacked16 result 0.314140
NOTICE: Decompressor pglz_decompress_vanilla result 0.357211
NOTICE: Payload 000000010000000000000006 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.218308
NOTICE: Decompressor pglz_decompress_hacked8 result 0.233817
NOTICE: Decompressor pglz_decompress_hacked16 result 0.236726
NOTICE: Decompressor pglz_decompress_vanilla result 0.277042
NOTICE: Payload 000000010000000000000008
NOTICE: Decompressor pglz_decompress_hacked result 0.221484
NOTICE: Decompressor pglz_decompress_hacked8 result 0.213249
NOTICE: Decompressor pglz_decompress_hacked16 result 0.242667
NOTICE: Decompressor pglz_decompress_vanilla result 0.220056
NOTICE: Payload 000000010000000000000008 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.743186
NOTICE: Decompressor pglz_decompress_hacked8 result 0.744402
NOTICE: Decompressor pglz_decompress_hacked16 result 0.847566
NOTICE: Decompressor pglz_decompress_vanilla result 0.806570
NOTICE: Payload 000000010000000000000008 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.988770
NOTICE: Decompressor pglz_decompress_hacked8 result 0.983024
NOTICE: Decompressor pglz_decompress_hacked16 result 1.130629
NOTICE: Decompressor pglz_decompress_vanilla result 1.028109
NOTICE: Payload 16398
NOTICE: Decompressor pglz_decompress_hacked result 0.051779
NOTICE: Decompressor pglz_decompress_hacked8 result 0.052629
NOTICE: Decompressor pglz_decompress_hacked16 result 0.055634
NOTICE: Decompressor pglz_decompress_vanilla result 0.145399
NOTICE: Payload 16398 sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.334166
NOTICE: Decompressor pglz_decompress_hacked8 result 0.329803
NOTICE: Decompressor pglz_decompress_hacked16 result 0.359756
NOTICE: Decompressor pglz_decompress_vanilla result 0.780205
NOTICE: Payload 16398 sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 0.262448
NOTICE: Decompressor pglz_decompress_hacked8 result 0.267446
NOTICE: Decompressor pglz_decompress_hacked16 result 0.276952
NOTICE: Decompressor pglz_decompress_vanilla result 0.729827
NOTICE: Payload shakespeare.txt
NOTICE: Decompressor pglz_decompress_hacked result 0.370638
NOTICE: Decompressor pglz_decompress_hacked8 result 0.342184
NOTICE: Decompressor pglz_decompress_hacked16 result 0.352841
NOTICE: Decompressor pglz_decompress_vanilla result 0.356302
NOTICE: Payload shakespeare.txt sliced by 2Kb
NOTICE: Decompressor pglz_decompress_hacked result 2.385342
NOTICE: Decompressor pglz_decompress_hacked8 result 2.185166
NOTICE: Decompressor pglz_decompress_hacked16 result 2.281774
NOTICE: Decompressor pglz_decompress_vanilla result 2.282807
NOTICE: Payload shakespeare.txt sliced by 8Kb
NOTICE: Decompressor pglz_decompress_hacked result 2.144908
NOTICE: Decompressor pglz_decompress_hacked8 result 2.004468
NOTICE: Decompressor pglz_decompress_hacked16 result 2.076991
NOTICE: Decompressor pglz_decompress_vanilla result 2.065610
NOTICE:
Decompressor score (summ of all times):
NOTICE: Decompressor pglz_decompress_hacked result 9.445808
NOTICE: Decompressor pglz_decompress_hacked8 result 9.105360
NOTICE: Decompressor pglz_decompress_hacked16 result 9.621833
NOTICE: Decompressor pglz_decompress_vanilla result 10.661968
test_pglz
-----------
(1 row)
x4mmm-osx:test_pglz x4mmm$ system_profiler SPHardwareDataType
Hardware:
Hardware Overview:
Model Name: MacBook Pro
Model Identifier: MacBookPro11,4
Processor Name: Intel Core i7
Processor Speed: 2,2 GHz
Number of Processors: 1
Total Number of Cores: 4
L2 Cache (per Core): 256 KB
L3 Cache: 6 MB
Memory: 16 GB
Boot ROM Version: 189.0.0.0.0
SMC Version (system): 2.29f24
Serial Number (system): C02SN6TKG8WN
Hardware UUID: 8967E265-30D7-5447-876C-2B4618497347
* Re: pglz performance
@ 2019-05-17 01:44 Michael Paquier <michael@paquier.xyz>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 70+ messages in thread
From: Michael Paquier @ 2019-05-17 01:44 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Thu, May 16, 2019 at 10:13:22PM +0500, Andrey Borodin wrote:
> Meanwhile I made some enhancements to test suit:
> Intel Server
> NOTICE: 00000: Decompressor pglz_decompress_hacked result 10.346763
> NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 11.192078
> NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 11.957727
> NOTICE: 00000: Decompressor pglz_decompress_vanilla result 14.262256
>
> ARM Server
> NOTICE: Decompressor pglz_decompress_hacked result 12.966668
> NOTICE: Decompressor pglz_decompress_hacked8 result 13.004935
> NOTICE: Decompressor pglz_decompress_hacked16 result 13.043015
> NOTICE: Decompressor pglz_decompress_vanilla result 18.239242
>
> Power9 Server
> NOTICE: Decompressor pglz_decompress_hacked result 10.992974
> NOTICE: Decompressor pglz_decompress_hacked8 result 11.747443
> NOTICE: Decompressor pglz_decompress_hacked16 result 11.026342
> NOTICE: Decompressor pglz_decompress_vanilla result 16.375315
>
> Intel laptop
> NOTICE: Decompressor pglz_decompress_hacked result 9.445808
> NOTICE: Decompressor pglz_decompress_hacked8 result 9.105360
> NOTICE: Decompressor pglz_decompress_hacked16 result 9.621833
> NOTICE: Decompressor pglz_decompress_vanilla result 10.661968
>
> From these results pglz_decompress_hacked looks best.
That's nice.
From the numbers you are presenting here, all of them are much better
than the original, and there is not much difference between any of the
patched versions. Having a 20%~30% improvement with a patch is very
nice.
After that comes the simplicity and the future maintainability of what
is proposed. I am not much into accepting a patch which has a 1%~2%
impact for some hardwares and makes pglz much more complex and harder
to understand. But I am really eager to see a patch with at least a
10% improvement which remains simple, even more if it simplifies the
logic used in pglz.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-05-17 10:59 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 70+ messages in thread
From: Andrey Borodin @ 2019-05-17 10:59 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
> 17 мая 2019 г., в 6:44, Michael Paquier <michael@paquier.xyz> написал(а):
>
> That's nice.
>
> From the numbers you are presenting here, all of them are much better
> than the original, and there is not much difference between any of the
> patched versions. Having a 20%~30% improvement with a patch is very
> nice.
>
> After that comes the simplicity and the future maintainability of what
> is proposed. I am not much into accepting a patch which has a 1%~2%
> impact for some hardwares and makes pglz much more complex and harder
> to understand. But I am really eager to see a patch with at least a
> 10% improvement which remains simple, even more if it simplifies the
> logic used in pglz.
Here are patches for both winning versions. I'll place them on CF.
My gut feeling is pglz_decompress_hacked8 should be better, but on most architectures benchmarks show opposite.
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
pglz_decompress_hacked8.diff (application/octet-stream, 1.0K) download
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 97b0e40e40..b4d0f255bc 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -731,15 +731,30 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT. The copied areas could overlap, to preven possible
+ * uncertanity, we copy only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ if (len > 8)
{
- *dp = dp[-off];
- dp++;
+ while (off <= len)
+ {
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp+=off;
+ off *= 2;
+ }
+ memcpy(dp, dp - off, len);
+ dp+=len;
+ }
+ else
+ {
+ /* use byte-loop for small matches */
+ while (len--)
+ {
+ *dp = dp[-off];
+ dp++;
+ }
}
}
else
pglz_decompress_hacked.diff (application/octet-stream, 881B) download
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 97b0e40e40..55b10ed06f 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -731,16 +731,19 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT. The copied areas could overlap, to preven possible
+ * uncertanity, we copy only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ while (off <= len)
{
- *dp = dp[-off];
- dp++;
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp+=off;
+ off *= 2;
}
+ memcpy(dp, dp - off, len);
+ dp+=len;
}
else
{
* Re: pglz performance
@ 2019-05-17 13:40 Gasper Zejn <zejn@owca.info>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 70+ messages in thread
From: Gasper Zejn @ 2019-05-17 13:40 UTC (permalink / raw)
To: pgsql-hackers@lists.postgresql.org
On 16. 05. 19 19:13, Andrey Borodin wrote:
>
>> 15 мая 2019 г., в 15:06, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>>
>> Owners of AMD and ARM devices are welcome.
I've tested according to instructions at the test repo
https://github.com/x4m/test_pglz
Test_pglz is at a97f63b and postgres at 6ba500.
Hardware is desktop AMD Ryzen 5 2600, 32GB RAM
Decompressor score (summ of all times):
NOTICE: Decompressor pglz_decompress_hacked result 6.988909
NOTICE: Decompressor pglz_decompress_hacked8 result 7.562619
NOTICE: Decompressor pglz_decompress_hacked16 result 8.316957
NOTICE: Decompressor pglz_decompress_vanilla result 10.725826
Attached is the full test run, if needed.
Kind regards,
Gasper
> Yandex hardware RND guys gave me ARM server and Power9 server. They are looking for AMD and some new Intel boxes.
>
> Meanwhile I made some enhancements to test suit:
> 1. I've added Shakespeare payload: concatenation of works of this prominent poet.
> 2. For each payload compute "sliced time" - time to decompress payload if it was sliced by 2Kb pieces or 8Kb pieces.
> 3. For each decompressor we compute "score": (sum of time to decompress each payload, each payload sliced by 2Kb and 8Kb) * 5 times
>
> I've attached full test logs, meanwhile here's results for different platforms.
>
> Intel Server
> NOTICE: 00000: Decompressor pglz_decompress_hacked result 10.346763
> NOTICE: 00000: Decompressor pglz_decompress_hacked8 result 11.192078
> NOTICE: 00000: Decompressor pglz_decompress_hacked16 result 11.957727
> NOTICE: 00000: Decompressor pglz_decompress_vanilla result 14.262256
>
> ARM Server
> NOTICE: Decompressor pglz_decompress_hacked result 12.966668
> NOTICE: Decompressor pglz_decompress_hacked8 result 13.004935
> NOTICE: Decompressor pglz_decompress_hacked16 result 13.043015
> NOTICE: Decompressor pglz_decompress_vanilla result 18.239242
>
> Power9 Server
> NOTICE: Decompressor pglz_decompress_hacked result 10.992974
> NOTICE: Decompressor pglz_decompress_hacked8 result 11.747443
> NOTICE: Decompressor pglz_decompress_hacked16 result 11.026342
> NOTICE: Decompressor pglz_decompress_vanilla result 16.375315
>
> Intel laptop
> NOTICE: Decompressor pglz_decompress_hacked result 9.445808
> NOTICE: Decompressor pglz_decompress_hacked8 result 9.105360
> NOTICE: Decompressor pglz_decompress_hacked16 result 9.621833
> NOTICE: Decompressor pglz_decompress_vanilla result 10.661968
>
> From these results pglz_decompress_hacked looks best.
>
> Best regards, Andrey Borodin.
>
Attachments:
pglz_benchmarks_amd.txt (text/plain, 83.0K) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-05-18 06:44 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Gasper Zejn <zejn@owca.info>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-05-18 06:44 UTC (permalink / raw)
To: Gasper Zejn <zejn@owca.info>; +Cc: pgsql-hackers@lists.postgresql.org
> 17 мая 2019 г., в 18:40, Gasper Zejn <zejn@owca.info> написал(а):
>
> I've tested according to instructions at the test repo
> https://github.com/x4m/test_pglz
>
> Test_pglz is at a97f63b and postgres at 6ba500.
>
> Hardware is desktop AMD Ryzen 5 2600, 32GB RAM
>
> Decompressor score (summ of all times):
>
> NOTICE: Decompressor pglz_decompress_hacked result 6.988909
> NOTICE: Decompressor pglz_decompress_hacked8 result 7.562619
> NOTICE: Decompressor pglz_decompress_hacked16 result 8.316957
> NOTICE: Decompressor pglz_decompress_vanilla result 10.725826
Thanks, Gasper! Basically we observe same 0.65 time reduction here.
That's very good that we have independent scores.
I'm still somewhat not sure that score is fair, on payload 000000010000000000000008 we have vanilla decompression sometimes slower than hacked by few percents. And this is especially visible on AMD. Degradation for 000000010000000000000008 sliced by 8Kb reaches 10%
I think this is because 000000010000000000000008 have highest entropy.It is almost random and matches are very short, but present.
000000010000000000000008
Entropy = 4.360546 bits per byte.
000000010000000000000006
Entropy = 1.450059 bits per byte.
000000010000000000000001
Entropy = 2.944235 bits per byte.
shakespeare.txt
Entropy = 3.603659 bits per byte
16398
Entropy = 1.897640 bits per byte.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-05-23 14:27 Binguo Bao <djydewang@gmail.com>
0 siblings, 0 replies; 70+ messages in thread
From: Binguo Bao @ 2019-05-23 14:27 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: pgsql-hackers
Hi hackers!
I am a student participating in GSoC 2019. I am looking forward to working
with you all and learning from you.
My project would aim to provide the ability to de-TOAST a fully TOAST'd and
compressed field using an iterator.
For more details, please take a look at my proposal[0]. Any suggestions or
comments about my immature ideas would be much appreciated:)
I've implemented the first step of the project, the segment pglz
compression provides the ability to get the subset of the raw data without
decompressing the entire field.
And I've done some test[1] for the compressor. The test result is as
follows:
NOTICE: Test summary:
NOTICE: Payload 000000010000000000000001
NOTICE: Decompressor name | Compression time (ns/bit) |
Decompression time (ns/bit) | ratio
NOTICE: pglz_decompress_hacked | 23.747444 |
0.578344 | 0.159809
NOTICE: pglz_decompress_hacked8 | 23.764193 |
0.677800 | 0.159809
NOTICE: pglz_decompress_hacked16 | 23.740351 |
0.704730 | 0.159809
NOTICE: pglz_decompress_vanilla | 23.797917 |
1.227868 | 0.159809
NOTICE: pglz_decompress_hacked_seg | 12.261808 |
0.625634 | 0.184952
Comment: Compression speed increased by nearly 100% with compression rate
dropped by 15%
NOTICE: Payload 000000010000000000000001 sliced by 2Kb
NOTICE: pglz_decompress_hacked | 12.616956 |
0.621223 | 0.156953
NOTICE: pglz_decompress_hacked8 | 12.583685 |
0.756741 | 0.156953
NOTICE: pglz_decompress_hacked16 | 12.512636 |
0.774980 | 0.156953
NOTICE: pglz_decompress_vanilla | 12.493062 |
1.262820 | 0.156953
NOTICE: pglz_decompress_hacked_seg | 11.986554 |
0.622654 | 0.159590
NOTICE: Payload 000000010000000000000001 sliced by 4Kb
NOTICE: pglz_decompress_hacked | 15.514469 |
0.565565 | 0.154213
NOTICE: pglz_decompress_hacked8 | 15.529144 |
0.699675 | 0.154213
NOTICE: pglz_decompress_hacked16 | 15.514040 |
0.721145 | 0.154213
NOTICE: pglz_decompress_vanilla | 15.558958 |
1.237237 | 0.154213
NOTICE: pglz_decompress_hacked_seg | 14.650309 |
0.563228 | 0.153652
NOTICE: Payload 000000010000000000000006
NOTICE: Decompressor name | Compression time (ns/bit) |
Decompression time (ns/bit) | ratio
NOTICE: pglz_decompress_hacked | 8.610177 |
0.153577 | 0.052294
NOTICE: pglz_decompress_hacked8 | 8.566785 |
0.168002 | 0.052294
NOTICE: pglz_decompress_hacked16 | 8.643126 |
0.167537 | 0.052294
NOTICE: pglz_decompress_vanilla | 8.574498 |
0.930738 | 0.052294
NOTICE: pglz_decompress_hacked_seg | 7.394731 |
0.171924 | 0.056081
NOTICE: Payload 000000010000000000000006 sliced by 2Kb
NOTICE: pglz_decompress_hacked | 6.724060 |
0.295043 | 0.065541
NOTICE: pglz_decompress_hacked8 | 6.623018 |
0.318527 | 0.065541
NOTICE: pglz_decompress_hacked16 | 6.898034 |
0.318360 | 0.065541
NOTICE: pglz_decompress_vanilla | 6.712711 |
1.045430 | 0.065541
NOTICE: pglz_decompress_hacked_seg | 6.630743 |
0.302589 | 0.068471
NOTICE: Payload 000000010000000000000006 sliced by 4Kb
NOTICE: pglz_decompress_hacked | 6.624067 |
0.220942 | 0.058865
NOTICE: pglz_decompress_hacked8 | 6.659424 |
0.240183 | 0.058865
NOTICE: pglz_decompress_hacked16 | 6.763864 |
0.240564 | 0.058865
NOTICE: pglz_decompress_vanilla | 6.743574 |
0.985348 | 0.058865
NOTICE: pglz_decompress_hacked_seg | 6.613123 |
0.227582 | 0.060330
NOTICE: Payload 000000010000000000000008
NOTICE: Decompressor name | Compression time (ns/bit) |
Decompression time (ns/bit) | ratio
NOTICE: pglz_decompress_hacked | 52.425957 |
1.050544 | 0.498941
NOTICE: pglz_decompress_hacked8 | 52.204561 |
1.261592 | 0.498941
NOTICE: pglz_decompress_hacked16 | 52.328491 |
1.466751 | 0.498941
NOTICE: pglz_decompress_vanilla | 52.465308 |
1.341271 | 0.498941
NOTICE: pglz_decompress_hacked_seg | 31.896341 |
1.113260 | 0.600998
NOTICE: Payload 000000010000000000000008 sliced by 2Kb
NOTICE: pglz_decompress_hacked | 30.620611 |
0.768542 | 0.351941
NOTICE: pglz_decompress_hacked8 | 30.557334 |
0.907421 | 0.351941
NOTICE: pglz_decompress_hacked16 | 32.064903 |
1.208913 | 0.351941
NOTICE: pglz_decompress_vanilla | 30.489886 |
1.014197 | 0.351941
NOTICE: pglz_decompress_hacked_seg | 27.145243 |
0.774193 | 0.352868
NOTICE: Payload 000000010000000000000008 sliced by 4Kb
NOTICE: pglz_decompress_hacked | 36.567903 |
1.054633 | 0.514047
NOTICE: pglz_decompress_hacked8 | 36.459124 |
1.267731 | 0.514047
NOTICE: pglz_decompress_hacked16 | 36.791718 |
1.479650 | 0.514047
NOTICE: pglz_decompress_vanilla | 36.241913 |
1.303136 | 0.514047
NOTICE: pglz_decompress_hacked_seg | 31.526327 |
1.059926 | 0.526875
NOTICE: Payload 16398
NOTICE: Decompressor name | Compression time (ns/bit) |
Decompression time (ns/bit) | ratio
NOTICE: pglz_decompress_hacked | 9.508625 |
0.435190 | 0.071816
NOTICE: pglz_decompress_hacked8 | 9.546987 |
0.473871 | 0.071816
NOTICE: pglz_decompress_hacked16 | 9.534496 |
0.471662 | 0.071816
NOTICE: pglz_decompress_vanilla | 9.559053 |
1.352561 | 0.071816
NOTICE: pglz_decompress_hacked_seg | 8.479486 |
0.441536 | 0.073232
NOTICE: Payload 16398 sliced by 2Kb
NOTICE: pglz_decompress_hacked | 6.808167 |
0.326570 | 0.082775
NOTICE: pglz_decompress_hacked8 | 6.790743 |
0.361720 | 0.082775
NOTICE: pglz_decompress_hacked16 | 6.886097 |
0.364549 | 0.082775
NOTICE: pglz_decompress_vanilla | 6.918429 |
1.191265 | 0.082775
NOTICE: pglz_decompress_hacked_seg | 6.752811 |
0.340805 | 0.085705
NOTICE: Payload 16398 sliced by 4Kb
NOTICE: pglz_decompress_hacked | 7.244472 |
0.261872 | 0.076860
NOTICE: pglz_decompress_hacked8 | 7.290275 |
0.295988 | 0.076860
NOTICE: pglz_decompress_hacked16 | 7.340706 |
0.294683 | 0.076860
NOTICE: pglz_decompress_vanilla | 7.429289 |
1.151645 | 0.076860
NOTICE: pglz_decompress_hacked_seg | 7.054166 |
0.267896 | 0.078325
NOTICE: Payload shakespeare.txt
NOTICE: Decompressor name | Compression time (ns/bit) |
Decompression time (ns/bit) | ratio
NOTICE: pglz_decompress_hacked | 25.998753 |
1.345542 | 0.281363
NOTICE: pglz_decompress_hacked8 | 26.121630 |
1.917667 | 0.281363
NOTICE: pglz_decompress_hacked16 | 26.139312 |
2.101329 | 0.281363
NOTICE: pglz_decompress_vanilla | 26.155571 |
2.082123 | 0.281363
NOTICE: pglz_decompress_hacked_seg | 16.792089 |
1.951269 | 0.436558
Comment: In this case, the compression rate has dropped dramatically.
NOTICE: Payload shakespeare.txt sliced by 2Kb
NOTICE: pglz_decompress_hacked | 14.992793 |
1.923663 | 0.436270
NOTICE: pglz_decompress_hacked8 | 14.982428 |
2.695319 | 0.436270
NOTICE: pglz_decompress_hacked16 | 15.211803 |
2.846615 | 0.436270
NOTICE: pglz_decompress_vanilla | 15.113214 |
2.580098 | 0.436270
NOTICE: pglz_decompress_hacked_seg | 15.120852 |
1.922596 | 0.439199
NOTICE: Payload shakespeare.txt sliced by 4Kb
NOTICE: pglz_decompress_hacked | 18.083400 |
1.687598 | 0.366936
NOTICE: pglz_decompress_hacked8 | 18.185038 |
2.395928 | 0.366936
NOTICE: pglz_decompress_hacked16 | 18.096120 |
2.554812 | 0.366936
NOTICE: pglz_decompress_vanilla | 18.435380 |
2.329129 | 0.366936
NOTICE: pglz_decompress_hacked_seg | 18.103267 |
1.705517 | 0.368400
NOTICE:
Decompressor score (summ of all times):
NOTICE: Decompressor pglz_decompress_hacked result 11.288848
NOTICE: Decompressor pglz_decompress_hacked8 result 14.438165
NOTICE: Decompressor pglz_decompress_hacked16 result 15.716280
NOTICE: Decompressor pglz_decompress_vanilla result 21.034867
NOTICE: Decompressor pglz_decompress_hacked_seg result 12.090609
NOTICE:
compressor score (summ of all times):
NOTICE: compressor pglz_compress_vanilla result 276.776671
NOTICE: compressor pglz_compress_hacked_seg result 222.407850
There are some questions now:
1. The compression algorithm is not compatible with the original
compression algorithm now.
2. If the idea works, we need to test more data, what kind of data is more
appropriate?
Any comments are much appreciated.
Best regards, Binguo Bao.
[0]
https://docs.google.com/document/d/1V4oXV5vGrGx24deBTKKM7bVdO3Cy-zfj-wQ4dkBUCl4/edit
[1] https://github.com/djydewang/test_pglz
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-06-24 08:44 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-06-24 08:44 UTC (permalink / raw)
To: Gasper Zejn <zejn@owca.info>; +Cc: pgsql-hackers@lists.postgresql.org
> 18 мая 2019 г., в 11:44, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
Hi!
Here's rebased version of patches.
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch (application/octet-stream, 1.4K) download
From dc7e8dee9d74058426dcdab44c7023089cf7e10b Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 24 Jun 2019 13:38:20 +0500
Subject: [PATCH] Use memcpy in pglz decompression for long matches
---
src/common/pg_lzcompress.c | 27 +++++++++++++++++++++------
1 file changed, 21 insertions(+), 6 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 988b3987d0..022ae7d434 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -731,15 +731,30 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT. The copied areas could overlap, to preven possible
+ * uncertanity, we copy only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ if (len > 8)
{
- *dp = dp[-off];
- dp++;
+ while (off <= len)
+ {
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp+=off;
+ off *= 2;
+ }
+ memcpy(dp, dp - off, len);
+ dp+=len;
+ }
+ else
+ {
+ /* use byte-loop for small matches */
+ while (len--)
+ {
+ *dp = dp[-off];
+ dp++;
+ }
}
}
else
--
2.20.1
=
0001-Use-memcpy-in-pglz-decompression.patch (application/octet-stream, 1.2K) download
From 0cf4687de005c81097aaf5a1055488e8f6b3c279 Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 24 Jun 2019 13:36:47 +0500
Subject: [PATCH] Use memcpy in pglz decompression
---
src/common/pg_lzcompress.c | 15 +++++++++------
1 file changed, 9 insertions(+), 6 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 988b3987d0..f199be109f 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -731,16 +731,19 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT. The copied areas could overlap, to preven possible
+ * uncertanity, we copy only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ while (off <= len)
{
- *dp = dp[-off];
- dp++;
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp+=off;
+ off *= 2;
}
+ memcpy(dp, dp - off, len);
+ dp+=len;
}
else
{
--
2.20.1
=
* Re: pglz performance
@ 2019-06-27 18:33 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Michael Paquier <michael@paquier.xyz>
1 sibling, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-06-27 18:33 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
> 13 мая 2019 г., в 12:14, Michael Paquier <michael@paquier.xyz> написал(а):
>
> Decompression can matter a lot for mostly-read workloads and
> compression can become a bottleneck for heavy-insert loads, so
> improving compression or decompression should be two separate
> problems, not two problems linked. Any improvement in one or the
> other, or even both, is nice to have.
Here's patch hacked by Vladimir for compression.
Key differences (as far as I see, maybe Vladimir will post more complete list of optimizations):
1. Use functions instead of macro-functions: not surprisingly it's easier to optimize them and provide less constraints for compiler to optimize.
2. More compact hash table: use indexes instead of pointers.
3. More robust segment comparison: like memcmp, but return index of first different byte
In weighted mix of different data (same as for compression), overall speedup is x1.43 on my machine.
Current implementation is integrated into test_pglz suit for benchmarking purposes[0].
Best regards, Andrey Borodin.
[0] https://github.com/x4m/test_pglz
Attachments:
0001-Reorganize-pglz-compression-code.patch (application/octet-stream, 24.2K) download
From f45e77eb7adff3e6142bc36ff57b52d84f80497d Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Thu, 27 Jun 2019 23:18:21 +0500
Subject: [PATCH] Reorganize pglz compression code
Convert macro-functions to regular functions, use more compact hash table and some other optimizations. Total speedup about x1.4
---
src/common/pg_lzcompress.c | 513 ++++++++++++++++++++++---------------
1 file changed, 305 insertions(+), 208 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 988b3987d0..7127927cc9 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -118,13 +118,13 @@
* our 4K maximum look-back distance is too small.
*
* The compressor creates a table for lists of positions.
- * For each input position (except the last 3), a hash key is
+ * For each input position (except the last 4), a hash key is
* built from the 4 next input bytes and the position remembered
* in the appropriate list. Thus, the table points to linked
* lists of likely to be at least in the first 4 characters
* matching strings. This is done on the fly while the input
* is compressed into the output area. Table entries are only
- * kept for the last 4096 input positions, since we cannot use
+ * kept for the last 4094 input positions, since we cannot use
* back-pointers larger than that anyway. The size of the hash
* table is chosen based on the size of the input - a larger table
* has a larger startup cost, as it needs to be initialized to
@@ -146,17 +146,15 @@
* used for the next tag in the output.
*
* For each subsequent entry in the history list, the "good_match"
- * is lowered by 10%. So the compressor will be more happy with
- * short matches the farer it has to go back in the history.
+ * is lowered roughly by 10%. So the compressor will be more happy
+ * with short matches the farer it has to go back in the history.
* Another "speed against ratio" preference characteristic of
* the algorithm.
*
- * Thus there are 3 stop conditions for the lookup of matches:
+ * Thus there are 2 stop conditions for the lookup of matches:
*
* - a match >= good_match is found
* - there are no more history entries to look at
- * - the next history entry is already too far back
- * to be coded into a tag.
*
* Finally the match algorithm checks that at least a match
* of 3 or more bytes has been found, because that is the smallest
@@ -172,6 +170,11 @@
*
* Jan Wieck
*
+ * Many thanks to Yann Collet for his article about unaligned
+ * memory access.
+ *
+ * Leskov Vladimir
+ *
* Copyright (c) 1999-2019, PostgreSQL Global Development Group
*
* src/common/pg_lzcompress.c
@@ -188,12 +191,57 @@
#include "common/pg_lzcompress.h"
+/**************************************
+ * CPU Feature Detection *
+ **************************************/
+/* PGLZ_FORCE_MEMORY_ACCESS
+ * By default, access to unaligned memory is controlled by `memcpy()`, which is safe and portable.
+ * Unfortunately, on some target/compiler combinations, the generated assembly is sub-optimal.
+ * The below switch allow to select different access method for improved performance.
+ * Method 0 (default) : use `memcpy()`. Safe and portable.
+ * Method 1 : direct access. This method is portable but violate C standard.
+ * It can generate buggy code on targets which assembly generation depends on alignment.
+ * But in some circumstances, it's the only known way to get the most performance (ie GCC + ARMv6)
+ * See https://fastcompression.blogspot.fr/2015/08/accessing-unaligned-memory.html for details.
+ * Prefer these methods in priority order (0 > 1)
+ */
+#ifndef PGLZ_FORCE_MEMORY_ACCESS /* can be defined externally */
+# if defined(__GNUC__) && \
+ ( defined(__ARM_ARCH_6__) || defined(__ARM_ARCH_6J__) || defined(__ARM_ARCH_6K__) \
+ || defined(__ARM_ARCH_6Z__) || defined(__ARM_ARCH_6ZK__) || defined(__ARM_ARCH_6T2__) )
+# define PGLZ_FORCE_MEMORY_ACCESS 1
+# endif
+#endif
+
+
+#if defined(__x86_64__)
+ typedef uint64 reg_t; /* 64-bits in x32 mode */
+#else
+ typedef size_t reg_t; /* 32-bits in x32 mode */
+#endif
+
+
+#if defined(PGLZ_FORCE_MEMORY_ACCESS) && (PGLZ_FORCE_MEMORY_ACCESS==1)
+/* lie to the compiler about data alignment; use with caution */
+
+static uint32 pglz_read32(const void* ptr) { return *(const uint32*) ptr; }
+
+#else /* safe and portable access using memcpy() */
+
+static uint32 pglz_read32(const void* ptr)
+{
+ uint32 val; memcpy(&val, ptr, sizeof(val)); return val;
+}
+
+#endif /* PGLZ_FORCE_MEMORY_ACCESS */
+
+
/* ----------
* Local definitions
* ----------
*/
#define PGLZ_MAX_HISTORY_LISTS 8192 /* must be power of 2 */
-#define PGLZ_HISTORY_SIZE 4096
+#define PGLZ_HISTORY_SIZE 0x0fff - 1 /* to avoid compare in iteration */
#define PGLZ_MAX_MATCH 273
@@ -202,17 +250,16 @@
*
* Linked list for the backward history lookup
*
- * All the entries sharing a hash key are linked in a doubly linked list.
- * This makes it easy to remove an entry when it's time to recycle it
- * (because it's more than 4K positions old).
+ * All the entries sharing a hash key are linked in a singly linked list.
+ * Links are not changed during insertion in order to speed it up.
+ * Instead more complicated stop condition is used during list iteration.
* ----------
*/
typedef struct PGLZ_HistEntry
{
- struct PGLZ_HistEntry *next; /* links for my hash key's list */
- struct PGLZ_HistEntry *prev;
- int hindex; /* my current hash key */
- const char *pos; /* my input position */
+ int16 next_id; /* links for my hash key's list */
+ uint16 hist_idx; /* my current hash key */
+ const unsigned char *pos; /* my input position */
} PGLZ_HistEntry;
@@ -274,117 +321,105 @@ static PGLZ_HistEntry hist_entries[PGLZ_HISTORY_SIZE + 1];
* hash keys more.
* ----------
*/
-#define pglz_hist_idx(_s,_e, _mask) ( \
- ((((_e) - (_s)) < 4) ? (int) (_s)[0] : \
- (((_s)[0] << 6) ^ ((_s)[1] << 4) ^ \
- ((_s)[2] << 2) ^ (_s)[3])) & (_mask) \
- )
+static inline uint16 pglz_hist_idx(const unsigned char* s, uint16 mask)
+{
+ return ((s[0] << 6) ^ (s[1] << 4) ^ (s[2] << 2) ^ s[3]) & mask;
+}
/* ----------
* pglz_hist_add -
*
- * Adds a new entry to the history table.
- *
- * If _recycle is true, then we are recycling a previously used entry,
- * and must first delink it from its old hashcode's linked list.
- *
- * NOTE: beware of multiple evaluations of macro's arguments, and note that
- * _hn and _recycle are modified in the macro.
+ * Adds a new entry to the history table
+ * and recalculates hash value.
* ----------
*/
-#define pglz_hist_add(_hs,_he,_hn,_recycle,_s,_e, _mask) \
-do { \
- int __hindex = pglz_hist_idx((_s),(_e), (_mask)); \
- int16 *__myhsp = &(_hs)[__hindex]; \
- PGLZ_HistEntry *__myhe = &(_he)[_hn]; \
- if (_recycle) { \
- if (__myhe->prev == NULL) \
- (_hs)[__myhe->hindex] = __myhe->next - (_he); \
- else \
- __myhe->prev->next = __myhe->next; \
- if (__myhe->next != NULL) \
- __myhe->next->prev = __myhe->prev; \
- } \
- __myhe->next = &(_he)[*__myhsp]; \
- __myhe->prev = NULL; \
- __myhe->hindex = __hindex; \
- __myhe->pos = (_s); \
- /* If there was an existing entry in this hash slot, link */ \
- /* this new entry to it. However, the 0th entry in the */ \
- /* entries table is unused, so we can freely scribble on it. */ \
- /* So don't bother checking if the slot was used - we'll */ \
- /* scribble on the unused entry if it was not, but that's */ \
- /* harmless. Avoiding the branch in this critical path */ \
- /* speeds this up a little bit. */ \
- /* if (*__myhsp != INVALID_ENTRY) */ \
- (_he)[(*__myhsp)].prev = __myhe; \
- *__myhsp = _hn; \
- if (++(_hn) >= PGLZ_HISTORY_SIZE + 1) { \
- (_hn) = 1; \
- (_recycle) = true; \
- } \
-} while (0)
+static inline int16
+pglz_hist_add(int16 hist_next, uint16 *hist_idx, const unsigned char* s, uint16 mask)
+{
+ int16* my_hist_start = &hist_start[*hist_idx];
+ PGLZ_HistEntry* entry = &(hist_entries)[hist_next];
+ /*
+ * Initialize entry with a new value.
+ */
+ entry->next_id = *my_hist_start;
+ entry->hist_idx = *hist_idx;
+ entry->pos = s;
-/* ----------
- * pglz_out_ctrl -
- *
- * Outputs the last and allocates a new control byte if needed.
- * ----------
- */
-#define pglz_out_ctrl(__ctrlp,__ctrlb,__ctrl,__buf) \
-do { \
- if ((__ctrl & 0xff) == 0) \
- { \
- *(__ctrlp) = __ctrlb; \
- __ctrlp = (__buf)++; \
- __ctrlb = 0; \
- __ctrl = 1; \
- } \
-} while (0)
+ /*
+ * Update linked list head pointer.
+ */
+ *my_hist_start = hist_next;
+
+ /*
+ * Recalculate hash value.
+ */
+ *hist_idx = ((((*hist_idx) ^ (s[0] << 6)) << 2) ^ s[4]) & mask;
+
+ /*
+ * Shift history pointer.
+ */
+ hist_next++;
+ if (hist_next == PGLZ_HISTORY_SIZE + 1)
+ {
+ hist_next = 1;
+ }
+ return hist_next;
+}
/* ----------
- * pglz_out_literal -
+ * pglz_out_tag -
*
- * Outputs a literal byte to the destination buffer including the
+ * Outputs a backward reference tag of 2-4 bytes (depending on
+ * offset and length) to the destination buffer including the
* appropriate control bit.
* ----------
*/
-#define pglz_out_literal(_ctrlp,_ctrlb,_ctrl,_buf,_byte) \
-do { \
- pglz_out_ctrl(_ctrlp,_ctrlb,_ctrl,_buf); \
- *(_buf)++ = (unsigned char)(_byte); \
- _ctrl <<= 1; \
-} while (0)
+static inline unsigned char*
+pglz_out_tag(unsigned char* dest_ptr, int32 match_len, int32 match_offset)
+{
+ if (match_len > 17)
+ {
+ *(dest_ptr++) = (unsigned char)((((match_offset) & 0xf00) >> 4) | 0x0f);
+ *(dest_ptr++) = (unsigned char)(((match_offset) & 0xff));
+ *(dest_ptr++) = (unsigned char)((match_len) - 18);
+ }
+ else
+ {
+ *(dest_ptr++) = (unsigned char)((((match_offset) & 0xf00) >> 4) | ((match_len) - 3));
+ *(dest_ptr++) = (unsigned char)((match_offset) & 0xff);
+ }
+ return dest_ptr;
+}
/* ----------
- * pglz_out_tag -
+ * pglz_compare -
*
- * Outputs a backward reference tag of 2-4 bytes (depending on
- * offset and length) to the destination buffer including the
- * appropriate control bit.
+ * Compares two sequences of bytes
+ * and returns positiion of first mismatch.
* ----------
*/
-#define pglz_out_tag(_ctrlp,_ctrlb,_ctrl,_buf,_len,_off) \
-do { \
- pglz_out_ctrl(_ctrlp,_ctrlb,_ctrl,_buf); \
- _ctrlb |= _ctrl; \
- _ctrl <<= 1; \
- if (_len > 17) \
- { \
- (_buf)[0] = (unsigned char)((((_off) & 0xf00) >> 4) | 0x0f); \
- (_buf)[1] = (unsigned char)(((_off) & 0xff)); \
- (_buf)[2] = (unsigned char)((_len) - 18); \
- (_buf) += 3; \
- } else { \
- (_buf)[0] = (unsigned char)((((_off) & 0xf00) >> 4) | ((_len) - 3)); \
- (_buf)[1] = (unsigned char)((_off) & 0xff); \
- (_buf) += 2; \
- } \
-} while (0)
+static inline int32
+pglz_compare(int32 len, int32 len_bound, const unsigned char* input_pos,
+ const unsigned char* hist_pos)
+{
+ while(len <= len_bound - 4 && pglz_read32(input_pos) == pglz_read32(hist_pos))
+ {
+ len += 4;
+ input_pos += 4;
+ hist_pos += 4;
+ }
+ while (len < len_bound && *input_pos == *hist_pos)
+ {
+ len++;
+ input_pos++;
+ hist_pos++;
+ }
+ return len;
+}
/* ----------
@@ -396,32 +431,40 @@ do { \
* ----------
*/
static inline int
-pglz_find_match(int16 *hstart, const char *input, const char *end,
- int *lenp, int *offp, int good_match, int good_drop, int mask)
+pglz_find_match(uint16 hist_idx, const unsigned char *input, const unsigned char *end,
+ int *len_ptr, int *offset_ptr, int good_match, int good_drop)
{
- PGLZ_HistEntry *hent;
- int16 hentno;
+ PGLZ_HistEntry *hist_entry;
+ int16 *hist_entry_number;
int32 len = 0;
- int32 off = 0;
+ int32 offset = 0;
+ int32 cur_len = 0;
+ int32 len_bound = Min(end - input, PGLZ_MAX_MATCH);
/*
* Traverse the linked history list until a good enough match is found.
*/
- hentno = hstart[pglz_hist_idx(input, end, mask)];
- hent = &hist_entries[hentno];
- while (hent != INVALID_ENTRY_PTR)
- {
- const char *ip = input;
- const char *hp = hent->pos;
- int32 thisoff;
- int32 thislen;
+ hist_entry_number = &hist_start[hist_idx];
+ if (*hist_entry_number == INVALID_ENTRY)
+ return 0;
+ hist_entry = &hist_entries[*hist_entry_number];
+ if (hist_idx != hist_entry->hist_idx)
+ {
/*
- * Stop if the offset does not fit into our tag anymore.
+ * If current linked list head points to invalid entry
+ * then clear it to reduce a number of comparisons in future.
*/
- thisoff = ip - hp;
- if (thisoff >= 0x0fff)
- break;
+ *hist_entry_number = INVALID_ENTRY;
+ return 0;
+ }
+
+ while (true)
+ {
+ const unsigned char *input_pos = input;
+ const unsigned char *hist_pos = hist_entry->pos;
+ const unsigned char *my_pos;
+ int32 cur_offset = input_pos - hist_pos;
/*
* Determine length of match. A better match must be larger than the
@@ -431,66 +474,65 @@ pglz_find_match(int16 *hstart, const char *input, const char *end,
* character by character comparison to know the exact position where
* the diff occurred.
*/
- thislen = 0;
if (len >= 16)
{
- if (memcmp(ip, hp, len) == 0)
+ if (memcmp(input_pos, hist_pos, len) == 0)
{
- thislen = len;
- ip += len;
- hp += len;
- while (ip < end && *ip == *hp && thislen < PGLZ_MAX_MATCH)
- {
- thislen++;
- ip++;
- hp++;
- }
+ offset = cur_offset;
+ len = pglz_compare(len, len_bound, input_pos + len, hist_pos + len);
}
}
else
{
- while (ip < end && *ip == *hp && thislen < PGLZ_MAX_MATCH)
+ if (pglz_read32(input_pos) == pglz_read32(hist_pos))
{
- thislen++;
- ip++;
- hp++;
+ cur_len = pglz_compare(4, len_bound, input_pos + 4, hist_pos + 4);
+ if (cur_len > len)
+ {
+ len = cur_len;
+ offset = cur_offset;
+ }
}
}
- /*
- * Remember this match as the best (if it is)
- */
- if (thislen > len)
- {
- len = thislen;
- off = thisoff;
- }
-
/*
* Advance to the next history entry
*/
- hent = hent->next;
+ my_pos = hist_entry->pos;
+ hist_entry = &hist_entries[hist_entry->next_id];
/*
- * Be happy with lesser good matches the more entries we visited. But
- * no point in doing calculation if we're at end of list.
+ * If current match length is ok then stop iteration.
+ * As outdated list links are not updated during insertion process
+ * then additional stop condition should be introduced to avoid following them.
+ * If recycled entry has another hash, then iteration stops.
+ * If it has the same hash then recycled cell would break input stream
+ * position monotonicity what is checked after.
*/
- if (hent != INVALID_ENTRY_PTR)
+ if (len >= good_match || hist_idx != hist_entry->hist_idx || my_pos <= hist_entry->pos)
{
- if (len >= good_match)
- break;
- good_match -= (good_match * good_drop) / 100;
+ break;
}
+ /*
+ * Be happy with lesser good matches the more entries we visited.
+ */
+ good_match -= (good_match * good_drop) >> 7;
}
+ /*
+ * As initial compare for short matches compares 4 bytes
+ * then for the end of stream length of match should be cut
+ */
+ len = Min(len, len_bound);
+
/*
* Return match information only if it results at least in one byte
* reduction.
*/
if (len > 2)
{
- *lenp = len;
- *offp = off;
+ *len_ptr = len;
+ *offset_ptr = offset;
return 1;
}
@@ -506,43 +548,47 @@ pglz_find_match(int16 *hstart, const char *input, const char *end,
* ----------
*/
int32
-pglz_compress(const char *source, int32 slen, char *dest,
+pglz_compress(const char *source, int32 src_len, char *dest,
const PGLZ_Strategy *strategy)
{
- unsigned char *bp = (unsigned char *) dest;
- unsigned char *bstart = bp;
- int hist_next = 1;
- bool hist_recycle = false;
- const char *dp = source;
- const char *dend = source + slen;
- unsigned char ctrl_dummy = 0;
- unsigned char *ctrlp = &ctrl_dummy;
- unsigned char ctrlb = 0;
- unsigned char ctrl = 0;
+ unsigned char *dest_ptr = (unsigned char *) dest;
+ unsigned char *dest_start = dest_ptr;
+ uint16 hist_next = 1;
+ uint16 hist_idx;
+ const unsigned char *src_ptr = (const unsigned char*)source;
+ const unsigned char *src_end = (const unsigned char*)source + src_len;
+ const unsigned char *compress_src_end = src_end - 4;
+ unsigned char control_dummy = 0;
+ unsigned char *control_ptr = &control_dummy;
+ unsigned char control_byte = 0;
+ unsigned char control_pos = 0;
bool found_match = false;
int32 match_len;
- int32 match_off;
+ int32 match_offset;
int32 good_match;
int32 good_drop;
int32 result_size;
int32 result_max;
int32 need_rate;
- int hashsz;
- int mask;
+ int hash_size;
+ uint16 mask;
+
/*
* Our fallback strategy is the default.
*/
if (strategy == NULL)
+ {
strategy = PGLZ_strategy_default;
+ }
/*
* If the strategy forbids compression (at all or if source chunk size out
* of range), fail.
*/
if (strategy->match_size_good <= 0 ||
- slen < strategy->min_input_size ||
- slen > strategy->max_input_size)
+ src_len < strategy->min_input_size ||
+ src_len > strategy->max_input_size)
return -1;
/*
@@ -559,6 +605,7 @@ pglz_compress(const char *source, int32 slen, char *dest,
good_drop = 0;
else if (good_drop > 100)
good_drop = 100;
+ good_drop = good_drop * 128 / 100;
need_rate = strategy->min_comp_rate;
if (need_rate < 0)
@@ -569,15 +616,17 @@ pglz_compress(const char *source, int32 slen, char *dest,
/*
* Compute the maximum result size allowed by the strategy, namely the
* input size minus the minimum wanted compression rate. This had better
- * be <= slen, else we might overrun the provided output buffer.
+ * be <= src_len, else we might overrun the provided output buffer.
*/
- if (slen > (INT_MAX / 100))
+ if (src_len > (INT_MAX / 100))
{
/* Approximate to avoid overflow */
- result_max = (slen / 100) * (100 - need_rate);
+ result_max = (src_len / 100) * (100 - need_rate);
}
else
- result_max = (slen * (100 - need_rate)) / 100;
+ {
+ result_max = (src_len * (100 - need_rate)) / 100;
+ }
/*
* Experiments suggest that these hash sizes work pretty well. A large
@@ -585,28 +634,39 @@ pglz_compress(const char *source, int32 slen, char *dest,
* small input, the startup cost dominates. The table size must be a power
* of two.
*/
- if (slen < 128)
- hashsz = 512;
- else if (slen < 256)
- hashsz = 1024;
- else if (slen < 512)
- hashsz = 2048;
- else if (slen < 1024)
- hashsz = 4096;
+ if (src_len < 128)
+ hash_size = 512;
+ else if (src_len < 256)
+ hash_size = 1024;
+ else if (src_len < 512)
+ hash_size = 2048;
+ else if (src_len < 1024)
+ hash_size = 4096;
else
- hashsz = 8192;
- mask = hashsz - 1;
+ hash_size = 8192;
+ mask = hash_size - 1;
/*
* Initialize the history lists to empty. We do not need to zero the
* hist_entries[] array; its entries are initialized as they are used.
*/
- memset(hist_start, 0, hashsz * sizeof(int16));
+ memset(hist_start, 0, hash_size * sizeof(int16));
+
+ /*
+ * Initialize INVALID_ENTRY for stopping during lookup.
+ */
+ hist_entries[INVALID_ENTRY].pos = src_end;
+ hist_entries[INVALID_ENTRY].hist_idx = hash_size;
+
+ /*
+ * Calculate initial hash value.
+ */
+ hist_idx = pglz_hist_idx(src_ptr, mask);
/*
* Compress the source directly into the output buffer.
*/
- while (dp < dend)
+ while (src_ptr < compress_src_end)
{
/*
* If we already exceeded the maximum result size, fail.
@@ -615,7 +675,7 @@ pglz_compress(const char *source, int32 slen, char *dest,
* bytes (a control byte and 3-byte tag), PGLZ_MAX_OUTPUT() had better
* allow 4 slop bytes.
*/
- if (bp - bstart >= result_max)
+ if (dest_ptr - dest_start >= result_max)
return -1;
/*
@@ -624,26 +684,35 @@ pglz_compress(const char *source, int32 slen, char *dest,
* reasonably quickly when looking at incompressible input (such as
* pre-compressed data).
*/
- if (!found_match && bp - bstart >= strategy->first_success_by)
+ if (!found_match && dest_ptr - dest_start >= strategy->first_success_by)
return -1;
+ /*
+ * Refresh control byte if needed.
+ */
+ if ((control_pos & 0xff) == 0)
+ {
+ *(control_ptr) = control_byte;
+ control_ptr = (dest_ptr)++;
+ control_byte = 0;
+ control_pos = 1;
+ }
/*
* Try to find a match in the history
*/
- if (pglz_find_match(hist_start, dp, dend, &match_len,
- &match_off, good_match, good_drop, mask))
+ if (pglz_find_match(hist_idx, src_ptr, compress_src_end, &match_len,
+ &match_offset, good_match, good_drop))
{
/*
* Create the tag and add history entries for all matched
* characters.
*/
- pglz_out_tag(ctrlp, ctrlb, ctrl, bp, match_len, match_off);
+ control_byte |= control_pos;
+ dest_ptr = pglz_out_tag(dest_ptr, match_len, match_offset);
while (match_len--)
{
- pglz_hist_add(hist_start, hist_entries,
- hist_next, hist_recycle,
- dp, dend, mask);
- dp++; /* Do not do this ++ in the line above! */
+ hist_next = pglz_hist_add(hist_next, &hist_idx, src_ptr, mask);
+ src_ptr++; /* Do not do this ++ in the line above! */
/* The macro would do it four times - Jan. */
}
found_match = true;
@@ -653,21 +722,49 @@ pglz_compress(const char *source, int32 slen, char *dest,
/*
* No match found. Copy one literal byte.
*/
- pglz_out_literal(ctrlp, ctrlb, ctrl, bp, *dp);
- pglz_hist_add(hist_start, hist_entries,
- hist_next, hist_recycle,
- dp, dend, mask);
- dp++; /* Do not do this ++ in the line above! */
+ hist_next = pglz_hist_add(hist_next, &hist_idx, src_ptr, mask);
+ *(dest_ptr)++ = (unsigned char)(*src_ptr);
+ src_ptr++; /* Do not do this ++ in the line above! */
/* The macro would do it four times - Jan. */
}
+ control_pos <<= 1;
+ }
+
+
+ while (src_ptr < src_end)
+ {
+ /*
+ * If we already exceeded the maximum result size, fail.
+ *
+ * We check once per loop; since the loop body could emit as many as 4
+ * bytes (a control byte and 3-byte tag), PGLZ_MAX_OUTPUT() had better
+ * allow 4 slop bytes.
+ */
+ if (dest_ptr - dest_start >= result_max)
+ return -1;
+
+ /*
+ * Refresh control byte if needed.
+ */
+ if ((control_pos & 0xff) == 0)
+ {
+ *(control_ptr) = control_byte;
+ control_ptr = (dest_ptr)++;
+ control_byte = 0;
+ control_pos = 1;
+ }
+ *(dest_ptr)++ = (unsigned char)(*src_ptr);
+ src_ptr++;
+ control_pos <<= 1;
}
+
/*
* Write out the last control byte and check that we haven't overrun the
* output size allowed by the strategy.
*/
- *ctrlp = ctrlb;
- result_size = bp - bstart;
+ *control_ptr = control_byte;
+ result_size = dest_ptr - dest_start;
if (result_size >= result_max)
return -1;
@@ -675,7 +772,6 @@ pglz_compress(const char *source, int32 slen, char *dest,
return result_size;
}
-
/* ----------
* pglz_decompress -
*
@@ -687,7 +783,7 @@ pglz_compress(const char *source, int32 slen, char *dest,
*/
int32
pglz_decompress(const char *source, int32 slen, char *dest,
- int32 rawsize, bool check_complete)
+ int32 rawsize, bool check_complete)
{
const unsigned char *sp;
const unsigned char *srcend;
@@ -759,9 +855,10 @@ pglz_decompress(const char *source, int32 slen, char *dest,
}
/*
- * Check we decompressed the right amount. If we are slicing, then we
- * won't necessarily be at the end of the source or dest buffers when we
- * hit a stop, so we don't test them.
+ * Check we decompressed the right amount.
+ * If we are slicing, then we won't necessarily
+ * be at the end of the source or dest buffers
+ * when we hit a stop, so we don't test them.
*/
if (check_complete && (dp != destend || sp != srcend))
return -1;
@@ -769,5 +866,5 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* That's it.
*/
- return (char *) dp - dest;
+ return (char*)dp - dest;
}
--
2.20.1
=
* Re: pglz performance
@ 2019-08-02 13:45 Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Konstantin Knizhnik @ 2019-08-02 13:45 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; Michael Paquier <michael@paquier.xyz>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On 27.06.2019 21:33, Andrey Borodin wrote:
>
>> 13 мая 2019 г., в 12:14, Michael Paquier <michael@paquier.xyz> написал(а):
>>
>> Decompression can matter a lot for mostly-read workloads and
>> compression can become a bottleneck for heavy-insert loads, so
>> improving compression or decompression should be two separate
>> problems, not two problems linked. Any improvement in one or the
>> other, or even both, is nice to have.
> Here's patch hacked by Vladimir for compression.
>
> Key differences (as far as I see, maybe Vladimir will post more complete list of optimizations):
> 1. Use functions instead of macro-functions: not surprisingly it's easier to optimize them and provide less constraints for compiler to optimize.
> 2. More compact hash table: use indexes instead of pointers.
> 3. More robust segment comparison: like memcmp, but return index of first different byte
>
> In weighted mix of different data (same as for compression), overall speedup is x1.43 on my machine.
>
> Current implementation is integrated into test_pglz suit for benchmarking purposes[0].
>
> Best regards, Andrey Borodin.
>
> [0] https://github.com/x4m/test_pglz
It takes me some time to understand that your memcpy optimization is
correct;)
I have tested different ways of optimizing this fragment of code, but
failed tooutperform your implementation!
Results at my computer is simlar with yours:
Decompressor score (summ of all times):
NOTICE: Decompressor pglz_decompress_hacked result 6.627355
NOTICE: Decompressor pglz_decompress_hacked_unrolled result 7.497114
NOTICE: Decompressor pglz_decompress_hacked8 result 7.412944
NOTICE: Decompressor pglz_decompress_hacked16 result 7.792978
NOTICE: Decompressor pglz_decompress_vanilla result 10.652603
Compressor score (summ of all times):
NOTICE: Compressor pglz_compress_vanilla result 116.970005
NOTICE: Compressor pglz_compress_hacked result 89.706105
But ... below are results for lz4:
Decompressor score (summ of all times):
NOTICE: Decompressor lz4_decompress result 3.660066
Compressor score (summ of all times):
NOTICE: Compressor lz4_compress result 10.288594
There is 2 times advantage in decompress speed and 10 times advantage in
compress speed.
So may be instead of "hacking" pglz algorithm we should better switch to
lz4?
--
Konstantin Knizhnik
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 14:43 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Tomas Vondra @ 2019-08-02 14:43 UTC (permalink / raw)
To: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Fri, Aug 02, 2019 at 04:45:43PM +0300, Konstantin Knizhnik wrote:
>
>
>On 27.06.2019 21:33, Andrey Borodin wrote:
>>
>>>13 мая 2019 г., в 12:14, Michael Paquier <michael@paquier.xyz> написал(а):
>>>
>>>Decompression can matter a lot for mostly-read workloads and
>>>compression can become a bottleneck for heavy-insert loads, so
>>>improving compression or decompression should be two separate
>>>problems, not two problems linked. Any improvement in one or the
>>>other, or even both, is nice to have.
>>Here's patch hacked by Vladimir for compression.
>>
>>Key differences (as far as I see, maybe Vladimir will post more complete list of optimizations):
>>1. Use functions instead of macro-functions: not surprisingly it's easier to optimize them and provide less constraints for compiler to optimize.
>>2. More compact hash table: use indexes instead of pointers.
>>3. More robust segment comparison: like memcmp, but return index of first different byte
>>
>>In weighted mix of different data (same as for compression), overall speedup is x1.43 on my machine.
>>
>>Current implementation is integrated into test_pglz suit for benchmarking purposes[0].
>>
>>Best regards, Andrey Borodin.
>>
>>[0] https://github.com/x4m/test_pglz
>
>It takes me some time to understand that your memcpy optimization is
>correct;)
>I have tested different ways of optimizing this fragment of code, but
>failed tooutperform your implementation!
>Results at my computer is simlar with yours:
>
>Decompressor score (summ of all times):
>NOTICE: Decompressor pglz_decompress_hacked result 6.627355
>NOTICE: Decompressor pglz_decompress_hacked_unrolled result 7.497114
>NOTICE: Decompressor pglz_decompress_hacked8 result 7.412944
>NOTICE: Decompressor pglz_decompress_hacked16 result 7.792978
>NOTICE: Decompressor pglz_decompress_vanilla result 10.652603
>
>Compressor score (summ of all times):
>NOTICE: Compressor pglz_compress_vanilla result 116.970005
>NOTICE: Compressor pglz_compress_hacked result 89.706105
>
>
>But ... below are results for lz4:
>
>Decompressor score (summ of all times):
>NOTICE: Decompressor lz4_decompress result 3.660066
>Compressor score (summ of all times):
>NOTICE: Compressor lz4_compress result 10.288594
>
>There is 2 times advantage in decompress speed and 10 times advantage
>in compress speed.
>So may be instead of "hacking" pglz algorithm we should better switch
>to lz4?
>
I think we should just bite the bullet and add initdb option to pick
compression algorithm. That's been discussed repeatedly, but we never
ended up actually doing that. See for example [1].
If there's anyone willing to put some effort into getting this feature
over the line, I'm willing to do reviews & commit. It's a seemingly
small change with rather insane potential impact.
But even if we end up doing that, it still makes sense to optimize the
hell out of pglz, because existing systems will still use that
(pg_upgrade can't switch from one compression algorithm to another).
regards
[1] https://www.postgresql.org/message-id/flat/55341569.1090107%402ndquadrant.com
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 15:40 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-08-02 15:40 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Thanks for looking into this!
> 2 авг. 2019 г., в 19:43, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>
> On Fri, Aug 02, 2019 at 04:45:43PM +0300, Konstantin Knizhnik wrote:
>>
>> It takes me some time to understand that your memcpy optimization is correct;)
Seems that comments are not explanatory enough... will try to fix.
>> I have tested different ways of optimizing this fragment of code, but failed tooutperform your implementation!
JFYI we tried optimizations with memcpy with const size (optimized into assembly instead of call), unrolling literal loop and some others. All these did not work better.
>> But ... below are results for lz4:
>>
>> Decompressor score (summ of all times):
>> NOTICE: Decompressor lz4_decompress result 3.660066
>> Compressor score (summ of all times):
>> NOTICE: Compressor lz4_compress result 10.288594
>>
>> There is 2 times advantage in decompress speed and 10 times advantage in compress speed.
>> So may be instead of "hacking" pglz algorithm we should better switch to lz4?
>>
>
> I think we should just bite the bullet and add initdb option to pick
> compression algorithm. That's been discussed repeatedly, but we never
> ended up actually doing that. See for example [1].
>
> If there's anyone willing to put some effort into getting this feature
> over the line, I'm willing to do reviews & commit. It's a seemingly
> small change with rather insane potential impact.
>
> But even if we end up doing that, it still makes sense to optimize the
> hell out of pglz, because existing systems will still use that
> (pg_upgrade can't switch from one compression algorithm to another).
We have some kind of "roadmap" of "extensible pglz". We plan to provide implementation on Novembers CF.
Currently, pglz starts with empty cache map: there is no prior 4k bytes before start. We can add imaginary prefix to any data with common substrings: this will enhance compression ratio.
It is hard to decide on training data set for this "common prefix". So we want to produce extension with aggregate function which produces some "adapted common prefix" from users's data.
Then we can "reserve" few negative bytes for "decompression commands". This command can instruct database on which common prefix to use.
But also system command can say "invoke decompression from extension".
Thus, user will be able to train database compression on his data and substitute pglz compression with custom compression method seamlessly.
This will make hard-choosen compression unneeded, but seems overly hacky. But there will be no need to have lz4, zstd, brotli, lzma and others in core. Why not provide e.g. "time series compression"? Or "DNA compression"? Whatever gun user wants for his foot.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 16:39 Andres Freund <andres@anarazel.de>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2019-08-02 16:39 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
> We have some kind of "roadmap" of "extensible pglz". We plan to provide implementation on Novembers CF.
I don't understand why it's a good idea to improve the compression side
of pglz. There's plenty other people that spent a lot of time developing
better compression algorithms.
> Currently, pglz starts with empty cache map: there is no prior 4k bytes before start. We can add imaginary prefix to any data with common substrings: this will enhance compression ratio.
> It is hard to decide on training data set for this "common prefix". So we want to produce extension with aggregate function which produces some "adapted common prefix" from users's data.
> Then we can "reserve" few negative bytes for "decompression commands". This command can instruct database on which common prefix to use.
> But also system command can say "invoke decompression from extension".
>
> Thus, user will be able to train database compression on his data and substitute pglz compression with custom compression method seamlessly.
>
> This will make hard-choosen compression unneeded, but seems overly hacky. But there will be no need to have lz4, zstd, brotli, lzma and others in core. Why not provide e.g. "time series compression"? Or "DNA compression"? Whatever gun user wants for his foot.
I think this is way too complicated, and will provide not particularly
much benefit for the majority users.
In fact, I'll argue that we should flat out reject any such patch until
we have at least one decent default compression algorithm in
core. You're trying to work around a poor compression algorithm with
complicated dictionary improvement, that require user interaction, and
only will work in a relatively small subset of the cases, and will very
often increase compression times.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 17:00 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Tomas Vondra @ 2019-08-02 17:00 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote:
>Hi,
>
>On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
>> We have some kind of "roadmap" of "extensible pglz". We plan to
>> provide implementation on Novembers CF.
>
>I don't understand why it's a good idea to improve the compression side
>of pglz. There's plenty other people that spent a lot of time
>developing better compression algorithms.
>
Isn't it beneficial for existing systems, that will be stuck with pglz
even if we end up adding other algorithms?
>
>> Currently, pglz starts with empty cache map: there is no prior 4k
>> bytes before start. We can add imaginary prefix to any data with
>> common substrings: this will enhance compression ratio. It is hard
>> to decide on training data set for this "common prefix". So we want
>> to produce extension with aggregate function which produces some
>> "adapted common prefix" from users's data. Then we can "reserve" few
>> negative bytes for "decompression commands". This command can
>> instruct database on which common prefix to use. But also system
>> command can say "invoke decompression from extension".
>>
>> Thus, user will be able to train database compression on his data and
>> substitute pglz compression with custom compression method
>> seamlessly.
>>
>> This will make hard-choosen compression unneeded, but seems overly
>> hacky. But there will be no need to have lz4, zstd, brotli, lzma and
>> others in core. Why not provide e.g. "time series compression"? Or
>> "DNA compression"? Whatever gun user wants for his foot.
>
>I think this is way too complicated, and will provide not particularly
>much benefit for the majority users.
>
I agree with this. I do see value in the feature, but probably not as a
drop-in replacement for the default compression algorithm. I'd compare
it to the "custom compression methods" patch that was submitted some
time ago.
>In fact, I'll argue that we should flat out reject any such patch until
>we have at least one decent default compression algorithm in core.
>You're trying to work around a poor compression algorithm with
>complicated dictionary improvement, that require user interaction, and
>only will work in a relatively small subset of the cases, and will very
>often increase compression times.
>
I wouldn't be so strict I guess. But I do agree an algorithm that
requires additional steps (training, ...) is unlikely to be a good
candidate for default instance compression alorithm.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 17:12 Andres Freund <andres@anarazel.de>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2019-08-02 17:12 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 2019-08-02 19:00:39 +0200, Tomas Vondra wrote:
> On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote:
> > Hi,
> >
> > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
> > > We have some kind of "roadmap" of "extensible pglz". We plan to
> > > provide implementation on Novembers CF.
> >
> > I don't understand why it's a good idea to improve the compression side
> > of pglz. There's plenty other people that spent a lot of time
> > developing better compression algorithms.
> >
>
> Isn't it beneficial for existing systems, that will be stuck with pglz
> even if we end up adding other algorithms?
Why would they be stuck continuing to *compress* with pglz? As we
fully retoast on write anyway we can just gradually switch over to the
better algorithm. Decompression speed is another story, of course.
Here's what I had a few years back:
https://www.postgresql.org/message-id/20130621000900.GA12425%40alap2.anarazel.de
see also
https://www.postgresql.org/message-id/20130605150144.GD28067%40alap2.anarazel.de
I think we should refresh something like that patch, and:
- make the compression algorithm GUC an enum, rename
- add --with-system-lz4
- obviously refresh the copy of lz4
- drop snappy
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 17:52 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 2 replies; 70+ messages in thread
From: Tomas Vondra @ 2019-08-02 17:52 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote:
>Hi,
>
>On 2019-08-02 19:00:39 +0200, Tomas Vondra wrote:
>> On Fri, Aug 02, 2019 at 09:39:48AM -0700, Andres Freund wrote:
>> > Hi,
>> >
>> > On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
>> > > We have some kind of "roadmap" of "extensible pglz". We plan to
>> > > provide implementation on Novembers CF.
>> >
>> > I don't understand why it's a good idea to improve the compression side
>> > of pglz. There's plenty other people that spent a lot of time
>> > developing better compression algorithms.
>> >
>>
>> Isn't it beneficial for existing systems, that will be stuck with pglz
>> even if we end up adding other algorithms?
>
>Why would they be stuck continuing to *compress* with pglz? As we
>fully retoast on write anyway we can just gradually switch over to the
>better algorithm. Decompression speed is another story, of course.
>
Hmmm, I don't remember the details of those patches so I didn't realize
it allows incremental recompression. If that's possible, that would mean
existing systems can start using it. Which is good.
Another question is whether we'd actually want to include the code in
core directly, or use system libraries (and if some packagers might
decide to disable that, for whatever reason).
But yeah, I agree you may have a point about optimizing pglz compression.
>
>Here's what I had a few years back:
>
>https://www.postgresql.org/message-id/20130621000900.GA12425%40alap2.anarazel.de
>see also
>https://www.postgresql.org/message-id/20130605150144.GD28067%40alap2.anarazel.de
>
>I think we should refresh something like that patch, and:
>- make the compression algorithm GUC an enum, rename
>- add --with-system-lz4
>- obviously refresh the copy of lz4
>- drop snappy
>
That's a reasonable plan, I guess.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 18:20 Andres Freund <andres@anarazel.de>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 2 replies; 70+ messages in thread
From: Andres Freund @ 2019-08-02 18:20 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 2019-08-02 19:52:39 +0200, Tomas Vondra wrote:
> Hmmm, I don't remember the details of those patches so I didn't realize
> it allows incremental recompression. If that's possible, that would mean
> existing systems can start using it. Which is good.
That depends on what do you mean by "incremental"? A single toasted
datum can only have one compression type, because we only update them
all in one anyway. But different datums can be compressed differently.
> Another question is whether we'd actually want to include the code in
> core directly, or use system libraries (and if some packagers might
> decide to disable that, for whatever reason).
I'd personally say we should have an included version, and a
--with-system-... flag that uses the system one.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-02 19:48 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 1 reply; 70+ messages in thread
From: Tomas Vondra @ 2019-08-02 19:48 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote:
>Hi,
>
>On 2019-08-02 19:52:39 +0200, Tomas Vondra wrote:
>> Hmmm, I don't remember the details of those patches so I didn't realize
>> it allows incremental recompression. If that's possible, that would mean
>> existing systems can start using it. Which is good.
>
>That depends on what do you mean by "incremental"? A single toasted
>datum can only have one compression type, because we only update them
>all in one anyway. But different datums can be compressed differently.
>
I meant different toast values using different compression algorithm,
sorry for the confusion.
>
>> Another question is whether we'd actually want to include the code in
>> core directly, or use system libraries (and if some packagers might
>> decide to disable that, for whatever reason).
>
>I'd personally say we should have an included version, and a
>--with-system-... flag that uses the system one.
>
OK. I'd say to require a system library, but that's a minor detail.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-03 05:37 Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
parent: Andres Freund <andres@anarazel.de>
1 sibling, 0 replies; 70+ messages in thread
From: Konstantin Knizhnik @ 2019-08-03 05:37 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On 02.08.2019 21:20, Andres Freund wrote:
> Another question is whether we'd actually want to include the code in
>> core directly, or use system libraries (and if some packagers might
>> decide to disable that, for whatever reason).
> I'd personally say we should have an included version, and a
> --with-system-... flag that uses the system one.
+1
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 00:41 Petr Jelinek <petr@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 4 replies; 70+ messages in thread
From: Petr Jelinek @ 2019-08-04 00:41 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 02/08/2019 21:48, Tomas Vondra wrote:
> On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote:
>
>>
>>> Another question is whether we'd actually want to include the code in
>>> core directly, or use system libraries (and if some packagers might
>>> decide to disable that, for whatever reason).
>>
>> I'd personally say we should have an included version, and a
>> --with-system-... flag that uses the system one.
>>
>
> OK. I'd say to require a system library, but that's a minor detail.
>
Same here.
Just so that we don't idly talk, what do you think about the attached?
It:
- adds new GUC compression_algorithm with possible values of pglz
(default) and lz4 (if lz4 is compiled in), requires SIGHUP
- adds --with-lz4 configure option (default yes, so the configure option
is actually --without-lz4) that enables the lz4, it's using system library
- uses the compression_algorithm for both TOAST and WAL compression (if on)
- supports slicing for lz4 as well (pglz was already supported)
- supports reading old TOAST values
- adds 1 byte header to the compressed data where we currently store the
algorithm kind, that leaves us with 254 more to add :) (that's an extra
overhead compared to the current state)
- changes the rawsize in TOAST header to 31 bits via bit packing
- uses the extra bit to differentiate between old and new format
- supports reading from table which has different rows stored with
different algorithm (so that the GUC itself can be freely changed)
Simple docs and a TAP test included.
I did some basic performance testing (it's not really my thing though,
so I would appreciate if somebody did more).
I get about 7x perf improvement on data load with lz4 compared to pglz
on my dataset but strangely only tiny decompression improvement. Perhaps
more importantly I also did before patch and after patch tests with pglz
and the performance difference with my data set was <1%.
Note that this will just link against lz4, it does not add lz4 into
PostgreSQL code-base.
The issues I know of:
- the pg_decompress function really ought to throw error in the default
branch but that file is also used in front-end so not sure how to do that
- the TAP test probably does not work with all possible configurations
(but that's why it needs to be set in PG_TEST_EXTRA like for example ssl)
- we don't really have any automated test for reading old TOAST format,
no idea how to do that
- I expect my changes to configure.in are not the greatest as I don't
have pretty much zero experience with autoconf
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
Attachments:
0001-Add-new-GUC-compression_algorithm.patch (text/x-patch, 34.7K) download
From 862919a271e6bfa151fde2fed19c9e5ae6227cdc Mon Sep 17 00:00:00 2001
From: Petr Jelinek <pjmodos@pjmodos.net>
Date: Sun, 4 Aug 2019 02:02:30 +0200
Subject: [PATCH] Add new GUC compression_algorithm
Sets which algorithm to use for TOAST and WAL compression.
Currently allows either pglz which is the standard PostgreSQL algorithm
or lz4 if the PostgreSQL was configured with --with-lz4 (which is the
default).
The implementation allows different values to have different compression
algorithms and also supports reading old TOAST format which always uses
the pglz.
---
configure | 116 ++++++++++++++++++--
configure.in | 19 ++++
doc/src/sgml/config.sgml | 38 +++++++
doc/src/sgml/storage.sgml | 5 +-
src/Makefile.global.in | 1 +
src/backend/access/heap/tuptoaster.c | 86 +++++++++++----
src/backend/access/transam/xloginsert.c | 5 +-
src/backend/utils/misc/guc.c | 23 +++-
src/common/pg_lzcompress.c | 134 +++++++++++++++++++++++-
src/include/common/pg_lzcompress.h | 49 +++++++--
src/include/pg_config.h.in | 3 +
src/include/postgres.h | 3 +-
src/test/Makefile | 8 +-
src/test/toast/.gitignore | 2 +
src/test/toast/Makefile | 25 +++++
src/test/toast/README | 25 +++++
src/test/toast/t/001_lz4.pl | 124 ++++++++++++++++++++++
17 files changed, 619 insertions(+), 47 deletions(-)
create mode 100644 src/test/toast/.gitignore
create mode 100644 src/test/toast/Makefile
create mode 100644 src/test/toast/README
create mode 100644 src/test/toast/t/001_lz4.pl
diff --git a/configure b/configure
index 7a6bfc2339..edd2bfefd6 100755
--- a/configure
+++ b/configure
@@ -704,6 +704,7 @@ with_system_tzdata
with_libxslt
with_libxml
XML2_CONFIG
+with_lz4
UUID_EXTRA_OBJS
with_uuid
with_systemd
@@ -795,6 +796,7 @@ infodir
docdir
oldincludedir
includedir
+runstatedir
localstatedir
sharedstatedir
sysconfdir
@@ -859,6 +861,7 @@ with_readline
with_libedit_preferred
with_uuid
with_ossp_uuid
+with_lz4
with_libxml
with_libxslt
with_system_tzdata
@@ -932,6 +935,7 @@ datadir='${datarootdir}'
sysconfdir='${prefix}/etc'
sharedstatedir='${prefix}/com'
localstatedir='${prefix}/var'
+runstatedir='${localstatedir}/run'
includedir='${prefix}/include'
oldincludedir='/usr/include'
docdir='${datarootdir}/doc/${PACKAGE_TARNAME}'
@@ -1184,6 +1188,15 @@ do
| -silent | --silent | --silen | --sile | --sil)
silent=yes ;;
+ -runstatedir | --runstatedir | --runstatedi | --runstated \
+ | --runstate | --runstat | --runsta | --runst | --runs \
+ | --run | --ru | --r)
+ ac_prev=runstatedir ;;
+ -runstatedir=* | --runstatedir=* | --runstatedi=* | --runstated=* \
+ | --runstate=* | --runstat=* | --runsta=* | --runst=* | --runs=* \
+ | --run=* | --ru=* | --r=*)
+ runstatedir=$ac_optarg ;;
+
-sbindir | --sbindir | --sbindi | --sbind | --sbin | --sbi | --sb)
ac_prev=sbindir ;;
-sbindir=* | --sbindir=* | --sbindi=* | --sbind=* | --sbin=* \
@@ -1321,7 +1334,7 @@ fi
for ac_var in exec_prefix prefix bindir sbindir libexecdir datarootdir \
datadir sysconfdir sharedstatedir localstatedir includedir \
oldincludedir docdir infodir htmldir dvidir pdfdir psdir \
- libdir localedir mandir
+ libdir localedir mandir runstatedir
do
eval ac_val=\$$ac_var
# Remove trailing slashes.
@@ -1474,6 +1487,7 @@ Fine tuning of the installation directories:
--sysconfdir=DIR read-only single-machine data [PREFIX/etc]
--sharedstatedir=DIR modifiable architecture-independent data [PREFIX/com]
--localstatedir=DIR modifiable single-machine data [PREFIX/var]
+ --runstatedir=DIR modifiable per-process data [LOCALSTATEDIR/run]
--libdir=DIR object code libraries [EPREFIX/lib]
--includedir=DIR C header files [PREFIX/include]
--oldincludedir=DIR C header files for non-gcc [/usr/include]
@@ -1564,6 +1578,7 @@ Optional Packages:
prefer BSD Libedit over GNU Readline
--with-uuid=LIB build contrib/uuid-ossp using LIB (bsd,e2fs,ossp)
--with-ossp-uuid obsolete spelling of --with-uuid=ossp
+ --without-lz4 do not build with LZ4 support
--with-libxml build with XML support
--with-libxslt use XSLT support when building contrib/xml2
--with-system-tzdata=DIR
@@ -8115,6 +8130,34 @@ fi
+#
+# LZ4
+#
+
+
+
+# Check whether --with-lz4 was given.
+if test "${with_lz4+set}" = set; then :
+ withval=$with_lz4;
+ case $withval in
+ yes)
+ :
+ ;;
+ no)
+ :
+ ;;
+ *)
+ as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5
+ ;;
+ esac
+
+else
+ with_lz4=yes
+
+fi
+
+
+
#
# XML
@@ -12661,6 +12704,55 @@ fi
fi
+# for lz4 compression support
+if test "$with_lz4" = yes ; then
+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_sizeofState in -llz4" >&5
+$as_echo_n "checking for LZ4_sizeofState in -llz4... " >&6; }
+if ${ac_cv_lib_lz4_LZ4_sizeofState+:} false; then :
+ $as_echo_n "(cached) " >&6
+else
+ ac_check_lib_save_LIBS=$LIBS
+LIBS="-llz4 $LIBS"
+cat confdefs.h - <<_ACEOF >conftest.$ac_ext
+/* end confdefs.h. */
+
+/* Override any GCC internal prototype to avoid an error.
+ Use char because int might match the return type of a GCC
+ builtin and then its argument prototype would still apply. */
+#ifdef __cplusplus
+extern "C"
+#endif
+char LZ4_sizeofState ();
+int
+main ()
+{
+return LZ4_sizeofState ();
+ ;
+ return 0;
+}
+_ACEOF
+if ac_fn_c_try_link "$LINENO"; then :
+ ac_cv_lib_lz4_LZ4_sizeofState=yes
+else
+ ac_cv_lib_lz4_LZ4_sizeofState=no
+fi
+rm -f core conftest.err conftest.$ac_objext \
+ conftest$ac_exeext conftest.$ac_ext
+LIBS=$ac_check_lib_save_LIBS
+fi
+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_sizeofState" >&5
+$as_echo "$ac_cv_lib_lz4_LZ4_sizeofState" >&6; }
+if test "x$ac_cv_lib_lz4_LZ4_sizeofState" = xyes; then :
+ COMPRESSION_LIBS=" -llz4"
+else
+ as_fn_error $? "library 'lz4' is required for LZ4 compression support" "$LINENO" 5
+fi
+
+
+$as_echo "#define HAVE_LZ4 1" >>confdefs.h
+
+fi
+LIBS="$LIBS$COMPRESSION_LIBS"
##
## Header files
@@ -13340,6 +13432,18 @@ fi
done
+fi
+
+# for lz4 compression support
+if test "$with_lz4" = yes ; then
+ ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default"
+if test "x$ac_cv_header_lz4_h" = xyes; then :
+
+else
+ as_fn_error $? "header file <lz4.h> is required for LZ4 support" "$LINENO" 5
+fi
+
+
fi
if test "$PORTNAME" = "win32" ; then
@@ -14683,7 +14787,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14729,7 +14833,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14753,7 +14857,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14798,7 +14902,7 @@ else
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
@@ -14822,7 +14926,7 @@ rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
int off_t_is_large[(LARGE_OFF_T % 2147483629 == 721
&& LARGE_OFF_T % 2147483647 == 1)
? 1 : -1];
diff --git a/configure.in b/configure.in
index dde3eec89f..393a820cc2 100644
--- a/configure.in
+++ b/configure.in
@@ -915,6 +915,12 @@ fi
AC_SUBST(with_uuid)
AC_SUBST(UUID_EXTRA_OBJS)
+#
+# LZ4
+#
+PGAC_ARG_BOOL(with, lz4, yes,
+ [do not build with LZ4 support])
+AC_SUBST(with_lz4)
#
# XML
@@ -1263,6 +1269,14 @@ elif test "$with_uuid" = ossp ; then
fi
AC_SUBST(UUID_LIBS)
+# for lz4 compression support
+if test "$with_lz4" = yes ; then
+ AC_CHECK_LIB(lz4, LZ4_sizeofState,
+ [COMPRESSION_LIBS=" -llz4"],
+ [AC_MSG_ERROR([library 'lz4' is required for LZ4 compression support])])
+ AC_DEFINE([HAVE_LZ4], 1, [Define to 1 to build with LZ4 support])
+fi
+LIBS="$LIBS$COMPRESSION_LIBS"
##
## Header files
@@ -1443,6 +1457,11 @@ elif test "$with_uuid" = ossp ; then
[AC_MSG_ERROR([header file <ossp/uuid.h> or <uuid.h> is required for OSSP UUID])])])
fi
+# for lz4 compression support
+if test "$with_lz4" = yes ; then
+ AC_CHECK_HEADER(lz4.h, [], [AC_MSG_ERROR([header file <lz4.h> is required for LZ4 support])])
+fi
+
if test "$PORTNAME" = "win32" ; then
AC_CHECK_HEADERS(crtdefs.h)
fi
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index c91e3e1550..81bd26b653 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1793,6 +1793,42 @@ include_dir 'conf.d'
<title>Disk</title>
<variablelist>
+
+ <varlistentry id="guc-compression-algorithm" xreflabel="compression_algorithm">
+ <term><varname>compression_algorithm</varname> (<type>enum</type>)
+ <indexterm>
+ <primary><varname>compression_algorithm</varname> configuration parameter</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Which compression algorithm to use for compressing
+ <acronym>TOAST</acronym> data and when
+ <xref linkend="guc-wal-compression"/> is turned on also for
+ <acronym>WAL</acronym>.
+ Possible values are:
+ </para>
+ <itemizedlist>
+ <listitem>
+ <para>
+ <literal>pglz</literal> (the internal PostgreSQL LZ family compression algorithm)
+ </para>
+ </listitem>
+ <listitem>
+ <para>
+ <literal>lz4</literal> (the LZ4 compression algorithm)
+ </para>
+ </listitem>
+ </itemizedlist>
+ <para>
+ Not all of these choices are available on all platforms.
+ The default is the pglz algorithm which is the one used by PostgrSQL
+ version 12 and earlier.
+ Only superusers can change this setting.
+ </para>
+ </listitem>
+ </varlistentry>
+
<varlistentry id="guc-temp-file-limit" xreflabel="temp_file_limit">
<term><varname>temp_file_limit</varname> (<type>integer</type>)
<indexterm>
@@ -2728,6 +2764,8 @@ include_dir 'conf.d'
<xref linkend="guc-full-page-writes"/> is on or during a base backup.
A compressed page image will be decompressed during WAL replay.
The default value is <literal>off</literal>.
+ The compression used is the one specified by
+ <xref linkend="guc-compression-algorithm"/>
Only superusers can change this setting.
</para>
diff --git a/doc/src/sgml/storage.sgml b/doc/src/sgml/storage.sgml
index 1047c77a63..306a0bb3d3 100644
--- a/doc/src/sgml/storage.sgml
+++ b/doc/src/sgml/storage.sgml
@@ -394,9 +394,8 @@ Further details appear in <xref linkend="storage-toast-inmemory"/>.
<para>
The compression technique used for either in-line or out-of-line compressed
-data is a fairly simple and very fast member
-of the LZ family of compression techniques. See
-<filename>src/common/pg_lzcompress.c</filename> for the details.
+data is chosen based on the <xref linkend="guc-compression-algorithm"/>
+setting.
</para>
<sect2 id="storage-toast-ondisk">
diff --git a/src/Makefile.global.in b/src/Makefile.global.in
index dc3f207e1c..8c3d38db1c 100644
--- a/src/Makefile.global.in
+++ b/src/Makefile.global.in
@@ -195,6 +195,7 @@ with_libxslt = @with_libxslt@
with_llvm = @with_llvm@
with_system_tzdata = @with_system_tzdata@
with_uuid = @with_uuid@
+with_lz4 = @with_lz4@
with_zlib = @with_zlib@
enable_rpath = @enable_rpath@
enable_nls = @enable_nls@
diff --git a/src/backend/access/heap/tuptoaster.c b/src/backend/access/heap/tuptoaster.c
index 74233bb931..14b64936ab 100644
--- a/src/backend/access/heap/tuptoaster.c
+++ b/src/backend/access/heap/tuptoaster.c
@@ -52,7 +52,20 @@
typedef struct toast_compress_header
{
int32 vl_len_; /* varlena header (do not touch directly!) */
- int32 rawsize;
+ /*
+ * The length cannot be more than 1GB due to general toast limitations
+ * we have the 2 high bits to encode aditional information.
+ *
+ * We use the last (highest) bit to mark this toast as the "new
+ * compression format" as the new pg_compress has it's own header
+ * which and the original pglz format is not distiguishable in any
+ * way from the format used by pg_compress. Thanks to this information
+ * the toast_decompress_datum can pick to either directly use
+ * pglz_decompress directly when dealing with data writen by older
+ * versions of postgres or let pg_decompress to autodetect format.
+ */
+ int32 rawsize:31;
+ uint32 cformat:1;
} toast_compress_header;
/*
@@ -61,10 +74,13 @@ typedef struct toast_compress_header
*/
#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header))
#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize)
+#define TOAST_COMPRESS_CFORMAT(ptr) (((toast_compress_header *) (ptr))->cformat)
#define TOAST_COMPRESS_RAWDATA(ptr) \
(((char *) (ptr)) + TOAST_COMPRESS_HDRSZ)
#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \
(((toast_compress_header *) (ptr))->rawsize = (len))
+#define TOAST_COMPRESS_SET_CFORMAT(ptr, fmt) \
+ (((toast_compress_header *) (ptr))->cformat = (fmt))
static void toast_delete_datum(Relation rel, Datum value, bool is_speculative);
static Datum toast_save_datum(Relation rel, Datum value,
@@ -385,7 +401,7 @@ toast_raw_datum_size(Datum value)
else if (VARATT_IS_COMPRESSED(attr))
{
/* here, va_rawsize is just the payload size */
- result = VARRAWSIZE_4B_C(attr) + VARHDRSZ;
+ result = TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ;
}
else if (VARATT_IS_SHORT(attr))
{
@@ -1363,6 +1379,7 @@ toast_compress_datum(Datum value)
{
struct varlena *tmp;
int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value));
+ int32 buffer_capacity;
int32 len;
Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value)));
@@ -1376,11 +1393,11 @@ toast_compress_datum(Datum value)
valsize > PGLZ_strategy_default->max_input_size)
return PointerGetDatum(NULL);
- tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) +
- TOAST_COMPRESS_HDRSZ);
+ buffer_capacity = pg_compress_bound(valsize);
+ tmp = (struct varlena *) palloc(buffer_capacity + TOAST_COMPRESS_HDRSZ);
/*
- * We recheck the actual size even if pglz_compress() reports success,
+ * We recheck the actual size even if pg_compress() reports success,
* because it might be satisfied with having saved as little as one byte
* in the compressed data --- which could turn into a net loss once you
* consider header and alignment padding. Worst case, the compressed
@@ -1389,14 +1406,17 @@ toast_compress_datum(Datum value)
* only one header byte and no padding if the value is short enough. So
* we insist on a savings of more than 2 bytes to ensure we have a gain.
*/
- len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)),
- valsize,
- TOAST_COMPRESS_RAWDATA(tmp),
- PGLZ_strategy_default);
+ len = pg_compress(VARDATA_ANY(DatumGetPointer(value)),
+ valsize,
+ TOAST_COMPRESS_RAWDATA(tmp),
+ buffer_capacity,
+ PGLZ_strategy_default);
+
if (len >= 0 &&
len + TOAST_COMPRESS_HDRSZ < valsize - 2)
{
TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize);
+ TOAST_COMPRESS_SET_CFORMAT(tmp, 1);
SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ);
/* successful compression */
return PointerGetDatum(tmp);
@@ -1520,7 +1540,7 @@ toast_save_datum(Relation rel, Datum value,
data_p = VARDATA(dval);
data_todo = VARSIZE(dval) - VARHDRSZ;
/* rawsize in a compressed datum is just the size of the payload */
- toast_pointer.va_rawsize = VARRAWSIZE_4B_C(dval) + VARHDRSZ;
+ toast_pointer.va_rawsize = TOAST_COMPRESS_RAWSIZE(dval) + VARHDRSZ;
toast_pointer.va_extsize = data_todo;
/* Assert that the numbers look like it's compressed */
Assert(VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer));
@@ -2277,18 +2297,40 @@ static struct varlena *
toast_decompress_datum(struct varlena *attr)
{
struct varlena *result;
+ uint32 compression_format;
+ int32 raw_size;
Assert(VARATT_IS_COMPRESSED(attr));
- result = (struct varlena *)
- palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ);
- SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ);
+ raw_size = TOAST_COMPRESS_RAWSIZE(attr);
- if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr),
- VARSIZE(attr) - TOAST_COMPRESS_HDRSZ,
- VARDATA(result),
- TOAST_COMPRESS_RAWSIZE(attr), true) < 0)
- elog(ERROR, "compressed data is corrupted");
+ compression_format = TOAST_COMPRESS_CFORMAT(attr);
+
+ result = (struct varlena *) palloc(raw_size + VARHDRSZ);
+ SET_VARSIZE(result, raw_size + VARHDRSZ);
+
+ /*
+ * Support for legacy compressed TOAST format which is always using pglz.
+ */
+ switch (compression_format)
+ {
+ case 0:
+ if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr),
+ VARSIZE(attr) - TOAST_COMPRESS_HDRSZ,
+ VARDATA(result),
+ raw_size, true) < 0)
+ elog(ERROR, "compressed data is corrupted");
+ break;
+ case 1:
+ if (pg_decompress(TOAST_COMPRESS_RAWDATA(attr),
+ VARSIZE(attr) - TOAST_COMPRESS_HDRSZ,
+ VARDATA(result),
+ raw_size, true) < 0)
+ elog(ERROR, "compressed data is corrupted");
+ break;
+ default:
+ pg_unreachable();
+ }
return result;
}
@@ -2311,10 +2353,10 @@ toast_decompress_datum_slice(struct varlena *attr, int32 slicelength)
result = (struct varlena *) palloc(slicelength + VARHDRSZ);
- rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr),
- VARSIZE(attr) - TOAST_COMPRESS_HDRSZ,
- VARDATA(result),
- slicelength, false);
+ rawsize = pg_decompress(TOAST_COMPRESS_RAWDATA(attr),
+ VARSIZE(attr) - TOAST_COMPRESS_HDRSZ,
+ VARDATA(result),
+ slicelength, false);
if (rawsize < 0)
elog(ERROR, "compressed data is corrupted");
diff --git a/src/backend/access/transam/xloginsert.c b/src/backend/access/transam/xloginsert.c
index 3ec67d468b..76decbf426 100644
--- a/src/backend/access/transam/xloginsert.c
+++ b/src/backend/access/transam/xloginsert.c
@@ -830,11 +830,12 @@ XLogCompressBackupBlock(char *page, uint16 hole_offset, uint16 hole_length,
source = page;
/*
- * We recheck the actual size even if pglz_compress() reports success and
+ * We recheck the actual size even if pg_compress() reports success and
* see if the number of bytes saved by compression is larger than the
* length of extra data needed for the compressed version of block image.
*/
- len = pglz_compress(source, orig_len, dest, PGLZ_strategy_default);
+ len = pg_compress(source, orig_len, dest, PGLZ_MAX_BLCKSZ,
+ PGLZ_strategy_default);
if (len >= 0 &&
len + extra_bytes < orig_len)
{
diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c
index fc463601ff..04d99b97bf 100644
--- a/src/backend/utils/misc/guc.c
+++ b/src/backend/utils/misc/guc.c
@@ -42,6 +42,7 @@
#include "commands/vacuum.h"
#include "commands/variable.h"
#include "commands/trigger.h"
+#include "common/pg_lzcompress.h"
#include "common/string.h"
#include "funcapi.h"
#include "jit/jit.h"
@@ -470,6 +471,14 @@ static struct config_enum_entry shared_memory_options[] = {
{NULL, 0, false}
};
+static const struct config_enum_entry compression_algorithm_options[] = {
+ {"pglz", COMPRESS_ALGO_PGLZ, false},
+#ifdef HAVE_LZ4
+ {"lz4", COMPRESS_ALGO_LZ4, false},
+#endif
+ {NULL, 0, false}
+};
+
/*
* Options for enum values stored in other modules
*/
@@ -4540,7 +4549,7 @@ static struct config_enum ConfigureNamesEnum[] =
{
{"ssl_max_protocol_version", PGC_SIGHUP, CONN_AUTH_SSL,
- gettext_noop("Sets the maximum SSL/TLS protocol version to use."),
+ gettext_noop("Chooses the compression algorithm for TOAST and WAL."),
NULL,
GUC_SUPERUSER_ONLY
},
@@ -4550,6 +4559,18 @@ static struct config_enum ConfigureNamesEnum[] =
NULL, NULL, NULL
},
+ {
+ {"compression_algorithm", PGC_SIGHUP, RESOURCES_DISK,
+ gettext_noop("Sets the maximum SSL/TLS protocol version to use."),
+ NULL,
+ GUC_SUPERUSER_ONLY
+ },
+ &compression_algorithm,
+ COMPRESS_ALGO_PGLZ,
+ compression_algorithm_options,
+ NULL, NULL, NULL
+ },
+
/* End-of-list marker */
{
{NULL, 0, 0, NULL, NULL}, NULL, 0, NULL, NULL, NULL, NULL
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 988b3987d0..8e8b966d67 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -187,6 +187,7 @@
#include "common/pg_lzcompress.h"
+int compression_algorithm = COMPRESS_ALGO_PGLZ;
/* ----------
* Local definitions
@@ -505,8 +506,8 @@ pglz_find_match(int16 *hstart, const char *input, const char *end,
* bytes written in buffer dest, or -1 if compression fails.
* ----------
*/
-int32
-pglz_compress(const char *source, int32 slen, char *dest,
+static int32
+pglz_compress(const char *source, int32 slen, char *dest, int32 capacity,
const PGLZ_Strategy *strategy)
{
unsigned char *bp = (unsigned char *) dest;
@@ -771,3 +772,132 @@ pglz_decompress(const char *source, int32 slen, char *dest,
*/
return (char *) dp - dest;
}
+
+#ifdef HAVE_LZ4
+#include "utils/elog.h"
+
+static int32
+lz4_compress(const char *source, int32 slen, char *dest, int32 capacity,
+ const PGLZ_Strategy *strategy)
+{
+ int ret;
+
+ ret = LZ4_compress_default(source, dest, slen, capacity);
+
+ /*
+ * In case of compression error, return -1 which callers should take
+ * as incompressible data.
+ */
+ if (ret == 0)
+ return -1;
+
+ return ret;
+}
+
+int32
+lz4_decompress(const char *source, int32 slen, char *dest,
+ int32 rawsize, bool check_complete)
+{
+ int ret;
+
+ if (check_complete)
+ {
+ ret = LZ4_decompress_safe(source, dest, slen, rawsize);
+
+ /*
+ * Check we decompressed the right amount.
+ */
+ if (ret != rawsize)
+ return -1;
+ }
+ else
+ ret = LZ4_decompress_safe_partial(source, dest, slen, rawsize,
+ rawsize);
+
+ return ret;
+}
+#endif
+
+/*
+ * Compress using configured algorithm
+ */
+int32
+pg_compress(const char *source, int32 slen, char *dest, int32 capacity,
+ const PGLZ_Strategy *strategy)
+{
+ int32 ret;
+
+ switch (compression_algorithm)
+ {
+ case COMPRESS_ALGO_PGLZ:
+ dest[0] = COMPRESS_ALGO_PGLZ;
+ ret = pglz_compress(source, slen, &dest[1], capacity - 1,
+ strategy);
+ break;
+#ifdef HAVE_LZ4
+ case COMPRESS_ALGO_LZ4:
+ dest[0] = COMPRESS_ALGO_LZ4;
+ ret = lz4_compress(source, slen, &dest[1], capacity - 1,
+ strategy);
+ break;
+#endif
+ default:
+ pg_unreachable();
+ }
+
+ if (ret >= 0)
+ return ret + 1;
+
+ return ret;
+}
+
+/*
+ * Decompress data compressed with one of the supported algorithms.
+ */
+int32
+pg_decompress(const char *source, int32 slen, char *dest,
+ int32 rawsize, bool check_complete)
+{
+ switch (source[0])
+ {
+ case COMPRESS_ALGO_PGLZ:
+ return pglz_decompress(&source[1], slen - 1, dest, rawsize,
+ check_complete);
+#ifdef HAVE_LZ4
+ case COMPRESS_ALGO_LZ4:
+ return lz4_decompress(&source[1], slen - 1, dest, rawsize,
+ check_complete);
+#endif
+ default:
+ Assert(false); /* XXX: Can't elog here. */
+ }
+
+ pg_unreachable();
+}
+
+/*
+ * Compute the buffer size required by pg_compress for a configured algorithm
+ * including our header size.
+ */
+int32
+pg_compress_bound(int32 slen)
+{
+ switch (compression_algorithm)
+ {
+ case COMPRESS_ALGO_PGLZ:
+ /*
+ * For pglz we allow 4 bytes for overrun before detecting
+ * compression failure.
+ */
+ return slen + 4 + SIZEOF_PG_COMPRESS_HEADER;
+#ifdef HAVE_LZ4
+ case COMPRESS_ALGO_LZ4:
+ /* LZ4 provides direct interface for calculating needed space. */
+ return LZ4_compressBound(slen) + SIZEOF_PG_COMPRESS_HEADER;
+#endif
+ default:
+ pg_unreachable();
+ }
+
+ pg_unreachable();
+}
diff --git a/src/include/common/pg_lzcompress.h b/src/include/common/pg_lzcompress.h
index 555576436c..04453f4574 100644
--- a/src/include/common/pg_lzcompress.h
+++ b/src/include/common/pg_lzcompress.h
@@ -10,15 +10,35 @@
#ifndef _PG_LZCOMPRESS_H_
#define _PG_LZCOMPRESS_H_
+#ifdef HAVE_LZ4
+#include "lz4.h"
-/* ----------
- * PGLZ_MAX_OUTPUT -
+#define SIZEOF_PG_COMPRESS_HEADER 1
+/*
+ * Macro version of pg_compress_bound, less precise, usable in places where
+ * we need compile time size information.
+ * We add +1 compared to what algorithms need because that's the size of
+ * pg_compress header.
+ */
+#define PGLZ_MAX_OUTPUT(_dlen) (Max((_dlen) + 4, LZ4_COMPRESSBOUND(_dlen)) + \
+ SIZEOF_PG_COMPRESS_HEADER)
+#else
+#define PGLZ_MAX_OUTPUT(_dlen) ((_dlen) + 4 + SIZEOF_PG_COMPRESS_HEADER)
+#endif
+
+/*
+ * PGLZCompressionAlgo
*
- * Macro to compute the buffer size required by pglz_compress().
- * We allow 4 bytes for overrun before detecting compression failure.
- * ----------
+ * Which algorithm to use for TOAST and WAL compression.
+ *
+ * COMPRESS_ALGO_PGLZ - use the builtin pglz algorithm
+ * COMPRESS_ALGO_LZ4 - use the LZ4 library
*/
-#define PGLZ_MAX_OUTPUT(_dlen) ((_dlen) + 4)
+typedef enum
+{
+ COMPRESS_ALGO_PGLZ = 0,
+ COMPRESS_ALGO_LZ4
+} PGLZCompressAlgo;
/* ----------
@@ -78,14 +98,25 @@ typedef struct PGLZ_Strategy
extern const PGLZ_Strategy *const PGLZ_strategy_default;
extern const PGLZ_Strategy *const PGLZ_strategy_always;
+/*
+ * Compression algorithm.
+ */
+
+extern int compression_algorithm;
/* ----------
* Global function declarations
* ----------
*/
-extern int32 pglz_compress(const char *source, int32 slen, char *dest,
- const PGLZ_Strategy *strategy);
extern int32 pglz_decompress(const char *source, int32 slen, char *dest,
- int32 rawsize, bool check_complete);
+ int32 rawsize, bool check_complete);
+extern int32 lz4_decompress(const char *source, int32 slen, char *dest,
+ int32 rawsize, bool check_complete);
+extern int32 pg_compress(const char *source, int32 slen, char *dest, int32 capacity,
+ const PGLZ_Strategy *strategy);
+extern int32 pg_decompress(const char *source, int32 slen, char *dest,
+ int32 rawsize, bool check_complete);
+
+extern int32 pg_compress_bound(int32 slen);
#endif /* _PG_LZCOMPRESS_H_ */
diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in
index 512213aa32..5e37aab4c5 100644
--- a/src/include/pg_config.h.in
+++ b/src/include/pg_config.h.in
@@ -718,6 +718,9 @@
/* Define to 1 if you have the <uuid/uuid.h> header file. */
#undef HAVE_UUID_UUID_H
+/* Define to 1 to build with LZ4 support. */
+#undef HAVE_LZ4
+
/* Define to 1 if you have the <wchar.h> header file. */
#undef HAVE_WCHAR_H
diff --git a/src/include/postgres.h b/src/include/postgres.h
index 057a3413ac..9578588880 100644
--- a/src/include/postgres.h
+++ b/src/include/postgres.h
@@ -145,7 +145,8 @@ typedef union
struct /* Compressed-in-line format */
{
uint32 va_header;
- uint32 va_rawsize; /* Original data size (excludes header) */
+ uint32 va_rawsize:31; /* Original data size (excludes header) */
+ int va_cformat:1;
char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */
} va_compressed;
} varattrib_4b;
diff --git a/src/test/Makefile b/src/test/Makefile
index efb206aa75..c6287e7a04 100644
--- a/src/test/Makefile
+++ b/src/test/Makefile
@@ -32,12 +32,18 @@ ifneq (,$(filter ssl,$(PG_TEST_EXTRA)))
SUBDIRS += ssl
endif
endif
+ifeq ($(with_lz4),yes)
+ifneq (,$(filter toast,$(PG_TEST_EXTRA)))
+SUBDIRS += toast
+endif
+endif
# We don't build or execute these by default, but we do want "make
# clean" etc to recurse into them. (We must filter out those that we
# have conditionally included into SUBDIRS above, else there will be
# make confusion.)
-ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread ssl)
+ALWAYS_SUBDIRS = $(filter-out $(SUBDIRS),examples kerberos ldap locale thread \
+ ssl toast)
# We want to recurse to all subdirs for all standard targets, except that
# installcheck and install should not recurse into the subdirectory "modules".
diff --git a/src/test/toast/.gitignore b/src/test/toast/.gitignore
new file mode 100644
index 0000000000..871e943d50
--- /dev/null
+++ b/src/test/toast/.gitignore
@@ -0,0 +1,2 @@
+# Generated by test suite
+/tmp_check/
diff --git a/src/test/toast/Makefile b/src/test/toast/Makefile
new file mode 100644
index 0000000000..5a1ff09a13
--- /dev/null
+++ b/src/test/toast/Makefile
@@ -0,0 +1,25 @@
+#-------------------------------------------------------------------------
+#
+# Makefile for src/test/toast
+#
+# Portions Copyright (c) 1996-2019, PostgreSQL Global Development Group
+# Portions Copyright (c) 1994, Regents of the University of California
+#
+# src/test/toast/Makefile
+#
+#-------------------------------------------------------------------------
+
+subdir = src/test/recovery
+top_builddir = ../../..
+include $(top_builddir)/src/Makefile.global
+
+export with_lz4
+
+check:
+ $(prove_check)
+
+installcheck:
+ $(prove_installcheck)
+
+clean distclean maintainer-clean:
+ rm -rf tmp_check
diff --git a/src/test/toast/README b/src/test/toast/README
new file mode 100644
index 0000000000..8802ecbe06
--- /dev/null
+++ b/src/test/toast/README
@@ -0,0 +1,25 @@
+src/test/tosat/README
+
+Regression tests for TOAST compression
+======================================
+
+This directory contains a test suite for TOAST compression and replication.
+
+Running the tests
+=================
+
+NOTE: You must have given the --enable-tap-tests argument to configure.
+Also, to use "make installcheck", you must have built and installed
+contrib/test_decoding in addition to the core code.
+
+Run
+ make check
+or
+ make installcheck
+You can use "make installcheck" if you previously did "make install".
+In that case, the code in the installation tree is tested. With
+"make check", a temporary installation tree is built from the current
+sources and then tested.
+
+Either way, this test initializes, starts, and stops several test Postgres
+clusters.
diff --git a/src/test/toast/t/001_lz4.pl b/src/test/toast/t/001_lz4.pl
new file mode 100644
index 0000000000..89f7cd177f
--- /dev/null
+++ b/src/test/toast/t/001_lz4.pl
@@ -0,0 +1,124 @@
+use strict;
+use warnings;
+use PostgresNode;
+use TestLib;
+use Test::More;
+
+use File::Copy;
+
+use FindBin;
+use lib $FindBin::RealBin;
+
+if ($ENV{with_lz4} eq 'yes')
+{
+ plan tests => 10;
+}
+else
+{
+ plan skip_all => 'LZ4 not supported by this build';
+}
+
+#### Set up the server.
+note "setting up data directory";
+my $node = get_new_node('master');
+$node->init;
+$node->append_conf('postgresql.conf', qq[
+compression_algorithm = lz4
+wal_compression = on
+]);
+$node->start;
+
+# Run this before we lock down access below.
+my $result = $node->safe_psql('postgres', "SHOW compression_algorithm");
+is($result, 'lz4', 'compression_algorithm set to lz4');
+
+$node->safe_psql('postgres',
+ qq[
+ CREATE TABLE toast_test (
+ id int,
+ data text
+ )]);
+
+$node->safe_psql('postgres',
+ 'ALTER TABLE toast_test ALTER COLUMN data SET STORAGE MAIN');
+
+# This will actually be compressed inline as it's easy to compress
+$node->safe_psql('postgres',
+qq[
+ INSERT INTO toast_test
+ SELECT n, repeat('toasted', 1000)
+ FROM generate_series(1, 100) s(n);
+]);
+
+
+my $toast_size = $node->safe_psql('postgres',
+qq[
+ SELECT pg_relation_size((SELECT reltoastrelid FROM pg_catalog.pg_class WHERE relname = 'toast_test'));
+]);
+
+ok($toast_size == 0, 'toast table is used');
+
+$node->safe_psql('postgres',
+ 'ALTER TABLE toast_test ALTER COLUMN data SET STORAGE EXTENDED');
+
+# Something less easily compressable so that it's in TOAST table
+$node->safe_psql('postgres',
+qq[
+ INSERT INTO toast_test
+ SELECT n, (SELECT string_agg(md5(t::text),'')
+ FROM generate_series(1, 200) q(t))
+ FROM generate_series(101, 200) s(n);
+]);
+
+$toast_size = $node->safe_psql('postgres',
+qq[
+ SELECT pg_relation_size((SELECT reltoastrelid FROM pg_catalog.pg_class WHERE relname = 'toast_test'));
+]);
+
+ok($toast_size > 0, 'toast table is used');
+
+# check if we can select data
+is($node->safe_psql('postgres',
+ qq[SELECT id, length(data) FROM toast_test WHERE id = 1]),
+ '1|7000', 'can select compressed data');
+is($node->safe_psql('postgres',
+ qq[SELECT id, length(data) FROM toast_test WHERE id = 200]),
+ '200|6400', 'can select TOAST compressed data');
+
+# test slicing
+is($node->safe_psql('postgres',
+ qq[SELECT id, substr(data, 1, 10) FROM toast_test WHERE id = 50]),
+ '50|toastedtoa', 'slicing of compressed data works');
+
+is($node->safe_psql('postgres',
+ qq[SELECT id, substr(data, 1, 10) FROM toast_test WHERE id = 150]),
+ '150|c4ca4238a0', 'slicing of TOAST works');
+
+$node->append_conf('postgresql.conf', qq[
+compression_algorithm = pglz
+]);
+$node->reload;
+
+# Run this before we lock down access below.
+$result = $node->safe_psql('postgres', "SHOW compression_algorithm");
+is($result, 'pglz', 'compression_algorithm set to pglz');
+
+$node->safe_psql('postgres',
+qq[
+ INSERT INTO toast_test
+ SELECT n, (SELECT string_agg(md5(t::text),'')
+ FROM generate_series(1, 200) q(t))
+ FROM generate_series(201, 300) s(n);
+]);
+
+is($node->safe_psql('postgres',
+ qq[SELECT id, length(data) FROM toast_test WHERE id IN (200, 201)]),
+q[200|6400
+201|6400], 'can select TOAST with different compression for different rows');
+
+is($node->safe_psql('postgres',
+ qq[SELECT id, substr(data, 1, 10) FROM toast_test WHERE id IN (150, 250)]),
+q[150|c4ca4238a0
+250|c4ca4238a0], 'slicing of TOAST works with different compression for different row');
+
+done_testing();
--
2.20.1
* Re: pglz performance
@ 2019-08-04 09:57 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Petr Jelinek <petr@2ndquadrant.com>
3 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-08-04 09:57 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
> 2 авг. 2019 г., в 21:39, Andres Freund <andres@anarazel.de> написал(а):
>
> On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
>> We have some kind of "roadmap" of "extensible pglz". We plan to provide implementation on Novembers CF.
>
> I don't understand why it's a good idea to improve the compression side
> of pglz. There's plenty other people that spent a lot of time developing
> better compression algorithms.
Improving compression side of pglz has two different projects:
1. Faster compression with less code and same compression ratio (patch in this thread).
2. Better compression ratio with at least same compression speed of uncompressed values.
Why I want to do patch for 2? Because it's interesting.
Will 1 or 2 be reviewed or committed? I have no idea.
Will many users benefit from 1 or 2? Yes, clearly. Unless we force everyone to stop compressing with pglz.
>> Currently, pglz starts with empty cache map: there is no prior 4k bytes before start. We can add imaginary prefix to any data with common substrings: this will enhance compression ratio.
>> It is hard to decide on training data set for this "common prefix". So we want to produce extension with aggregate function which produces some "adapted common prefix" from users's data.
>> Then we can "reserve" few negative bytes for "decompression commands". This command can instruct database on which common prefix to use.
>> But also system command can say "invoke decompression from extension".
>>
>> Thus, user will be able to train database compression on his data and substitute pglz compression with custom compression method seamlessly.
>>
>> This will make hard-choosen compression unneeded, but seems overly hacky. But there will be no need to have lz4, zstd, brotli, lzma and others in core. Why not provide e.g. "time series compression"? Or "DNA compression"? Whatever gun user wants for his foot.
>
> I think this is way too complicated, and will provide not particularly
> much benefit for the majority users.
>
> In fact, I'll argue that we should flat out reject any such patch until
> we have at least one decent default compression algorithm in
> core. You're trying to work around a poor compression algorithm with
> complicated dictionary improvement
OK. The idea of something plugged into pglz seemed odd even to me.
But looks like it restarted lz4 discussion :)
> , that require user interaction, and
> only will work in a relatively small subset of the cases, and will very
> often increase compression times.
No, surely, if implementation of "common prefix" will increase compression times I will not even post a patch.
BTW, lz4 also supports "common prefix", let's do that too?
Here's link on Zstd dictionary builder, but it is compatible with lz4
https://github.com/facebook/zstd#the-case-for-small-data-compression
We actually have small datums.
> 4 авг. 2019 г., в 5:41, Petr Jelinek <petr@2ndquadrant.com> написал(а):
>
> Just so that we don't idly talk, what do you think about the attached?
> It:
> - adds new GUC compression_algorithm with possible values of pglz (default) and lz4 (if lz4 is compiled in), requires SIGHUP
> - adds --with-lz4 configure option (default yes, so the configure option is actually --without-lz4) that enables the lz4, it's using system library
> - uses the compression_algorithm for both TOAST and WAL compression (if on)
> - supports slicing for lz4 as well (pglz was already supported)
> - supports reading old TOAST values
> - adds 1 byte header to the compressed data where we currently store the algorithm kind, that leaves us with 254 more to add :) (that's an extra overhead compared to the current state)
> - changes the rawsize in TOAST header to 31 bits via bit packing
> - uses the extra bit to differentiate between old and new format
> - supports reading from table which has different rows stored with different algorithm (so that the GUC itself can be freely changed)
That's cool. I suggest defaulting to lz4 if it is available. You cannot start cluster on non-lz4 binaries which used lz4 once.
Do we plan the possibility of compression algorithm as extension? Or will all algorithms be packed into that byte in core?
What about lz4 "common prefix"? System or user-defined. If lz4 is compiled in we can even offer in-system training, just make sure that trained prefixes will make their way to standbys.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 11:52 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Petr Jelinek <petr@2ndquadrant.com>
3 siblings, 1 reply; 70+ messages in thread
From: Tomas Vondra @ 2019-08-04 11:52 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Sun, Aug 04, 2019 at 02:41:24AM +0200, Petr Jelinek wrote:
>Hi,
>
>On 02/08/2019 21:48, Tomas Vondra wrote:
>>On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote:
>>
>>>
>>>>Another question is whether we'd actually want to include the code in
>>>>core directly, or use system libraries (and if some packagers might
>>>>decide to disable that, for whatever reason).
>>>
>>>I'd personally say we should have an included version, and a
>>>--with-system-... flag that uses the system one.
>>>
>>
>>OK. I'd say to require a system library, but that's a minor detail.
>>
>
>Same here.
>
>Just so that we don't idly talk, what do you think about the attached?
>It:
>- adds new GUC compression_algorithm with possible values of pglz
>(default) and lz4 (if lz4 is compiled in), requires SIGHUP
>- adds --with-lz4 configure option (default yes, so the configure
>option is actually --without-lz4) that enables the lz4, it's using
>system library
>- uses the compression_algorithm for both TOAST and WAL compression (if on)
>- supports slicing for lz4 as well (pglz was already supported)
>- supports reading old TOAST values
>- adds 1 byte header to the compressed data where we currently store
>the algorithm kind, that leaves us with 254 more to add :) (that's an
>extra overhead compared to the current state)
>- changes the rawsize in TOAST header to 31 bits via bit packing
>- uses the extra bit to differentiate between old and new format
>- supports reading from table which has different rows stored with
>different algorithm (so that the GUC itself can be freely changed)
>
Cool.
>Simple docs and a TAP test included.
>
>I did some basic performance testing (it's not really my thing though,
>so I would appreciate if somebody did more).
>I get about 7x perf improvement on data load with lz4 compared to pglz
>on my dataset but strangely only tiny decompression improvement.
>Perhaps more importantly I also did before patch and after patch tests
>with pglz and the performance difference with my data set was <1%.
>
>Note that this will just link against lz4, it does not add lz4 into
>PostgreSQL code-base.
>
WFM, although I think Andres wanted to do both (link against system and
add lz4 code as a fallback). I think the question is what happens when
you run with lz4 for a while, and then switch to binaries without lz4
support. Or when you try to replicate from lz4-enabled instance to an
instance without it. Especially for physical replication, but I suppose
it may affect logical replication with binary protocol?
A very brief review:
1) I wonder what "runstatedir" is about.
2) This seems rather suspicious, and obviously the comment is now
entirely bogus:
/* Check that off_t can represent 2**63 - 1 correctly.
We can't simply define LARGE_OFF_T to be 9223372036854775807,
since some C++ compilers masquerading as C compilers
incorrectly reject 9223372036854775807. */
-#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
+#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31) << 31))
3) I can't really build without lz4:
config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
pg_lzcompress.c: In function ‘pg_compress_bound’:
pg_lzcompress.c:892:22: error: ‘SIZEOF_PG_COMPRESS_HEADER’ undeclared (first use in this function)
return slen + 4 + SIZEOF_PG_COMPRESS_HEADER;
^~~~~~~~~~~~~~~~~~~~~~~~~
pg_lzcompress.c:892:22: note: each undeclared identifier is reported only once for each function it appears in
4) I did a simple test with physical replication, with lz4 enabled on
both sides (well, can't build without lz4 anyway, per previous point).
It immediately failed like this:
FATAL: failed to restore block image
CONTEXT: WAL redo at 0/5000A40 for Btree/INSERT_LEAF: off 138
LOG: startup process (PID 15937) exited with exit code 1
This is a simple UPDATE on a trivial table:
create table t (a int primary key);
insert into t select i from generate_series(1,1000) s(i);
update t set a = a - 100000 where random () < 0.1;
with some checkpoints to force FPW (and wal_compression=on, of course).
I haven't tried `make check-world` but I suppose some of the TAP tests
should fail because of this. And if not, we need to improve coverage.
5) I wonder why compression_algorithm is defined as PGC_SIGHUP. Why not
to allow users to set it per session? I suppose we might have a separate
option for WAL compression_algorithm.
6) It seems a bit strange that pg_compress/pg_decompress are now defined
in pglz_compress.{c,h}. Maybe we should invent src/common/compress.c?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 15:52 Petr Jelinek <petr@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 0 replies; 70+ messages in thread
From: Petr Jelinek @ 2019-08-04 15:52 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 04/08/2019 11:57, Andrey Borodin wrote:
>
>
>> 2 авг. 2019 г., в 21:39, Andres Freund <andres@anarazel.de> написал(а):
>>
>> On 2019-08-02 20:40:51 +0500, Andrey Borodin wrote:
>>> We have some kind of "roadmap" of "extensible pglz". We plan to provide implementation on Novembers CF.
>>
>> I don't understand why it's a good idea to improve the compression side
>> of pglz. There's plenty other people that spent a lot of time developing
>> better compression algorithms.
> Improving compression side of pglz has two different projects:
> 1. Faster compression with less code and same compression ratio (patch in this thread).
> 2. Better compression ratio with at least same compression speed of uncompressed values.
> Why I want to do patch for 2? Because it's interesting.
> Will 1 or 2 be reviewed or committed? I have no idea.
> Will many users benefit from 1 or 2? Yes, clearly. Unless we force everyone to stop compressing with pglz.
>
FWIW I agree.
>> Just so that we don't idly talk, what do you think about the attached?
>> It:
>> - adds new GUC compression_algorithm with possible values of pglz (default) and lz4 (if lz4 is compiled in), requires SIGHUP
>> - adds --with-lz4 configure option (default yes, so the configure option is actually --without-lz4) that enables the lz4, it's using system library
>> - uses the compression_algorithm for both TOAST and WAL compression (if on)
>> - supports slicing for lz4 as well (pglz was already supported)
>> - supports reading old TOAST values
>> - adds 1 byte header to the compressed data where we currently store the algorithm kind, that leaves us with 254 more to add :) (that's an extra overhead compared to the current state)
>> - changes the rawsize in TOAST header to 31 bits via bit packing
>> - uses the extra bit to differentiate between old and new format
>> - supports reading from table which has different rows stored with different algorithm (so that the GUC itself can be freely changed)
> That's cool. I suggest defaulting to lz4 if it is available. You cannot start cluster on non-lz4 binaries which used lz4 once.
> Do we plan the possibility of compression algorithm as extension? Or will all algorithms be packed into that byte in core?
What I wrote does not expect extensions providing new compression. We'd
have to somehow reserve compression ids for specific extensions and that
seems like a lot of extra complexity for little benefit. I don't see
much benefit in having more than say 3 generic compressors (I could
imagine adding zstd). If you are thinking about data type specific
compression then I think this is wrong layer.
> What about lz4 "common prefix"? System or user-defined. If lz4 is compiled in we can even offer in-system training, just make sure that trained prefixes will make their way to standbys.
>
I definitely don't plan to work on common prefix. But don't see why that
could not be added later.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 15:53 Petr Jelinek <petr@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 2 replies; 70+ messages in thread
From: Petr Jelinek @ 2019-08-04 15:53 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 04/08/2019 13:52, Tomas Vondra wrote:
> On Sun, Aug 04, 2019 at 02:41:24AM +0200, Petr Jelinek wrote:
>> Hi,
>>
>> On 02/08/2019 21:48, Tomas Vondra wrote:
>>> On Fri, Aug 02, 2019 at 11:20:03AM -0700, Andres Freund wrote:
>>>
>>>>
>>>>> Another question is whether we'd actually want to include the code in
>>>>> core directly, or use system libraries (and if some packagers might
>>>>> decide to disable that, for whatever reason).
>>>>
>>>> I'd personally say we should have an included version, and a
>>>> --with-system-... flag that uses the system one.
>>>>
>>>
>>> OK. I'd say to require a system library, but that's a minor detail.
>>>
>>
>> Same here.
>>
>> Just so that we don't idly talk, what do you think about the attached?
>> It:
>> - adds new GUC compression_algorithm with possible values of pglz
>> (default) and lz4 (if lz4 is compiled in), requires SIGHUP
>> - adds --with-lz4 configure option (default yes, so the configure
>> option is actually --without-lz4) that enables the lz4, it's using
>> system library
>> - uses the compression_algorithm for both TOAST and WAL compression
>> (if on)
>> - supports slicing for lz4 as well (pglz was already supported)
>> - supports reading old TOAST values
>> - adds 1 byte header to the compressed data where we currently store
>> the algorithm kind, that leaves us with 254 more to add :) (that's an
>> extra overhead compared to the current state)
>> - changes the rawsize in TOAST header to 31 bits via bit packing
>> - uses the extra bit to differentiate between old and new format
>> - supports reading from table which has different rows stored with
>> different algorithm (so that the GUC itself can be freely changed)
>>
>
> Cool.
>
>> Simple docs and a TAP test included.
>>
>> I did some basic performance testing (it's not really my thing though,
>> so I would appreciate if somebody did more).
>> I get about 7x perf improvement on data load with lz4 compared to pglz
>> on my dataset but strangely only tiny decompression improvement.
>> Perhaps more importantly I also did before patch and after patch tests
>> with pglz and the performance difference with my data set was <1%.
>>
>> Note that this will just link against lz4, it does not add lz4 into
>> PostgreSQL code-base.
>>
>
> WFM, although I think Andres wanted to do both (link against system and
> add lz4 code as a fallback). I think the question is what happens when
> you run with lz4 for a while, and then switch to binaries without lz4
> support. Or when you try to replicate from lz4-enabled instance to an
> instance without it. Especially for physical replication, but I suppose
> it may affect logical replication with binary protocol?
>
I generally prefer having system library, we don't include for example
ICU either.
>
> A very brief review:
>
>
> 1) I wonder what "runstatedir" is about.
>
No idea, that stuff is generated by autoconf from configure.in.
>
> 2) This seems rather suspicious, and obviously the comment is now
> entirely bogus:
>
> /* Check that off_t can represent 2**63 - 1 correctly.
> We can't simply define LARGE_OFF_T to be 9223372036854775807,
> since some C++ compilers masquerading as C compilers
> incorrectly reject 9223372036854775807. */
> -#define LARGE_OFF_T (((off_t) 1 << 62) - 1 + ((off_t) 1 << 62))
> +#define LARGE_OFF_T ((((off_t) 1 << 31) << 31) - 1 + (((off_t) 1 << 31)
> << 31))
>
Same as above. TBH I am not sure why we even include configure in git
repo given that different autoconf versions can build different outputs.
>
> 3) I can't really build without lz4:
>
> config.status: linking src/makefiles/Makefile.linux to src/Makefile.port
> pg_lzcompress.c: In function ‘pg_compress_bound’:
> pg_lzcompress.c:892:22: error: ‘SIZEOF_PG_COMPRESS_HEADER’ undeclared
> (first use in this function)
> return slen + 4 + SIZEOF_PG_COMPRESS_HEADER;
> ^~~~~~~~~~~~~~~~~~~~~~~~~
> pg_lzcompress.c:892:22: note: each undeclared identifier is reported
> only once for each function it appears in
>
Okay, that's just problem of SIZEOF_PG_COMPRESS_HEADER being defined
inside the HAVE_LZ4 ifdef while it should be defined above ifdef.
>
> 4) I did a simple test with physical replication, with lz4 enabled on
> both sides (well, can't build without lz4 anyway, per previous point).
> It immediately failed like this:
>
> FATAL: failed to restore block image
> CONTEXT: WAL redo at 0/5000A40 for Btree/INSERT_LEAF: off 138
> LOG: startup process (PID 15937) exited with exit code 1
>
> This is a simple UPDATE on a trivial table:
>
> create table t (a int primary key);
> insert into t select i from generate_series(1,1000) s(i);
> update t set a = a - 100000 where random () < 0.1;
>
> with some checkpoints to force FPW (and wal_compression=on, of course).
>
> I haven't tried `make check-world` but I suppose some of the TAP tests
> should fail because of this. And if not, we need to improve coverage.
>
FWIW I did run check-world without problems, will have to look into this.
>
> 5) I wonder why compression_algorithm is defined as PGC_SIGHUP. Why not
> to allow users to set it per session? I suppose we might have a separate
> option for WAL compression_algorithm.
>
Yeah I was thinking we might want to change wal_compression to enum as
well. Although that complicates the code quite a bit (the caller has to
decide algorithm instead compression system doing it).
>
> 6) It seems a bit strange that pg_compress/pg_decompress are now defined
> in pglz_compress.{c,h}. Maybe we should invent src/common/compress.c?
>
It does not seem worth it to invent new module for like 20 lines of
wrapper code.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 17:30 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Petr Jelinek <petr@2ndquadrant.com>
1 sibling, 0 replies; 70+ messages in thread
From: Tomas Vondra @ 2019-08-04 17:30 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Sun, Aug 04, 2019 at 05:53:26PM +0200, Petr Jelinek wrote:
>
> ...
>
>>
>>4) I did a simple test with physical replication, with lz4 enabled on
>>both sides (well, can't build without lz4 anyway, per previous point).
>>It immediately failed like this:
>>
>>FATAL: failed to restore block image
>>CONTEXT: WAL redo at 0/5000A40 for Btree/INSERT_LEAF: off 138
>>LOG: startup process (PID 15937) exited with exit code 1
>>
>>This is a simple UPDATE on a trivial table:
>>
>>create table t (a int primary key);
>>insert into t select i from generate_series(1,1000) s(i);
>>update t set a = a - 100000 where random () < 0.1;
>>
>>with some checkpoints to force FPW (and wal_compression=on, of course).
>>
>>I haven't tried `make check-world` but I suppose some of the TAP tests
>>should fail because of this. And if not, we need to improve coverage.
>>
>
>FWIW I did run check-world without problems, will have to look into this.
>
Not sure if we bother to set wal_compression=on for check-world (I don't
think we do, but I may be missing something), so maybe check-world does
not really test wal compression.
IMO the issue is that RestoreBlockImage() still calls pglz_decompress
directly, instead of going through pg_decompress().
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 19:20 Andres Freund <andres@anarazel.de>
parent: Petr Jelinek <petr@2ndquadrant.com>
3 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2019-08-04 19:20 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 2019-08-04 02:41:24 +0200, Petr Jelinek wrote:
> Same here.
>
> Just so that we don't idly talk, what do you think about the attached?
Cool!
> It:
> - adds new GUC compression_algorithm with possible values of pglz (default)
> and lz4 (if lz4 is compiled in), requires SIGHUP
As Tomas remarked, I think it shouldn't be SIGHUP but USERSET. And I
think lz4 should be preferred, if available. I could see us using a
list style guc, so we could set it to lz4, pglz, and the first available
one would be used.
> - adds 1 byte header to the compressed data where we currently store the
> algorithm kind, that leaves us with 254 more to add :) (that's an extra
> overhead compared to the current state)
Hm. Why do we need an additional byte? IIRC my patch added that only
for the case we would run out of space for compression formats without
extending any sizes?
> - changes the rawsize in TOAST header to 31 bits via bit packing
> - uses the extra bit to differentiate between old and new format
Hm. Wouldn't it be easier to just use a different vartag for this?
> - I expect my changes to configure.in are not the greatest as I don't have
> pretty much zero experience with autoconf
FWIW the configure output changes are likely because you used a modified
version of autoconf. Unfortunately debian/ubuntu ship one with vendor
patches.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 22:08 Petr Jelinek <petr@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 70+ messages in thread
From: Petr Jelinek @ 2019-08-04 22:08 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 04/08/2019 21:20, Andres Freund wrote:
> On 2019-08-04 02:41:24 +0200, Petr Jelinek wrote:
>> Same here.
>>
>> Just so that we don't idly talk, what do you think about the attached?
>
> Cool!
>
>> It:
>> - adds new GUC compression_algorithm with possible values of pglz (default)
>> and lz4 (if lz4 is compiled in), requires SIGHUP
>
> As Tomas remarked, I think it shouldn't be SIGHUP but USERSET. And I
> think lz4 should be preferred, if available. I could see us using a
> list style guc, so we could set it to lz4, pglz, and the first available
> one would be used.
>
Sounds reasonable.
>> - adds 1 byte header to the compressed data where we currently store the
>> algorithm kind, that leaves us with 254 more to add :) (that's an extra
>> overhead compared to the current state)
>
> Hm. Why do we need an additional byte? IIRC my patch added that only
> for the case we would run out of space for compression formats without
> extending any sizes?
>
Yeah your patch worked differently (I didn't actually use any code from
it). The main reason why I add the byte is that I am storing the
algorithm in the compressed value itself, not in varlena header. I was
mainly trying to not have every caller care about storing and loading
the compression algorithm. I also can't say I particularly like that
hack in your patch.
However if we'd want to have separate GUCs for TOAST and WAL then we'll
have to do that anyway so maybe it does not matter anymore (we can't use
similar hack there AFAICS though).
>
>> - changes the rawsize in TOAST header to 31 bits via bit packing
>> - uses the extra bit to differentiate between old and new format
>
> Hm. Wouldn't it be easier to just use a different vartag for this?
>
That would only work for external TOAST pointers right? The compressed
varlena can also be stored inline and potentially in index tuple.
>
>> - I expect my changes to configure.in are not the greatest as I don't have
>> pretty much zero experience with autoconf
>
> FWIW the configure output changes are likely because you used a modified
> version of autoconf. Unfortunately debian/ubuntu ship one with vendor
> patches.
>
Yeah, Ubuntu here, that explains.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 22:15 Andres Freund <andres@anarazel.de>
parent: Petr Jelinek <petr@2ndquadrant.com>
1 sibling, 1 reply; 70+ messages in thread
From: Andres Freund @ 2019-08-04 22:15 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 2019-08-04 17:53:26 +0200, Petr Jelinek wrote:
> > 5) I wonder why compression_algorithm is defined as PGC_SIGHUP. Why not
> > to allow users to set it per session? I suppose we might have a separate
> > option for WAL compression_algorithm.
> >
>
> Yeah I was thinking we might want to change wal_compression to enum as well.
> Although that complicates the code quite a bit (the caller has to decide
> algorithm instead compression system doing it).
Isn't that basically required anyway? The WAL record will need to carry
information about the type of compression used, independent of
PGC_SIGHUP/PGC_USERSET, unless you want to make it an initdb option or
something super heavyweight like that.
We could just have the wal_compression assign hook set a the compression
callback and a compression type integer or such if we want to avoid
doing that kind of thing at runtime.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-04 22:19 Petr Jelinek <petr@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 1 reply; 70+ messages in thread
From: Petr Jelinek @ 2019-08-04 22:19 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 05/08/2019 00:15, Andres Freund wrote:
> Hi,
>
> On 2019-08-04 17:53:26 +0200, Petr Jelinek wrote:
>>> 5) I wonder why compression_algorithm is defined as PGC_SIGHUP. Why not
>>> to allow users to set it per session? I suppose we might have a separate
>>> option for WAL compression_algorithm.
>>>
>>
>> Yeah I was thinking we might want to change wal_compression to enum as well.
>> Although that complicates the code quite a bit (the caller has to decide
>> algorithm instead compression system doing it).
>
> Isn't that basically required anyway? The WAL record will need to carry
> information about the type of compression used, independent of
> PGC_SIGHUP/PGC_USERSET, unless you want to make it an initdb option or
> something super heavyweight like that.
>
It carries that information inside the compressed value, like I said in
the other reply, that's why the extra byte.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-05 07:04 Michael Paquier <michael@paquier.xyz>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 70+ messages in thread
From: Michael Paquier @ 2019-08-05 07:04 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andres Freund <andres@anarazel.de>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
On Fri, Aug 02, 2019 at 07:52:39PM +0200, Tomas Vondra wrote:
> On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote:
>> Why would they be stuck continuing to *compress* with pglz? As we
>> fully retoast on write anyway we can just gradually switch over to the
>> better algorithm. Decompression speed is another story, of course.
>
> Hmmm, I don't remember the details of those patches so I didn't realize
> it allows incremental recompression. If that's possible, that would mean
> existing systems can start using it. Which is good.
It may become a problem on some platforms though (Windows?), so
patches to improve either the compression or decompression of pglz are
not that much crazy as they are still likely going to be used, and for
read-mostly switching to a new algo may not be worth the extra cost so
it is not like we are going to drop it completely either. My take,
still the same as upthread, is that it mostly depends on the amount of
complexity each patch introduces compared to the performance gain.
> Another question is whether we'd actually want to include the code in
> core directly, or use system libraries (and if some packagers might
> decide to disable that, for whatever reason).
Linking to system libraries would make our maintenance much easier,
and when it comes to have a copy of something else in the tree we
would be stuck with more maintenance around it. These tend to rot
easily. After that comes the case where the compression algo is not
in the binary across one server to another, in which case we have an
automatic ERROR in case of a mismatching algo, or FATAL for
deompression of FPWs at recovery when wal_compression is used.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-05 07:26 Andres Freund <andres@anarazel.de>
parent: Petr Jelinek <petr@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andres Freund @ 2019-08-05 07:26 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 2019-08-05 00:19:28 +0200, Petr Jelinek wrote:
> It carries that information inside the compressed value, like I said in the
> other reply, that's why the extra byte.
I'm not convinced that that is a good plan - imo the reference to the
compressed data should carry that information.
I.e. in the case of toast, at least toast pointers should hold enough
information to determine the compression algorithm. And in the case of
WAL, the WAL record should contain that.
Consider e.g. adding support for slice fetching of datums compressed
with some algorithm - we should be able to determine whether that's
possible without fetching the datum (so we can take a non-exceptional
path for datums compressed otherwise). Similarly, for WAL, we should be
able to detect whether an incompatible compression format is used,
without having to invoke a generic compression routine that then fails
in some way. Or adding compression reporting for WAL to xlogdump.
I also don't particularly like baking in the assumption that we don't
support tuples larger than 1GB in further places. To me it seems likely
that we're going to have to fix that, and it's hard enough already... I
know that my patch did that too...
For external datums I suggest encoding the compression method as a
distinct VARTAG_ONDISK_COMPRESSED, and then have that include the
compression method as a field.
For in-line compressed values (so VARATT_IS_4B_C), doing something
roughly like you did, indicating the type of metadata following using
the high bit sounds reasonable. But I think I'd make it so that if the
highbit is set, the struct is instead entirely different, keeping a full
4 byte byte length, and including the compression type header inside the
struct. Perhaps I'd combine the compression type with the high-bit-set
part? So when the high bit is set, it'd be something like
{
int32 vl_len_; /* varlena header (do not touch directly!) */
/*
* Actually only 7 bit, the high bit determines whether this
* is the old compression header (if unset), or this type of header
* (if set).
*/
uint8 type;
/*
* Stored as uint8[4], to avoid unnecessary alignment padding.
*/
uint8[4] length;
char va_data[FLEXIBLE_ARRAY_MEMBER];
}
I think it's worth spending some effort trying to get this right - we'll
be bound by design choices for a while.
It's kinda annoying that toast datums aren't better designed. Creating
them from scratch, I'd make it something like:
1) variable-width integer describing the "physical length", so that
tuple deforming can quickly determine length - all the ifs necessary
to determine lengths are a bottleneck. I'd probably just use a 127bit
encoding, if the high bit is set, there's a following length byte.
2) type of toasted datum, probably also in a variable width encoding,
starting at 1byte. Not that I think it's likely we'd overrun 256
types - but it's cheap enough to just declare the high bit as an
length extension bit.
These are always stored unaligned. So there's no need to deal with
padding bytes having to be zero to determine whether we're dealing with
a 1byte datum etc.
Then, type dependant:
For In-line uncompressed datums
3a) alignment padding, amount determined by 2) above, i.e. we'd just
have different types for different amounts of alignment. Probably
using some heuristic to use unaligned when either dealing with data
that doesn't need alignment, or when the datum is fairly small, so
copying to get the data as unaligned won't be a significant penalty.
4a) data
For in-line compressed datums
3b) compression metadata {varint rawsize, varint compression algorithm}
4b) unaligned compressed data - there's no benefit in keeping it aligned
For External toast for uncompressed data:
3d) {toastrelid, valueid, varint rawsize}
For External toast for compressed data:
3e) {valueid, toastrelid, varint compression_algorithm, varint rawsize, varint extsize}
That'd make it a lot more extensible, easier to understand, faster to
decode in a lot of cases, remove a lot of arbitrary limits. Yes, it'd
increase the header size for small datums to two bytes, but I think
that'd be more than bought back by the other improvements.
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-08-06 07:04 Andres Freund <andres@anarazel.de>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 70+ messages in thread
From: Andres Freund @ 2019-08-06 07:04 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 2019-08-05 16:04:46 +0900, Michael Paquier wrote:
> On Fri, Aug 02, 2019 at 07:52:39PM +0200, Tomas Vondra wrote:
> > On Fri, Aug 02, 2019 at 10:12:58AM -0700, Andres Freund wrote:
> >> Why would they be stuck continuing to *compress* with pglz? As we
> >> fully retoast on write anyway we can just gradually switch over to the
> >> better algorithm. Decompression speed is another story, of course.
> >
> > Hmmm, I don't remember the details of those patches so I didn't realize
> > it allows incremental recompression. If that's possible, that would mean
> > existing systems can start using it. Which is good.
>
> It may become a problem on some platforms though (Windows?), so
> patches to improve either the compression or decompression of pglz are
> not that much crazy as they are still likely going to be used, and for
> read-mostly switching to a new algo may not be worth the extra cost so
> it is not like we are going to drop it completely either.
What's the platform dependency that you're thinking of? And how's
compression speed relevant to "read mostly"? Switching would just
happen whenever tuple fields are changed. And it'll not have an
additional cost, because all it does is reduce the cost of a toast write
that'd otherwise happened with pglz.
> Linking to system libraries would make our maintenance much easier,
> and when it comes to have a copy of something else in the tree we
> would be stuck with more maintenance around it. These tend to rot
> easily.
I don't think it's really our experience that they "rot easily".
> After that comes the case where the compression algo is not
> in the binary across one server to another, in which case we have an
> automatic ERROR in case of a mismatching algo, or FATAL for
> deompression of FPWs at recovery when wal_compression is used.
Huh? That's a failure case that only exists if you don't include it in
the tree (with the option to use an out-of-tree lib)?
Greetings,
Andres Freund
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-04 09:09 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Peter Eisentraut @ 2019-09-04 09:09 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; Gasper Zejn <zejn@owca.info>; +Cc: pgsql-hackers@lists.postgresql.org
On 2019-06-24 10:44, Andrey Borodin wrote:
>> 18 мая 2019 г., в 11:44, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>>
> Hi!
> Here's rebased version of patches.
>
> Best regards, Andrey Borodin.
I think this is the most recent patch for the CF entry
<https://commitfest.postgresql.org/24/2119/;.
What about the two patches? Which one is better?
Have you also considered using memmove() to deal with the overlap issue?
Benchmarks have been posted in this thread. Where is the benchmarking
tool? Should we include that in the source somehow?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-04 09:22 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 2 replies; 70+ messages in thread
From: Andrey Borodin @ 2019-09-04 09:22 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
Hi, Peter! Thanks for looking into this.
> 4 сент. 2019 г., в 14:09, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> написал(а):
>
> On 2019-06-24 10:44, Andrey Borodin wrote:
>>> 18 мая 2019 г., в 11:44, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>>>
>> Hi!
>> Here's rebased version of patches.
>>
>> Best regards, Andrey Borodin.
>
> I think this is the most recent patch for the CF entry
> <https://commitfest.postgresql.org/24/2119/;.
>
> What about the two patches? Which one is better?
On our observations pglz_decompress_hacked.patch is best for most of tested platforms.
Difference is that pglz_decompress_hacked8.patch will not appply optimization if decompressed match is not greater than 8 bytes. This optimization was suggested by Tom, that's why we benchmarked it specifically.
> Have you also considered using memmove() to deal with the overlap issue?
Yes, memmove() resolves ambiguity of copying overlapping regions in a way that is not compatible with pglz. In proposed patch we never copy overlapping regions.
> Benchmarks have been posted in this thread. Where is the benchmarking
> tool? Should we include that in the source somehow?
Benchmarking tool is here [0]. Well, code of the benchmarking tool do not adhere to our standards in some places, we did not consider its inclusion in core.
However, most questionable part of benchmarking is choice of test data. It's about 100Mb of useless WALs, datafile and valuable Shakespeare writings.
Best regards, Andrey Borodin.
[0] https://github.com/x4m/test_pglz
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-04 12:40 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 70+ messages in thread
From: Peter Eisentraut @ 2019-09-04 12:40 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-09-04 11:22, Andrey Borodin wrote:
>> What about the two patches? Which one is better?
> On our observations pglz_decompress_hacked.patch is best for most of tested platforms.
> Difference is that pglz_decompress_hacked8.patch will not appply optimization if decompressed match is not greater than 8 bytes. This optimization was suggested by Tom, that's why we benchmarked it specifically.
The patches attached to the message I was replying to are named
0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch
0001-Use-memcpy-in-pglz-decompression.patch
Are those the same ones?
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-04 12:45 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-09-04 12:45 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
> 4 сент. 2019 г., в 17:40, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> написал(а):
>
> On 2019-09-04 11:22, Andrey Borodin wrote:
>>> What about the two patches? Which one is better?
>> On our observations pglz_decompress_hacked.patch is best for most of tested platforms.
>> Difference is that pglz_decompress_hacked8.patch will not appply optimization if decompressed match is not greater than 8 bytes. This optimization was suggested by Tom, that's why we benchmarked it specifically.
>
> The patches attached to the message I was replying to are named
>
> 0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch
> 0001-Use-memcpy-in-pglz-decompression.patch
>
> Are those the same ones?
Yes. Sorry for this confusion.
The only difference of 0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch is that it fallbacks to byte-loop if len is <= 8.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-12 18:58 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 70+ messages in thread
From: Alvaro Herrera @ 2019-09-12 18:58 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
I just noticed we had two CF items pointing to this thread,
https://commitfest.postgresql.org/24/2119/
https://commitfest.postgresql.org/24/2180/
so I marked the newer one as withdrawn.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-15 10:57 Oleg Bartunov <obartunov@postgrespro.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 70+ messages in thread
From: Oleg Bartunov @ 2019-09-15 10:57 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Wed, Sep 4, 2019 at 12:22 PM Andrey Borodin <x4mmm@yandex-team.ru> wrote:
>
> Hi, Peter! Thanks for looking into this.
>
> > 4 сент. 2019 г., в 14:09, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> написал(а):
> >
> > On 2019-06-24 10:44, Andrey Borodin wrote:
> >>> 18 мая 2019 г., в 11:44, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
> >>>
> >> Hi!
> >> Here's rebased version of patches.
> >>
> >> Best regards, Andrey Borodin.
> >
> > I think this is the most recent patch for the CF entry
> > <https://commitfest.postgresql.org/24/2119/;.
> >
> > What about the two patches? Which one is better?
> On our observations pglz_decompress_hacked.patch is best for most of tested platforms.
> Difference is that pglz_decompress_hacked8.patch will not appply optimization if decompressed match is not greater than 8 bytes. This optimization was suggested by Tom, that's why we benchmarked it specifically.
>
> > Have you also considered using memmove() to deal with the overlap issue?
> Yes, memmove() resolves ambiguity of copying overlapping regions in a way that is not compatible with pglz. In proposed patch we never copy overlapping regions.
>
> > Benchmarks have been posted in this thread. Where is the benchmarking
> > tool? Should we include that in the source somehow?
>
> Benchmarking tool is here [0]. Well, code of the benchmarking tool do not adhere to our standards in some places, we did not consider its inclusion in core.
> However, most questionable part of benchmarking is choice of test data. It's about 100Mb of useless WALs, datafile and valuable Shakespeare writings.
Why not use 'Silesia compression corpus'
(http://sun.aei.polsl.pl/~sdeor/index.php?page=silesia), which used by
lzbench (https://github.com/inikep/lzbench) ? I and Teodor remember
that testing on non-english texts could be very important.
>
> Best regards, Andrey Borodin.
>
>
> [0] https://github.com/x4m/test_pglz
>
>
>
--
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-25 21:24 Petr Jelinek <petr@2ndquadrant.com>
parent: Andres Freund <andres@anarazel.de>
0 siblings, 0 replies; 70+ messages in thread
From: Petr Jelinek @ 2019-09-25 21:24 UTC (permalink / raw)
To: Andres Freund <andres@anarazel.de>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi,
On 05/08/2019 09:26, Andres Freund wrote:
> Hi,
>
> On 2019-08-05 00:19:28 +0200, Petr Jelinek wrote:
>> It carries that information inside the compressed value, like I said in the
>> other reply, that's why the extra byte.
>
> I'm not convinced that that is a good plan - imo the reference to the
> compressed data should carry that information.
>
> I.e. in the case of toast, at least toast pointers should hold enough
> information to determine the compression algorithm. And in the case of
> WAL, the WAL record should contain that.
>
Point taken.
>
> For external datums I suggest encoding the compression method as a
> distinct VARTAG_ONDISK_COMPRESSED, and then have that include the
> compression method as a field.
So the new reads/writes will use this and reads of old format won't
change? Sounds fine.
>
> For in-line compressed values (so VARATT_IS_4B_C), doing something
> roughly like you did, indicating the type of metadata following using
> the high bit sounds reasonable. But I think I'd make it so that if the
> highbit is set, the struct is instead entirely different, keeping a full
> 4 byte byte length, and including the compression type header inside the
> struct. Perhaps I'd combine the compression type with the high-bit-set
> part? So when the high bit is set, it'd be something like
>
> {
> int32 vl_len_; /* varlena header (do not touch directly!) */
>
> /*
> * Actually only 7 bit, the high bit determines whether this
> * is the old compression header (if unset), or this type of header
> * (if set).
> */
> uint8 type;
>
> /*
> * Stored as uint8[4], to avoid unnecessary alignment padding.
> */
> uint8[4] length;
>
> char va_data[FLEXIBLE_ARRAY_MEMBER];
> }
>
Won't this break BW compatibility on big-endian (if I understand
corretly what you are trying to achieve here)?
> I think it's worth spending some effort trying to get this right - we'll
> be bound by design choices for a while.
>
Sure, however I am not in business of redesigning TOAST from scratch
right now, even if I do agree that the current header is far from ideal.
--
Petr Jelinek
2ndQuadrant - PostgreSQL Solutions for the Enterprise
https://www.2ndQuadrant.com/
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-27 11:41 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Peter Eisentraut @ 2019-09-27 11:41 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-09-04 14:45, Andrey Borodin wrote:
>> On 2019-09-04 11:22, Andrey Borodin wrote:
>>>> What about the two patches? Which one is better?
>>> On our observations pglz_decompress_hacked.patch is best for most of tested platforms.
>>> Difference is that pglz_decompress_hacked8.patch will not appply optimization if decompressed match is not greater than 8 bytes. This optimization was suggested by Tom, that's why we benchmarked it specifically.
>>
>> The patches attached to the message I was replying to are named
>>
>> 0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch
>> 0001-Use-memcpy-in-pglz-decompression.patch
>>
>> Are those the same ones?
>
> Yes. Sorry for this confusion.
>
> The only difference of 0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch is that it fallbacks to byte-loop if len is <= 8.
After reviewing this thread and more testing, I think
0001-Use-memcpy-in-pglz-decompression.patch appears to be a solid win
and we should move ahead with it.
I don't, however, fully understand the code changes, and I think this
could use more and better comments. In particular, I wonder about
off *= 2;
This is new logic that isn't explained anywhere.
This whole function is commented a bit strangely. It begins with
"Otherwise", but there is nothing before it. And what does "from OUTPUT
to OUTPUT" mean? There is no "output" variable. We should make this
match the code better.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-09-28 08:29 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-09-28 08:29 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
Oleg, Peter, thanks for looking into this!
I hope to benchmark decompression on Silesian corpus soon.
PFA v2 with better comments.
> 27 сент. 2019 г., в 14:41, Peter Eisentraut <peter.eisentraut@2ndquadrant.com> написал(а):
>
> After reviewing this thread and more testing, I think
> 0001-Use-memcpy-in-pglz-decompression.patch appears to be a solid win
> and we should move ahead with it.
>
> I don't, however, fully understand the code changes, and I think this
> could use more and better comments. In particular, I wonder about
>
> off *= 2;
I've changed this to
off += off;
>
> This is new logic that isn't explained anywhere.
>
> This whole function is commented a bit strangely. It begins with
> "Otherwise", but there is nothing before it. And what does "from OUTPUT
> to OUTPUT" mean? There is no "output" variable. We should make this
> match the code better.
I've added small example to illustrate what is going on.
Thanks!
--
Andrey Borodin
Open source RDBMS development team leader
Yandex.Cloud
Attachments:
v2-0001-Use-memcpy-in-pglz-decompression.patch (application/octet-stream, 3.2K) download
From 74bff8b4f8c9f9ab34d7dc8887ac27fc7ab56c2b Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 24 Jun 2019 13:36:47 +0500
Subject: [PATCH] Use memcpy in pglz decompression
Previously pglz was using byte loop during decompression.
This commit is employing memcpy for this purpose.
---
src/common/pg_lzcompress.c | 47 +++++++++++++++++++++++++++-----------
1 file changed, 34 insertions(+), 13 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 988b3987d0..563144a3ad 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -714,11 +714,13 @@ pglz_decompress(const char *source, int32 slen, char *dest,
if (ctrl & 1)
{
/*
- * Otherwise it contains the match length minus 3 and the
- * upper 4 bits of the offset. The next following byte
- * contains the lower 8 bits of the offset. If the length is
- * coded as 18, another extension tag byte tells how much
- * longer the match really was (0-255).
+ * Set control bit means we must read a match tag.
+ * The match is coded with two bytes. First byte use lower
+ * nibble to code length - 3. Higher nibble contain upper 4
+ * bits of the offset. The next following byte contains the
+ * lower 8 bits of the offset. If the length is coded as 18,
+ * another extension tag byte tells how much longer the match
+ * really was (0-255).
*/
int32 len;
int32 off;
@@ -731,22 +733,41 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT (copy len bytes from dp - off to dp). The copied
+ * areas could overlap, to preven possible uncertanity, we copy
+ * only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ while (off < len)
{
- *dp = dp[-off];
- dp++;
+ /*
+ * When offset is smaller than lengh - source and
+ * destination regions overlap. memmove() is resolving this
+ * overlap in an incompatible way with pglz. Thus we resort
+ * to memcpy()-ing non-overlapping regions.
+ * Consider input: 112341234123412341234
+ * At byte 5 here ^ we have match with length 16 and
+ * offset 4. 11234M(len=16, off=4)
+ * We are decoding first period of match and rewrite match
+ * 112341234M(len=12, off=8)
+ * Same match is now at position 9, it aims to same start
+ * byte of output, but from another position: offset is
+ * doubled. We iterate through this offset growth until we
+ * can proceed to usual memcpy().
+ */
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp += off;
+ off += off;
}
+ memcpy(dp, dp - off, len);
+ dp+=len;
}
else
{
/*
- * An unset control bit means LITERAL BYTE. So we just copy
- * one from INPUT to OUTPUT.
+ * An unset control bit means LITERAL BYTE. So we just
+ * copy one from INPUT to OUTPUT.
*/
*dp++ = *sp++;
}
--
2.20.1
=
* Re: pglz performance
@ 2019-10-21 09:09 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-10-21 09:09 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
> 28 сент. 2019 г., в 10:29, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> I hope to benchmark decompression on Silesian corpus soon.
I've done it. And results are quite controversial.
Dataset adds 12 payloads to our 5. Payloads have relatively high entropy. In many cases pglz cannot compress them at all, so decompression is nop, data is stored as is.
Decompressor pglz_decompress_hacked result 48.281747
Decompressor pglz_decompress_hacked8 result 33.868779
Decompressor pglz_decompress_vanilla result 42.510165
Tested on Intel(R) Xeon(R) CPU E5-2650 v2 @ 2.60GHz
With Silesian corpus pglz_decompress_hacked is actually decreasing performance on high-entropy data.
Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress.
In spite of this benchmarks, I think that pglz_decompress_hacked8 is safer option.
I've updated test suite [0] and anyone interested can verify benchmarks.
--
Andrey Borodin
Open source RDBMS development team leader
Yandex.Cloud
[0] https://github.com/x4m/test_pglz
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-10-25 05:05 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-10-25 05:05 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
> 21 окт. 2019 г., в 14:09, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>
> With Silesian corpus pglz_decompress_hacked is actually decreasing performance on high-entropy data.
> Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress.
> In spite of this benchmarks, I think that pglz_decompress_hacked8 is safer option.
Here's v3 which takes into account recent benchmarks with Silesian Corpus and have better comments.
Thanks!
--
Andrey Borodin
Open source RDBMS development team leader
Yandex.Cloud
Attachments:
v3-0001-Use-memcpy-in-pglz-decompression.patch (application/octet-stream, 3.4K) download
From 7c4d479cd9331335322d42a4471356c4a9927718 Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 24 Jun 2019 13:36:47 +0500
Subject: [PATCH] Use memcpy in pglz decompression
Previously pglz was using byte loop during decompression.
This commit is employing memcpy for this purpose if match length
is greater than 8 bytes.
---
src/common/pg_lzcompress.c | 59 +++++++++++++++++++++++++++++---------
1 file changed, 46 insertions(+), 13 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 3adad62d9a..c88a921788 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -714,11 +714,13 @@ pglz_decompress(const char *source, int32 slen, char *dest,
if (ctrl & 1)
{
/*
- * Otherwise it contains the match length minus 3 and the
- * upper 4 bits of the offset. The next following byte
- * contains the lower 8 bits of the offset. If the length is
- * coded as 18, another extension tag byte tells how much
- * longer the match really was (0-255).
+ * Set control bit means we must read a match tag.
+ * The match is coded with two bytes. First byte use lower
+ * nibble to code length - 3. Higher nibble contain upper 4
+ * bits of the offset. The next following byte contains the
+ * lower 8 bits of the offset. If the length is coded as 18,
+ * another extension tag byte tells how much longer the match
+ * really was (0-255).
*/
int32 len;
int32 off;
@@ -731,22 +733,53 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT (copy len bytes from dp - off to dp). The copied
+ * areas could overlap, to preven possible uncertanity, we copy
+ * only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ if (len <= 8)
{
- *dp = dp[-off];
- dp++;
+ /* use byte-loop for small matches */
+ while (len--)
+ {
+ *dp = dp[-off];
+ dp++;
+ }
+ }
+ else
+ {
+ while (off < len)
+ {
+ /*
+ * When offset is smaller than lengh - source and
+ * destination regions overlap. memmove() is resolving this
+ * overlap in an incompatible way with pglz. Thus we resort
+ * to memcpy()-ing non-overlapping regions.
+ * Consider input: 112341234123412341234
+ * At byte 5 here ^ we have match with length 16 and
+ * offset 4. 11234M(len=16, off=4)
+ * We are decoding first period of match and rewrite match
+ * 112341234M(len=12, off=8)
+ * Same match is now at position 9, it aims to same start
+ * byte of output, but from another position: offset is
+ * doubled. We iterate through this offset growth until we
+ * can proceed to usual memcpy().
+ */
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp += off;
+ off += off;
+ }
+ memcpy(dp, dp - off, len);
+ dp+=len;
}
}
else
{
/*
- * An unset control bit means LITERAL BYTE. So we just copy
- * one from INPUT to OUTPUT.
+ * An unset control bit means LITERAL BYTE. So we just
+ * copy one from INPUT to OUTPUT.
*/
*dp++ = *sp++;
}
--
2.20.1
=
* Re: pglz performance
@ 2019-11-01 12:33 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Peter Eisentraut @ 2019-11-01 12:33 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-10-25 07:05, Andrey Borodin wrote:
>> 21 окт. 2019 г., в 14:09, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>>
>> With Silesian corpus pglz_decompress_hacked is actually decreasing performance on high-entropy data.
>> Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress.
>> In spite of this benchmarks, I think that pglz_decompress_hacked8 is safer option.
>
> Here's v3 which takes into account recent benchmarks with Silesian Corpus and have better comments.
Your message from 21 October appears to say that this change makes the
performance worse. So I don't know how to proceed with this.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-01 15:48 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 2 replies; 70+ messages in thread
From: Alvaro Herrera @ 2019-11-01 15:48 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-Nov-01, Peter Eisentraut wrote:
> On 2019-10-25 07:05, Andrey Borodin wrote:
> > > 21 окт. 2019 г., в 14:09, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
> > >
> > > With Silesian corpus pglz_decompress_hacked is actually decreasing performance on high-entropy data.
> > > Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress.
> > > In spite of this benchmarks, I think that pglz_decompress_hacked8 is safer option.
> >
> > Here's v3 which takes into account recent benchmarks with Silesian Corpus and have better comments.
>
> Your message from 21 October appears to say that this change makes the
> performance worse. So I don't know how to proceed with this.
As I understand that report, in these results "less is better", so the
hacked8 variant shows better performance (33.8) than current (42.5).
The "hacked" variant shows worse performance (48.2) that the current
code. The "in spite" phrase seems to have been a mistake.
I am surprised that there is so much variability in the performance
numbers, though, based on such small tweaks of the code.
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-01 16:59 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 2 replies; 70+ messages in thread
From: Tomas Vondra @ 2019-11-01 16:59 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Fri, Nov 01, 2019 at 12:48:28PM -0300, Alvaro Herrera wrote:
>On 2019-Nov-01, Peter Eisentraut wrote:
>
>> On 2019-10-25 07:05, Andrey Borodin wrote:
>> > > 21 окт. 2019 г., в 14:09, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>> > >
>> > > With Silesian corpus pglz_decompress_hacked is actually decreasing performance on high-entropy data.
>> > > Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress.
>> > > In spite of this benchmarks, I think that pglz_decompress_hacked8 is safer option.
>> >
>> > Here's v3 which takes into account recent benchmarks with Silesian Corpus and have better comments.
>>
>> Your message from 21 October appears to say that this change makes the
>> performance worse. So I don't know how to proceed with this.
>
>As I understand that report, in these results "less is better", so the
>hacked8 variant shows better performance (33.8) than current (42.5).
>The "hacked" variant shows worse performance (48.2) that the current
>code. The "in spite" phrase seems to have been a mistake.
>
>I am surprised that there is so much variability in the performance
>numbers, though, based on such small tweaks of the code.
>
I'd try running the benchmarks to verify the numbers, and maybe do some
additional tests, but it's not clear to me which patches should I use.
I think the last patches with 'hacked' and 'hacked8' in the name are a
couple of months old, and the recent posts attach just a single patch.
Andrey, can you post current versions of both patches?
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-02 11:30 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-11-02 11:30 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
> 1 нояб. 2019 г., в 18:48, Alvaro Herrera <alvherre@2ndquadrant.com> написал(а):
>
> On 2019-Nov-01, Peter Eisentraut wrote:
>
>> On 2019-10-25 07:05, Andrey Borodin wrote:
>>>> 21 окт. 2019 г., в 14:09, Andrey Borodin <x4mmm@yandex-team.ru> написал(а):
>>>>
>>>> With Silesian corpus pglz_decompress_hacked is actually decreasing performance on high-entropy data.
>>>> Meanwhile pglz_decompress_hacked8 is still faster than usual pglz_decompress.
>>>> In spite of this benchmarks, I think that pglz_decompress_hacked8 is safer option.
>>>
>>> Here's v3 which takes into account recent benchmarks with Silesian Corpus and have better comments.
>>
>> Your message from 21 October appears to say that this change makes the
>> performance worse. So I don't know how to proceed with this.
>
> As I understand that report, in these results "less is better", so the
> hacked8 variant shows better performance (33.8) than current (42.5).
> The "hacked" variant shows worse performance (48.2) that the current
> code.
This is correct. Thanks, Álvaro.
> The "in spite" phrase seems to have been a mistake.
Yes. Sorry, I actually thought that "in spite" is a contradiction of "despite" and means "In view of".
> I am surprised that there is so much variability in the performance
> numbers, though, based on such small tweaks of the code.
Silesian Corpus is very different from WALs and PG data files. Data files are rich in long sequences of same byte. This sequences are long, thus unrolled very effectively by memcpy method.
But Silesian corpus is rich in short matches of few bytes.
> 1 нояб. 2019 г., в 19:59, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
> I'd try running the benchmarks to verify the numbers, and maybe do some
> additional tests, but it's not clear to me which patches should I use.
Cool, thanks!
> I think the last patches with 'hacked' and 'hacked8' in the name are a
> couple of months old, and the recent posts attach just a single patch.
> Andrey, can you post current versions of both patches?
PFA two patches:
v4-0001-Use-memcpy-in-pglz-decompression.patch (known as 'hacked' in test_pglz extension)
v4-0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch (known as 'hacked8')
Best regards, Andrey Borodin.
Attachments: download all (2 files, .tar.gz)
v4-0001-Use-memcpy-in-pglz-decompression.patch (application/octet-stream, 3.2K) download
From 74bff8b4f8c9f9ab34d7dc8887ac27fc7ab56c2b Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 24 Jun 2019 13:36:47 +0500
Subject: [PATCH] Use memcpy in pglz decompression
Previously pglz was using byte loop during decompression.
This commit is employing memcpy for this purpose.
---
src/common/pg_lzcompress.c | 47 +++++++++++++++++++++++++++-----------
1 file changed, 34 insertions(+), 13 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 988b3987d0..563144a3ad 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -714,11 +714,13 @@ pglz_decompress(const char *source, int32 slen, char *dest,
if (ctrl & 1)
{
/*
- * Otherwise it contains the match length minus 3 and the
- * upper 4 bits of the offset. The next following byte
- * contains the lower 8 bits of the offset. If the length is
- * coded as 18, another extension tag byte tells how much
- * longer the match really was (0-255).
+ * Set control bit means we must read a match tag.
+ * The match is coded with two bytes. First byte use lower
+ * nibble to code length - 3. Higher nibble contain upper 4
+ * bits of the offset. The next following byte contains the
+ * lower 8 bits of the offset. If the length is coded as 18,
+ * another extension tag byte tells how much longer the match
+ * really was (0-255).
*/
int32 len;
int32 off;
@@ -731,22 +733,41 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT (copy len bytes from dp - off to dp). The copied
+ * areas could overlap, to preven possible uncertanity, we copy
+ * only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ while (off < len)
{
- *dp = dp[-off];
- dp++;
+ /*
+ * When offset is smaller than lengh - source and
+ * destination regions overlap. memmove() is resolving this
+ * overlap in an incompatible way with pglz. Thus we resort
+ * to memcpy()-ing non-overlapping regions.
+ * Consider input: 112341234123412341234
+ * At byte 5 here ^ we have match with length 16 and
+ * offset 4. 11234M(len=16, off=4)
+ * We are decoding first period of match and rewrite match
+ * 112341234M(len=12, off=8)
+ * Same match is now at position 9, it aims to same start
+ * byte of output, but from another position: offset is
+ * doubled. We iterate through this offset growth until we
+ * can proceed to usual memcpy().
+ */
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp += off;
+ off += off;
}
+ memcpy(dp, dp - off, len);
+ dp+=len;
}
else
{
/*
- * An unset control bit means LITERAL BYTE. So we just copy
- * one from INPUT to OUTPUT.
+ * An unset control bit means LITERAL BYTE. So we just
+ * copy one from INPUT to OUTPUT.
*/
*dp++ = *sp++;
}
--
2.20.1
=
v4-0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch (application/octet-stream, 3.4K) download
From 7c4d479cd9331335322d42a4471356c4a9927718 Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 24 Jun 2019 13:36:47 +0500
Subject: [PATCH] Use memcpy in pglz decompression
Previously pglz was using byte loop during decompression.
This commit is employing memcpy for this purpose if match length
is greater than 8 bytes.
---
src/common/pg_lzcompress.c | 59 +++++++++++++++++++++++++++++---------
1 file changed, 46 insertions(+), 13 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 3adad62d9a..c88a921788 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -714,11 +714,13 @@ pglz_decompress(const char *source, int32 slen, char *dest,
if (ctrl & 1)
{
/*
- * Otherwise it contains the match length minus 3 and the
- * upper 4 bits of the offset. The next following byte
- * contains the lower 8 bits of the offset. If the length is
- * coded as 18, another extension tag byte tells how much
- * longer the match really was (0-255).
+ * Set control bit means we must read a match tag.
+ * The match is coded with two bytes. First byte use lower
+ * nibble to code length - 3. Higher nibble contain upper 4
+ * bits of the offset. The next following byte contains the
+ * lower 8 bits of the offset. If the length is coded as 18,
+ * another extension tag byte tells how much longer the match
+ * really was (0-255).
*/
int32 len;
int32 off;
@@ -731,22 +733,53 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT (copy len bytes from dp - off to dp). The copied
+ * areas could overlap, to preven possible uncertanity, we copy
+ * only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ if (len <= 8)
{
- *dp = dp[-off];
- dp++;
+ /* use byte-loop for small matches */
+ while (len--)
+ {
+ *dp = dp[-off];
+ dp++;
+ }
+ }
+ else
+ {
+ while (off < len)
+ {
+ /*
+ * When offset is smaller than lengh - source and
+ * destination regions overlap. memmove() is resolving this
+ * overlap in an incompatible way with pglz. Thus we resort
+ * to memcpy()-ing non-overlapping regions.
+ * Consider input: 112341234123412341234
+ * At byte 5 here ^ we have match with length 16 and
+ * offset 4. 11234M(len=16, off=4)
+ * We are decoding first period of match and rewrite match
+ * 112341234M(len=12, off=8)
+ * Same match is now at position 9, it aims to same start
+ * byte of output, but from another position: offset is
+ * doubled. We iterate through this offset growth until we
+ * can proceed to usual memcpy().
+ */
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp += off;
+ off += off;
+ }
+ memcpy(dp, dp - off, len);
+ dp+=len;
}
}
else
{
/*
- * An unset control bit means LITERAL BYTE. So we just copy
- * one from INPUT to OUTPUT.
+ * An unset control bit means LITERAL BYTE. So we just
+ * copy one from INPUT to OUTPUT.
*/
*dp++ = *sp++;
}
--
2.20.1
=
* Re: pglz performance
@ 2019-11-03 09:24 Tels <nospam-pg-abuse@bloodgate.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Tels @ 2019-11-03 09:24 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
Hello Andrey,
On 2019-11-02 12:30, Andrey Borodin wrote:
>> 1 нояб. 2019 г., в 18:48, Alvaro Herrera <alvherre@2ndquadrant.com>
>> написал(а):
> PFA two patches:
> v4-0001-Use-memcpy-in-pglz-decompression.patch (known as 'hacked' in
> test_pglz extension)
> v4-0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch (known
> as 'hacked8')
Looking at the patches, it seems only the case of a match is changed.
But when we observe a literal byte, this is copied byte-by-byte with:
else
{
* An unset control bit means LITERAL BYTE. So we just
* copy one from INPUT to OUTPUT.
*/
*dp++ = *sp++;
}
Maybe we can optimize this, too. For instance, you could just increase a
counter:
else
{
/*
* An unset control bit means LITERAL BYTE. We count
* these and copy them later.
*/
literal_bytes ++;
}
and in the case of:
if (ctrl & 1)
{
/* First copy all the literal bytes */
if (literal_bytes > 0)
{
memcpy( sp, dp, literal_bytes);
sp += literal_bytes;
dp += literal_bytes;
literal_bytes = 0;
}
(Code untested!)
The same would need to be done at the very end, if the input ends
without any new CTRL-byte.
Wether that gains us anything depends on how common literal bytes are.
It might be that highly compressible input has almost none, while input
that is a mix of incompressible strings and compressible ones might have
longer stretches. One example would be something like an SHA-256, that
is repeated twice. The first instance would be incompressible, the
second one would be just a copy. This might not happens that often in
practical inputs, though.
I wonder if you agree and what would happen if you try this variant on
your corpus tests.
Best regards,
Tels
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-04 07:14 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Tels <nospam-pg-abuse@bloodgate.com>
0 siblings, 0 replies; 70+ messages in thread
From: Andrey Borodin @ 2019-11-04 07:14 UTC (permalink / raw)
To: Tels <nospam-pg-abuse@bloodgate.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
Hi Tels!
Thanks for your interest in fast decompression.
> 3 нояб. 2019 г., в 12:24, Tels <nospam-pg-abuse@bloodgate.com> написал(а):
>
> I wonder if you agree and what would happen if you try this variant on your corpus tests.
I've tried some different optimization for literals. For example loop unrolling[0] and literals bulk-copying.
This approaches were brining some performance improvement. But with noise. Statistically they were somewhere better, somewhere worse, net win, but that "net win" depends on what we consider important data and important platform.
Proposed patch makes clearly decompression faster on any dataset, and platform.
I believe improving pglz further is viable, but optimizations like common data prefix seems more promising to me.
Also, I think we actually need real codecs like lz4, zstd and brotli instead of our own invented wheel.
If you have some spare time - Pull Requests to test_pglz are welcome, lets benchmark more micro optimizations, it brings a lot of fun :)
--
Andrey Borodin
Open source RDBMS development team leader
Yandex.Cloud
[0] https://github.com/x4m/test_pglz/blob/master/pg_lzcompress_hacked.c#L166
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-06 08:03 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 0 replies; 70+ messages in thread
From: Peter Eisentraut @ 2019-11-06 08:03 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-11-01 16:48, Alvaro Herrera wrote:
> As I understand that report, in these results "less is better", so the
> hacked8 variant shows better performance (33.8) than current (42.5).
> The "hacked" variant shows worse performance (48.2) that the current
> code.
Which appears to be the opposite of, or at least inconsistent with,
results earlier in the thread.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-06 08:04 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 70+ messages in thread
From: Peter Eisentraut @ 2019-11-06 08:04 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-11-01 17:59, Tomas Vondra wrote:
> I'd try running the benchmarks to verify the numbers, and maybe do some
> additional tests, but it's not clear to me which patches should I use.
>
> I think the last patches with 'hacked' and 'hacked8' in the name are a
> couple of months old, and the recent posts attach just a single patch.
> Andrey, can you post current versions of both patches?
OK, waiting on some independent verification of benchmark numbers.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-25 08:03 Michael Paquier <michael@paquier.xyz>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Michael Paquier @ 2019-11-25 08:03 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Wed, Nov 06, 2019 at 09:04:25AM +0100, Peter Eisentraut wrote:
> OK, waiting on some independent verification of benchmark numbers.
Still waiting for these after 19 days, so the patch has been marked as
returned with feedback.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-25 08:21 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-11-25 08:21 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
> 25 нояб. 2019 г., в 13:03, Michael Paquier <michael@paquier.xyz> написал(а):
>
> On Wed, Nov 06, 2019 at 09:04:25AM +0100, Peter Eisentraut wrote:
>> OK, waiting on some independent verification of benchmark numbers.
>
> Still waiting for these after 19 days, so the patch has been marked as
> returned with feedback.
I think status Needs Review describes what is going on better. It's not like something is awaited from my side.
Thanks.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-25 08:29 Michael Paquier <michael@paquier.xyz>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 1 reply; 70+ messages in thread
From: Michael Paquier @ 2019-11-25 08:29 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Tomas Vondra <tomas.vondra@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote:
> I think status Needs Review describes what is going on better. It's
> not like something is awaited from my side.
Indeed. You are right so I have moved the patch instead, with "Needs
review". The patch status was actually incorrect in the CF app, as it
was marked as waiting on author.
@Tomas: updated versions of the patches have been sent by Andrey.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-26 09:43 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 2 replies; 70+ messages in thread
From: Tomas Vondra @ 2019-11-26 09:43 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Mon, Nov 25, 2019 at 05:29:40PM +0900, Michael Paquier wrote:
>On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote:
>> I think status Needs Review describes what is going on better. It's
>> not like something is awaited from my side.
>
>Indeed. You are right so I have moved the patch instead, with "Needs
>review". The patch status was actually incorrect in the CF app, as it
>was marked as waiting on author.
>
>@Tomas: updated versions of the patches have been sent by Andrey.
I've done benchmarks on the two last patches, using the data sets from
test_pglz repository [1], but using three simple queries:
1) prefix - first 100 bytes of the value
SELECT length(substr(value, 0, 100)) FROM t
2) infix - 100 bytes from the middle
SELECT length(substr(value, test_length/2, 100)) FROM t
3) suffix - last 100 bytes
SELECT length(substr(value, test_length - 100, 100)) FROM t
See the two attached scripts, implementing this benchmark. The test
itself did a 60-second pgbench runs (single client) measuring tps on two
different machines.
patch 1: v4-0001-Use-memcpy-in-pglz-decompression.patch
patch 2: v4-0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch
The results (compared to master) from the first machine (i5-2500k CPU)
look like this:
patch 1 | patch 2
dataset prefix infix suffix | prefix infix suffix
-------------------------------------------------------------------------
000000010000000000000001 99% 134% 161% | 100% 126% 152%
000000010000000000000006 99% 260% 287% | 100% 257% 279%
000000010000000000000008 100% 100% 100% | 100% 95% 91%
16398 100% 168% 221% | 100% 159% 215%
shakespeare.txt 100% 138% 141% | 100% 116% 117%
mr 99% 120% 128% | 100% 107% 108%
dickens 100% 129% 132% | 100% 100% 100%
mozilla 100% 119% 120% | 100% 102% 104%
nci 100% 149% 141% | 100% 143% 135%
ooffice 99% 121% 123% | 100% 97% 98%
osdb 100% 99% 99% | 100% 100% 99%
reymont 99% 130% 132% | 100% 106% 107%
samba 100% 126% 132% | 100% 105% 111%
sao 100% 100% 99% | 100% 100% 100%
webster 100% 127% 127% | 100% 106% 106%
x-ray 99% 99% 99% | 100% 100% 100%
xml 100% 144% 144% | 100% 130% 128%
and on the other one (xeon e5-2620v4) looks like this:
patch 1 | patch 2
dataset prefix infix suffix | prefix infix suffix
------------------------------------------------------------------------
000000010000000000000001 98% 147% 170% | 98% 132% 159%
000000010000000000000006 100% 340% 314% | 98% 334% 355%
000000010000000000000008 99% 100% 105% | 99% 99% 101%
16398 101% 153% 205% | 99% 148% 201%
shakespeare.txt 100% 147% 149% | 99% 117% 118%
mr 100% 131% 139% | 99% 112% 108%
dickens 100% 143% 143% | 99% 103% 102%
mozilla 100% 122% 122% | 99% 105% 106%
nci 100% 151% 135% | 100% 135% 125%
ooffice 99% 127% 129% | 98% 101% 102%
osdb 102% 100% 101% | 102% 100% 99%
reymont 101% 142% 143% | 100% 108% 108%
samba 100% 132% 136% | 99% 109% 112%
sao 99% 101% 100% | 99% 100% 100%
webster 100% 132% 129% | 100% 106% 106%
x-ray 99% 101% 100% | 90% 101% 101%
xml 100% 147% 148% | 100% 127% 125%
In general, I think the results for both patches seem clearly a win, but
maybe patch 1 is bit better, especially on the newer (xeon) CPU. So I'd
probably go with that one.
[1] https://github.com/x4m/test_pglz
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
Attachments:
pglz-test.sh (application/x-sh, 766B) download
pglz-load.sql (application/sql, 2.1K) download
pglz.ods (application/vnd.oasis.opendocument.spreadsheet, 22.8K) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-26 19:17 Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 70+ messages in thread
From: Peter Eisentraut @ 2019-11-26 19:17 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-11-26 10:43, Tomas Vondra wrote:
> In general, I think the results for both patches seem clearly a win, but
> maybe patch 1 is bit better, especially on the newer (xeon) CPU. So I'd
> probably go with that one.
Patch 1 is also the simpler patch, so it seems clearly preferable.
--
Peter Eisentraut http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-26 20:05 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Tomas Vondra @ 2019-11-26 20:05 UTC (permalink / raw)
To: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Andrey Borodin <x4mmm@yandex-team.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Tue, Nov 26, 2019 at 08:17:13PM +0100, Peter Eisentraut wrote:
>On 2019-11-26 10:43, Tomas Vondra wrote:
>>In general, I think the results for both patches seem clearly a win, but
>>maybe patch 1 is bit better, especially on the newer (xeon) CPU. So I'd
>>probably go with that one.
>
>Patch 1 is also the simpler patch, so it seems clearly preferable.
>
Yeah, although the difference is minimal. We could probably construct a
benchmark where #2 wins, but I think these queries are fairly realistic.
So I'd just go with #1.
Code-wise I think the patches are mostly fine, although the comments
might need some proof-reading.
1) I wasn't really sure what a "nibble" is, but maybe it's just me and
it's a well-known term.
2) First byte use lower -> First byte uses lower
3) nibble contain upper -> nibble contains upper
4) to preven possible uncertanity -> to prevent possible uncertainty
5) I think we should briefly explain why memmove would be incompatible
with pglz, it's not quite clear to me.
6) I'm pretty sure the comment in the 'while (off < len)' branch will be
badly mangled by pgindent.
7) The last change moving "copy" to the next line seems unnecessary.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-27 08:01 Michael Paquier <michael@paquier.xyz>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Michael Paquier @ 2019-11-27 08:01 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Tue, Nov 26, 2019 at 09:05:59PM +0100, Tomas Vondra wrote:
> Yeah, although the difference is minimal. We could probably construct a
> benchmark where #2 wins, but I think these queries are fairly realistic.
> So I'd just go with #1.
Nice results. Using your benchmarks it indeed looks like patch 1 is a
winner here.
> Code-wise I think the patches are mostly fine, although the comments
> might need some proof-reading.
>
> 1) I wasn't really sure what a "nibble" is, but maybe it's just me and
> it's a well-known term.
>
> 2) First byte use lower -> First byte uses lower
>
> 3) nibble contain upper -> nibble contains upper
>
> 4) to preven possible uncertanity -> to prevent possible uncertainty
>
> 5) I think we should briefly explain why memmove would be incompatible
> with pglz, it's not quite clear to me.
>
> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be
> badly mangled by pgindent.
>
> 7) The last change moving "copy" to the next line seems unnecessary.
Patch 1 has a typo as well here:
+ * When offset is smaller than lengh - source and
s/lengh/length/
Okay, if we are reaching a conclusion here, Tomas or Peter, are you
planning to finish brushing the patch and potentially commit it?
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-27 09:42 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Michael Paquier <michael@paquier.xyz>
0 siblings, 0 replies; 70+ messages in thread
From: Tomas Vondra @ 2019-11-27 09:42 UTC (permalink / raw)
To: Michael Paquier <michael@paquier.xyz>; +Cc: Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Andrey Borodin <x4mmm@yandex-team.ru>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Wed, Nov 27, 2019 at 05:01:47PM +0900, Michael Paquier wrote:
>On Tue, Nov 26, 2019 at 09:05:59PM +0100, Tomas Vondra wrote:
>> Yeah, although the difference is minimal. We could probably construct a
>> benchmark where #2 wins, but I think these queries are fairly realistic.
>> So I'd just go with #1.
>
>Nice results. Using your benchmarks it indeed looks like patch 1 is a
>winner here.
>
>> Code-wise I think the patches are mostly fine, although the comments
>> might need some proof-reading.
>>
>> 1) I wasn't really sure what a "nibble" is, but maybe it's just me and
>> it's a well-known term.
>>
>> 2) First byte use lower -> First byte uses lower
>>
>> 3) nibble contain upper -> nibble contains upper
>>
>> 4) to preven possible uncertanity -> to prevent possible uncertainty
>>
>> 5) I think we should briefly explain why memmove would be incompatible
>> with pglz, it's not quite clear to me.
>>
>> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be
>> badly mangled by pgindent.
>>
>> 7) The last change moving "copy" to the next line seems unnecessary.
>
>Patch 1 has a typo as well here:
>+ * When offset is smaller than lengh - source and
>s/lengh/length/
>
>Okay, if we are reaching a conclusion here, Tomas or Peter, are you
>planning to finish brushing the patch and potentially commit it?
I'd like some feedback from Andrey regarding the results and memmove
comment, ant then I'll polish and push.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-27 12:47 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-11-27 12:47 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
Hi Tomas!
Thanks for benchmarking this!
> 26 нояб. 2019 г., в 14:43, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>
> On Mon, Nov 25, 2019 at 05:29:40PM +0900, Michael Paquier wrote:
>> On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote:
>>> I think status Needs Review describes what is going on better. It's
>>> not like something is awaited from my side.
>>
>> Indeed. You are right so I have moved the patch instead, with "Needs
>> review". The patch status was actually incorrect in the CF app, as it
>> was marked as waiting on author.
>>
>> @Tomas: updated versions of the patches have been sent by Andrey.
>
> I've done benchmarks on the two last patches, using the data sets from
> test_pglz repository [1], but using three simple queries:
>
> 1) prefix - first 100 bytes of the value
>
> SELECT length(substr(value, 0, 100)) FROM t
>
> 2) infix - 100 bytes from the middle
>
> SELECT length(substr(value, test_length/2, 100)) FROM t
>
> 3) suffix - last 100 bytes
>
> SELECT length(substr(value, test_length - 100, 100)) FROM t
>
> See the two attached scripts, implementing this benchmark. The test
> itself did a 60-second pgbench runs (single client) measuring tps on two
> different machines.
>
> patch 1: v4-0001-Use-memcpy-in-pglz-decompression.patch
> patch 2: v4-0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch
>
> The results (compared to master) from the first machine (i5-2500k CPU)
> look like this:
>
> patch 1 | patch 2
> dataset prefix infix suffix | prefix infix suffix
> -------------------------------------------------------------------------
> 000000010000000000000001 99% 134% 161% | 100% 126% 152%
> 000000010000000000000006 99% 260% 287% | 100% 257% 279%
> 000000010000000000000008 100% 100% 100% | 100% 95% 91%
> 16398 100% 168% 221% | 100% 159% 215%
> shakespeare.txt 100% 138% 141% | 100% 116% 117%
> mr 99% 120% 128% | 100% 107% 108%
> dickens 100% 129% 132% | 100% 100% 100%
> mozilla 100% 119% 120% | 100% 102% 104%
> nci 100% 149% 141% | 100% 143% 135%
> ooffice 99% 121% 123% | 100% 97% 98%
> osdb 100% 99% 99% | 100% 100% 99%
> reymont 99% 130% 132% | 100% 106% 107%
> samba 100% 126% 132% | 100% 105% 111%
> sao 100% 100% 99% | 100% 100% 100%
> webster 100% 127% 127% | 100% 106% 106%
> x-ray 99% 99% 99% | 100% 100% 100%
> xml 100% 144% 144% | 100% 130% 128%
>
> and on the other one (xeon e5-2620v4) looks like this:
>
> patch 1 | patch 2
> dataset prefix infix suffix | prefix infix suffix
> ------------------------------------------------------------------------
> 000000010000000000000001 98% 147% 170% | 98% 132% 159%
> 000000010000000000000006 100% 340% 314% | 98% 334% 355%
> 000000010000000000000008 99% 100% 105% | 99% 99% 101%
> 16398 101% 153% 205% | 99% 148% 201%
> shakespeare.txt 100% 147% 149% | 99% 117% 118%
> mr 100% 131% 139% | 99% 112% 108%
> dickens 100% 143% 143% | 99% 103% 102%
> mozilla 100% 122% 122% | 99% 105% 106%
> nci 100% 151% 135% | 100% 135% 125%
> ooffice 99% 127% 129% | 98% 101% 102%
> osdb 102% 100% 101% | 102% 100% 99%
> reymont 101% 142% 143% | 100% 108% 108%
> samba 100% 132% 136% | 99% 109% 112%
> sao 99% 101% 100% | 99% 100% 100%
> webster 100% 132% 129% | 100% 106% 106%
> x-ray 99% 101% 100% | 90% 101% 101%
> xml 100% 147% 148% | 100% 127% 125%
>
> In general, I think the results for both patches seem clearly a win, but
> maybe patch 1 is bit better, especially on the newer (xeon) CPU. So I'd
> probably go with that one.
From my POV there are two interesting new points in your benchmarks:
1. They are more or lesss end-to-end benchmarks with whole system involved.
2. They provide per-payload breakdown
Prefix experiment is mostly related to reading from page cache and not directly connected with decompression. It's a bit strange that we observe 1% degradation in certain experiments, but I believe it's a noise.
Infix and Suffix results are correlated. We observe no impact of the patch on compressed data.
test_pglz also includes slicing by 2Kb and 8Kb. This was done to imitate toasting. But as far as I understand, in your test data payload will be inserted into toast table too, won't it? If so, I agree that patch 1 looks like a better option.
> 27 нояб. 2019 г., в 1:05, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>
> Code-wise I think the patches are mostly fine, although the comments
> might need some proof-reading.
>
> 1) I wasn't really sure what a "nibble" is, but maybe it's just me and
> it's a well-known term.
I've took the word from pg_lzcompress.c comments
* The offset is in the upper nibble of T1 and in T2.
* The length is in the lower nibble of T1.
>
> 2) First byte use lower -> First byte uses lower
>
> 3) nibble contain upper -> nibble contains upper
>
> 4) to preven possible uncertanity -> to prevent possible uncertainty
>
> 5) I think we should briefly explain why memmove would be incompatible
> with pglz, it's not quite clear to me.
Here's the example
+ * Consider input: 112341234123412341234
+ * At byte 5 here ^ we have match with length 16 and
+ * offset 4. 11234M(len=16, off=4)
If we simply memmove() this 16 bytes we will produce 112341234XXXXXXXXXXXX, where series of X is 12 undefined bytes, that were at bytes [6:18].
>
> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be
> badly mangled by pgindent.
I think I can just write it without line limit and then run pgindent. Will try to do it this evening. Also, I will try to write more about memmove.
>
> 7) The last change moving "copy" to the next line seems unnecessary.
Oh, looks like I had been rewording this comment, and eventually came to the same text..Yes, this change is absolutely unnecessary.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-27 15:28 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 2 replies; 70+ messages in thread
From: Tomas Vondra @ 2019-11-27 15:28 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Wed, Nov 27, 2019 at 05:47:25PM +0500, Andrey Borodin wrote:
>Hi Tomas!
>
>Thanks for benchmarking this!
>
>> 26 нояб. 2019 г., в 14:43, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>>
>> On Mon, Nov 25, 2019 at 05:29:40PM +0900, Michael Paquier wrote:
>>> On Mon, Nov 25, 2019 at 01:21:27PM +0500, Andrey Borodin wrote:
>>>> I think status Needs Review describes what is going on better. It's
>>>> not like something is awaited from my side.
>>>
>>> Indeed. You are right so I have moved the patch instead, with "Needs
>>> review". The patch status was actually incorrect in the CF app, as it
>>> was marked as waiting on author.
>>>
>>> @Tomas: updated versions of the patches have been sent by Andrey.
>>
>> I've done benchmarks on the two last patches, using the data sets from
>> test_pglz repository [1], but using three simple queries:
>>
>> 1) prefix - first 100 bytes of the value
>>
>> SELECT length(substr(value, 0, 100)) FROM t
>>
>> 2) infix - 100 bytes from the middle
>>
>> SELECT length(substr(value, test_length/2, 100)) FROM t
>>
>> 3) suffix - last 100 bytes
>>
>> SELECT length(substr(value, test_length - 100, 100)) FROM t
>>
>> See the two attached scripts, implementing this benchmark. The test
>> itself did a 60-second pgbench runs (single client) measuring tps on two
>> different machines.
>>
>> patch 1: v4-0001-Use-memcpy-in-pglz-decompression.patch
>> patch 2: v4-0001-Use-memcpy-in-pglz-decompression-for-long-matches.patch
>>
>> The results (compared to master) from the first machine (i5-2500k CPU)
>> look like this:
>>
>> patch 1 | patch 2
>> dataset prefix infix suffix | prefix infix suffix
>> -------------------------------------------------------------------------
>> 000000010000000000000001 99% 134% 161% | 100% 126% 152%
>> 000000010000000000000006 99% 260% 287% | 100% 257% 279%
>> 000000010000000000000008 100% 100% 100% | 100% 95% 91%
>> 16398 100% 168% 221% | 100% 159% 215%
>> shakespeare.txt 100% 138% 141% | 100% 116% 117%
>> mr 99% 120% 128% | 100% 107% 108%
>> dickens 100% 129% 132% | 100% 100% 100%
>> mozilla 100% 119% 120% | 100% 102% 104%
>> nci 100% 149% 141% | 100% 143% 135%
>> ooffice 99% 121% 123% | 100% 97% 98%
>> osdb 100% 99% 99% | 100% 100% 99%
>> reymont 99% 130% 132% | 100% 106% 107%
>> samba 100% 126% 132% | 100% 105% 111%
>> sao 100% 100% 99% | 100% 100% 100%
>> webster 100% 127% 127% | 100% 106% 106%
>> x-ray 99% 99% 99% | 100% 100% 100%
>> xml 100% 144% 144% | 100% 130% 128%
>>
>> and on the other one (xeon e5-2620v4) looks like this:
>>
>> patch 1 | patch 2
>> dataset prefix infix suffix | prefix infix suffix
>> ------------------------------------------------------------------------
>> 000000010000000000000001 98% 147% 170% | 98% 132% 159%
>> 000000010000000000000006 100% 340% 314% | 98% 334% 355%
>> 000000010000000000000008 99% 100% 105% | 99% 99% 101%
>> 16398 101% 153% 205% | 99% 148% 201%
>> shakespeare.txt 100% 147% 149% | 99% 117% 118%
>> mr 100% 131% 139% | 99% 112% 108%
>> dickens 100% 143% 143% | 99% 103% 102%
>> mozilla 100% 122% 122% | 99% 105% 106%
>> nci 100% 151% 135% | 100% 135% 125%
>> ooffice 99% 127% 129% | 98% 101% 102%
>> osdb 102% 100% 101% | 102% 100% 99%
>> reymont 101% 142% 143% | 100% 108% 108%
>> samba 100% 132% 136% | 99% 109% 112%
>> sao 99% 101% 100% | 99% 100% 100%
>> webster 100% 132% 129% | 100% 106% 106%
>> x-ray 99% 101% 100% | 90% 101% 101%
>> xml 100% 147% 148% | 100% 127% 125%
>>
>> In general, I think the results for both patches seem clearly a win, but
>> maybe patch 1 is bit better, especially on the newer (xeon) CPU. So I'd
>> probably go with that one.
>
>
>
From my POV there are two interesting new points in your benchmarks:
>1. They are more or lesss end-to-end benchmarks with whole system involved.
>2. They provide per-payload breakdown
>
Yes. I was considering using the test_pglz extension first, but in the
end I decided an end-to-end test is easier to do and more relevant.
>Prefix experiment is mostly related to reading from page cache and not
>directly connected with decompression. It's a bit strange that we
>observe 1% degradation in certain experiments, but I believe it's a
>noise.
>
Yes, I agree it's probably noise - it's not always a degradation, there
are cases where it actually improves by ~1%. Perhaps more runs would
even this out, or maybe it's due to different bin layout or something.
I should have some results from a test with longer (10-minute) run soon,
but I don't think this is a massive issue.
>Infix and Suffix results are correlated. We observe no impact of the
>patch on compressed data.
>
TBH I have not looked at which data sets are compressible etc. so I
can't really comment on this.
FWIW the reason why I did the prefix/infix/suffix is primarily that I
was involved in some recent patches tweaking TOAST slicing, so I wanted
to se if this happens to negatively affect it somehow. And it does not.
>test_pglz also includes slicing by 2Kb and 8Kb. This was done to
>imitate toasting. But as far as I understand, in your test data payload
>will be inserted into toast table too, won't it? If so, I agree that
>patch 1 looks like a better option.
>
Yes, the tests simply do whatever PostgreSQL would do when loading and
storing this data, including TOASTing.
>> 27 нояб. 2019 г., в 1:05, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>>
>> Code-wise I think the patches are mostly fine, although the comments
>> might need some proof-reading.
>>
>> 1) I wasn't really sure what a "nibble" is, but maybe it's just me and
>> it's a well-known term.
>I've took the word from pg_lzcompress.c comments
> * The offset is in the upper nibble of T1 and in T2.
> * The length is in the lower nibble of T1.
Aha, good. I haven't noticed that word before, so I assumed it's
introduced by those patches. And the first thing I thought of was
"nibbles" video game [1]. Which obviously left me a bit puzzled ;-)
But it seems to be a well-known term, I just never heard it before.
[1] https://en.wikipedia.org/wiki/Nibbles_(video_game)
>>
>> 2) First byte use lower -> First byte uses lower
>>
>> 3) nibble contain upper -> nibble contains upper
>>
>> 4) to preven possible uncertanity -> to prevent possible uncertainty
>>
>> 5) I think we should briefly explain why memmove would be incompatible
>> with pglz, it's not quite clear to me.
>Here's the example
>+ * Consider input: 112341234123412341234
>+ * At byte 5 here ^ we have match with length 16 and
>+ * offset 4. 11234M(len=16, off=4)
>
>If we simply memmove() this 16 bytes we will produce
>112341234XXXXXXXXXXXX, where series of X is 12 undefined bytes, that
>were at bytes [6:18].
>
OK, thanks.
>>
>> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be
>> badly mangled by pgindent.
>
>I think I can just write it without line limit and then run pgindent.
>Will try to do it this evening. Also, I will try to write more about
>memmove.
>
>>
>> 7) The last change moving "copy" to the next line seems unnecessary.
>
>Oh, looks like I had been rewording this comment, and eventually came
>to the same text..Yes, this change is absolutely unnecessary.
>
>Thanks!
>
Good. I'll wait for an updated version of the patch and then try to get
it pushed by the end of the CF.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-27 18:27 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 2 replies; 70+ messages in thread
From: Andrey Borodin @ 2019-11-27 18:27 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
> 27 нояб. 2019 г., в 20:28, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>
>>>
>>> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be
>>> badly mangled by pgindent.
>>
>> I think I can just write it without line limit and then run pgindent.
>> Will try to do it this evening. Also, I will try to write more about
>> memmove.
Well, yes, I could not make pgindent format some parts of that comment, gave up and left only simple text.
>>
>>>
>>> 7) The last change moving "copy" to the next line seems unnecessary.
>>
>> Oh, looks like I had been rewording this comment, and eventually came
>> to the same text..Yes, this change is absolutely unnecessary.
>>
>> Thanks!
>>
>
> Good. I'll wait for an updated version of the patch and then try to get
> it pushed by the end of the CF.
PFA v5.
Thanks!
Best regards, Andrey Borodin.
Attachments:
v5-0001-Use-memcpy-in-pglz-decompression.patch (application/octet-stream, 3.4K) download
From 4da4bfb62a9b45549f0c45ddf6086fa301486812 Mon Sep 17 00:00:00 2001
From: Andrey <amborodin@acm.org>
Date: Mon, 24 Jun 2019 13:36:47 +0500
Subject: [PATCH] Use memcpy in pglz decompression
Previously pglz was using byte loop during decompression.
This commit is employing memcpy for this purpose.
---
src/common/pg_lzcompress.c | 48 ++++++++++++++++++++++++++++----------
1 file changed, 36 insertions(+), 12 deletions(-)
diff --git a/src/common/pg_lzcompress.c b/src/common/pg_lzcompress.c
index 3adad62d9a..438015e135 100644
--- a/src/common/pg_lzcompress.c
+++ b/src/common/pg_lzcompress.c
@@ -714,11 +714,13 @@ pglz_decompress(const char *source, int32 slen, char *dest,
if (ctrl & 1)
{
/*
- * Otherwise it contains the match length minus 3 and the
- * upper 4 bits of the offset. The next following byte
- * contains the lower 8 bits of the offset. If the length is
- * coded as 18, another extension tag byte tells how much
- * longer the match really was (0-255).
+ * Set control bit means we must read a match tag. The match
+ * is coded with two bytes. First byte uses lower nibble to
+ * code length - 3. Higher nibble contains upper 4 bits of the
+ * offset. The next following byte contains the lower 8 bits
+ * of the offset. If the length is coded as 18, another
+ * extension tag byte tells how much longer the match really
+ * was (0-255).
*/
int32 len;
int32 off;
@@ -731,16 +733,38 @@ pglz_decompress(const char *source, int32 slen, char *dest,
/*
* Now we copy the bytes specified by the tag from OUTPUT to
- * OUTPUT. It is dangerous and platform dependent to use
- * memcpy() here, because the copied areas could overlap
- * extremely!
+ * OUTPUT (copy len bytes from dp - off to dp). The copied
+ * areas could overlap, to preven possible uncertainty, we
+ * copy only non-overlapping regions.
*/
len = Min(len, destend - dp);
- while (len--)
+ while (off < len)
{
- *dp = dp[-off];
- dp++;
+ /*
+ * When offset is smaller than length - source and
+ * destination regions overlap. memmove() is resolving
+ * this overlap in an incompatible way with pglz. Thus we
+ * resort to memcpy()-ing non-overlapping regions.
+ * Consider input: "112341234123412341234" At byte with
+ * position 5 we have match with length 16 and offset 4:
+ * 11234M(len=16, off=4). We are decoding first period of
+ * match and rewrite match 112341234M(len=12, off=8). Same
+ * match is now at position 9, it aims to same start byte
+ * of output, but from another position: offset is
+ * doubled. We iterate through this offset growth until we
+ * can proceed to usual memcpy(). If we would try to
+ * decode the match at byte 5 (len=16, off=4) by memmove()
+ * we would issue memmove(5, 1, 16) which would produce
+ * 112341234XXXXXXXXXXXX, where series of X is 12
+ * undefined bytes, that were at bytes [5:17].
+ */
+ memcpy(dp, dp - off, off);
+ len -= off;
+ dp += off;
+ off += off;
}
+ memcpy(dp, dp - off, len);
+ dp += len;
}
else
{
@@ -790,7 +814,7 @@ pglz_decompress(const char *source, int32 slen, char *dest,
int32
pglz_maximum_compressed_size(int32 rawsize, int32 total_compressed_size)
{
- int32 compressed_size;
+ int32 compressed_size;
/*
* pglz uses one control bit per byte, so we need (rawsize * 9) bits. We
--
2.20.1
=
* Re: pglz performance
@ 2019-11-27 18:41 Alvaro Herrera <alvherre@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 0 replies; 70+ messages in thread
From: Alvaro Herrera @ 2019-11-27 18:41 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On 2019-Nov-27, Andrey Borodin wrote:
>
>
> > 27 нояб. 2019 г., в 20:28, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
> >
> >>>
> >>> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be
> >>> badly mangled by pgindent.
> >>
> >> I think I can just write it without line limit and then run pgindent.
> >> Will try to do it this evening. Also, I will try to write more about
> >> memmove.
> Well, yes, I could not make pgindent format some parts of that
> comment, gave up and left only simple text.
Please don't. The way to avoid pgindent from messing with the comment
is to surround it with dashes, /*--------- and end it with *------- */
Just make sure that you use well-aligned and lines shorter than 80, for
cleanliness; but whatever you do, if you use the dashes then pgindent
won't touch it.
(I think the closing dash line is not necessary, but it looks better for
things to be symmetrical.)
--
Álvaro Herrera https://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-28 02:39 Michael Paquier <michael@paquier.xyz>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
1 sibling, 0 replies; 70+ messages in thread
From: Michael Paquier @ 2019-11-28 02:39 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Andrey Borodin <x4mmm@yandex-team.ru>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Wed, Nov 27, 2019 at 04:28:18PM +0100, Tomas Vondra wrote:
> Yes. I was considering using the test_pglz extension first, but in the
> end I decided an end-to-end test is easier to do and more relevant.
I actually got something in this area in one of my trees:
https://github.com/michaelpq/pg_plugins/tree/master/compress_test
> Good. I'll wait for an updated version of the patch and then try to get
> it pushed by the end of the CF.
Sounds like a plan. Thanks.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-28 22:43 Tomas Vondra <tomas.vondra@2ndquadrant.com>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
1 sibling, 1 reply; 70+ messages in thread
From: Tomas Vondra @ 2019-11-28 22:43 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Wed, Nov 27, 2019 at 11:27:49PM +0500, Andrey Borodin wrote:
>
>
>> 27 нояб. 2019 г., в 20:28, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>>
>>>>
>>>> 6) I'm pretty sure the comment in the 'while (off < len)' branch will be
>>>> badly mangled by pgindent.
>>>
>>> I think I can just write it without line limit and then run pgindent.
>>> Will try to do it this evening. Also, I will try to write more about
>>> memmove.
>Well, yes, I could not make pgindent format some parts of that comment, gave up and left only simple text.
>>>
>>>>
>>>> 7) The last change moving "copy" to the next line seems unnecessary.
>>>
>>> Oh, looks like I had been rewording this comment, and eventually came
>>> to the same text..Yes, this change is absolutely unnecessary.
>>>
>>> Thanks!
>>>
>>
>> Good. I'll wait for an updated version of the patch and then try to get
>> it pushed by the end of the CF.
>
OK, pushed, with some minor cosmetic tweaks on the comments (essentially
using the formatting trick pointed out by Alvaro), and removing one
unnecessary change in pglz_maximum_compressed_size.
regards
--
Tomas Vondra http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-11-29 05:18 Andrey Borodin <x4mmm@yandex-team.ru>
parent: Tomas Vondra <tomas.vondra@2ndquadrant.com>
0 siblings, 1 reply; 70+ messages in thread
From: Andrey Borodin @ 2019-11-29 05:18 UTC (permalink / raw)
To: Tomas Vondra <tomas.vondra@2ndquadrant.com>; +Cc: Michael Paquier <michael@paquier.xyz>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
> 29 нояб. 2019 г., в 3:43, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>
> OK, pushed, with some minor cosmetic tweaks on the comments (essentially
> using the formatting trick pointed out by Alvaro), and removing one
> unnecessary change in pglz_maximum_compressed_size.
Cool, thanks!
> pglz_maximum_compressed_size
It was an artifact of pgindent.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2019-12-01 02:50 Michael Paquier <michael@paquier.xyz>
parent: Andrey Borodin <x4mmm@yandex-team.ru>
0 siblings, 0 replies; 70+ messages in thread
From: Michael Paquier @ 2019-12-01 02:50 UTC (permalink / raw)
To: Andrey Borodin <x4mmm@yandex-team.ru>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Peter Eisentraut <peter.eisentraut@2ndquadrant.com>; Alvaro Herrera <alvherre@2ndquadrant.com>; Gasper Zejn <zejn@owca.info>; pgsql-hackers@lists.postgresql.org
On Fri, Nov 29, 2019 at 10:18:18AM +0500, Andrey Borodin wrote:
>> 29 нояб. 2019 г., в 3:43, Tomas Vondra <tomas.vondra@2ndquadrant.com> написал(а):
>>
>> OK, pushed, with some minor cosmetic tweaks on the comments (essentially
>> using the formatting trick pointed out by Alvaro), and removing one
>> unnecessary change in pglz_maximum_compressed_size.
>
> Cool, thanks!
Yippee. Thanks.
--
Michael
Attachments:
signature.asc (application/pgp-signature, 832B) download
^ permalink raw reply [nested|flat] 70+ messages in thread
* Re: pglz performance
@ 2020-03-13 05:32 Andrey M. Borodin <x4mmm@yandex-team.ru>
parent: Petr Jelinek <petr@2ndquadrant.com>
3 siblings, 0 replies; 70+ messages in thread
From: Andrey M. Borodin @ 2020-03-13 05:32 UTC (permalink / raw)
To: Petr Jelinek <petr@2ndquadrant.com>; +Cc: Tomas Vondra <tomas.vondra@2ndquadrant.com>; Andres Freund <andres@anarazel.de>; Konstantin Knizhnik <k.knizhnik@postgrespro.ru>; Michael Paquier <michael@paquier.xyz>; pgsql-hackers; Vladimir Leskov <vladimirlesk@yandex-team.ru>
Hi Petr!
> 4 авг. 2019 г., в 05:41, Petr Jelinek <petr@2ndquadrant.com> написал(а):
>
> Just so that we don't idly talk, what do you think about the attached?
> It:
> - adds new GUC compression_algorithm with possible values of pglz (default) and lz4 (if lz4 is compiled in), requires SIGHUP
> - adds --with-lz4 configure option (default yes, so the configure option is actually --without-lz4) that enables the lz4, it's using system library
Do you plan to work on lz4 aiming at 13 or 14?
Maybe let's register it on CF 2020-07?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 70+ messages in thread
end of thread, other threads:[~2020-03-13 05:32 UTC | newest]
Thread overview: 70+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2019-05-13 02:45 pglz performance Andrey Borodin <x4mmm@yandex-team.ru>
2019-05-13 07:14 ` Michael Paquier <michael@paquier.xyz>
2019-05-15 10:06 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-05-16 17:13 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-05-17 01:44 ` Michael Paquier <michael@paquier.xyz>
2019-05-17 10:59 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-05-17 13:40 ` Gasper Zejn <zejn@owca.info>
2019-05-18 06:44 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-06-24 08:44 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-09-04 09:09 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-04 09:22 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-09-04 12:40 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-04 12:45 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-09-27 11:41 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-28 08:29 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-10-21 09:09 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-10-25 05:05 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-11-01 12:33 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-11-01 15:48 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-11-01 16:59 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-11-02 11:30 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-11-03 09:24 ` Tels <nospam-pg-abuse@bloodgate.com>
2019-11-04 07:14 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-11-06 08:04 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-11-25 08:03 ` Michael Paquier <michael@paquier.xyz>
2019-11-25 08:21 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-11-25 08:29 ` Michael Paquier <michael@paquier.xyz>
2019-11-26 09:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-11-26 19:17 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-11-26 20:05 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-11-27 08:01 ` Michael Paquier <michael@paquier.xyz>
2019-11-27 09:42 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-11-27 12:47 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-11-27 15:28 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-11-27 18:27 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-11-27 18:41 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-11-28 22:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-11-29 05:18 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-12-01 02:50 ` Michael Paquier <michael@paquier.xyz>
2019-11-28 02:39 ` Michael Paquier <michael@paquier.xyz>
2019-11-06 08:03 ` Peter Eisentraut <peter.eisentraut@2ndquadrant.com>
2019-09-15 10:57 ` Oleg Bartunov <obartunov@postgrespro.ru>
2019-06-27 18:33 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-08-02 13:45 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-08-02 14:43 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-02 15:40 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-08-02 16:39 ` Andres Freund <andres@anarazel.de>
2019-08-02 17:00 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-02 17:12 ` Andres Freund <andres@anarazel.de>
2019-08-02 17:52 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-02 18:20 ` Andres Freund <andres@anarazel.de>
2019-08-02 19:48 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-04 00:41 ` Petr Jelinek <petr@2ndquadrant.com>
2019-08-04 09:57 ` Andrey Borodin <x4mmm@yandex-team.ru>
2019-08-04 15:52 ` Petr Jelinek <petr@2ndquadrant.com>
2019-08-04 11:52 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-04 15:53 ` Petr Jelinek <petr@2ndquadrant.com>
2019-08-04 17:30 ` Tomas Vondra <tomas.vondra@2ndquadrant.com>
2019-08-04 22:15 ` Andres Freund <andres@anarazel.de>
2019-08-04 22:19 ` Petr Jelinek <petr@2ndquadrant.com>
2019-08-05 07:26 ` Andres Freund <andres@anarazel.de>
2019-09-25 21:24 ` Petr Jelinek <petr@2ndquadrant.com>
2019-08-04 19:20 ` Andres Freund <andres@anarazel.de>
2019-08-04 22:08 ` Petr Jelinek <petr@2ndquadrant.com>
2020-03-13 05:32 ` Andrey M. Borodin <x4mmm@yandex-team.ru>
2019-08-03 05:37 ` Konstantin Knizhnik <k.knizhnik@postgrespro.ru>
2019-08-05 07:04 ` Michael Paquier <michael@paquier.xyz>
2019-08-06 07:04 ` Andres Freund <andres@anarazel.de>
2019-09-12 18:58 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2019-05-23 14:27 Re: pglz performance Binguo Bao <djydewang@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox