MIME-Version: 1.0
References: 
 <CAJ7c6TOtAB0z1UrksvGTStNE-herK-43bj22=5xVBg7S4vr5rQ@mail.gmail.com>
 <CAAWbhmgtHaVdh4cDHcDWOd7m4KL6RfBcdtYgy5WDSKN=Sucb5w@mail.gmail.com>
 <CAJ7c6TMThnxjn6sM7s8y5Yib4+BDhvGGFhW=qK8R+YMiKTwvBQ@mail.gmail.com>
 <CANbhV-HpCF852WcZuU0wyh1jMU4p6XLbV6rCRkZpnpeKQ9OenQ@mail.gmail.com>
In-Reply-To: 
 <CANbhV-HpCF852WcZuU0wyh1jMU4p6XLbV6rCRkZpnpeKQ9OenQ@mail.gmail.com>
From: Aleksander Alekseev <aleksander@timescale.com>
Date: Tue, 28 Jun 2022 15:37:14 +0300
Message-ID: 
 <CAJ7c6TOkpCBFV_GHUfcf5q-aBw9EpkAfDdbhMorLs3QA=OwW5g@mail.gmail.com>
Subject: Re: [PATCH] Compression dictionaries for JSONB
To: PostgreSQL Hackers <pgsql-hackers@lists.postgresql.org>
Cc: Jacob Champion <jchampion@timescale.com>,
 Simon Riggs <simon.riggs@enterprisedb.com>,
	Alvaro Herrera <alvherre@alvh.no-ip.org>,
	Matthias van de Meent <boekewurm+postgres@gmail.com>
Content-Type: text/plain; charset="UTF-8"
Archived-At: 
 <https://www.postgresql.org/message-id/CAJ7c6TOkpCBFV_GHUfcf5q-aBw9EpkAfDdbhMorLs3QA%3DOwW5g%40mail.gmail.com>
Precedence: bulk

Hi Simon,

Many thanks for your feedback!

I'm going to submit an updated version of the patch in a bit. I just
wanted to reply to some of your questions / comments.

> Dictionaries have no versioning. [...]

> Does the order of entries in the dictionary allow us to express a priority? i.e. to allow Huffman coding. [...]

This is something we discussed in the RFC thread. I got an impression
that the consensus was reached:

1. To simply use 32-bit codes in the compressed documents, instead of
16-bit ones as it was done in ZSON;
2. Not to use any sort of variable-length coding;
3. Not to use dictionary versions. New codes can be added to the
existing dictionaries by executing ALTER TYPE mydict ADD ENTRY. (This
also may answer your comment regarding a limit on SQL statement size.)
4. The compression scheme can be altered in the future if needed.
Every compressed document stores algorithm_version (1 byte).

Does this plan of action sound OK to you? At this point it is not too
difficult to make design changes.

-- 
Best regards,
Aleksander Alekseev