Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qmg5n-008Zlv-Bc for pgsql-hackers@arkaria.postgresql.org; Sat, 30 Sep 2023 19:57:36 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1qmg5l-00FNB2-M4 for pgsql-hackers@arkaria.postgresql.org; Sat, 30 Sep 2023 19:57:33 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qmDfV-00FZ9U-Rh for pgsql-hackers@lists.postgresql.org; Fri, 29 Sep 2023 13:36:34 +0000 Received: from owa.datagile.ru ([185.42.62.141]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qmDfS-008HP4-PO for pgsql-hackers@postgresql.org; Fri, 29 Sep 2023 13:36:33 +0000 Received: from localhost (localhost [127.0.0.1]) by owa.datagile.ru (Postfix) with ESMTP id 7AFC6154BB5 for ; Fri, 29 Sep 2023 13:36:28 +0000 (UTC) Received: from owa.datagile.ru ([127.0.0.1]) by localhost (owa.datagile.ru [127.0.0.1]) (amavis, port 10032) with ESMTP id MUmrb_amKeEg for ; Fri, 29 Sep 2023 13:36:28 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by owa.datagile.ru (Postfix) with ESMTP id 1349615EA51 for ; Fri, 29 Sep 2023 13:36:28 +0000 (UTC) DKIM-Filter: OpenDKIM Filter v2.10.3 owa.datagile.ru 1349615EA51 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datagile.ru; s=B504547E-FE03-11ED-9F7F-496073366B9D; t=1695994588; bh=lMjKa57+ZYnVXH8MMACWpqBcsNBUmNwdma3qnNquVNE=; h=From:To:Date:Message-ID:MIME-Version; b=QxfSGoUp8zefrA9A4qpfaWz44KYc4oA3YzdYSurQDcACD1RtowDkcC9YApfunJwPf yFcrKbcKC/AW4xo5IgHB8lDhb21MXJlt7jNpWgQIGYZs0wPaOprXjdED9OHoZ123en R3JxwlRBip/ysZk+v3HL/1fSPqJGNdQL0f73RbcXrWUqhYe6BhNHb19vxFk33eqkzY fbJiJfEvmiVSe87kGgVizGFGTi3QPRFDlCXPn5BY6tdJmWHEopL5Fx9IUzX5akGq4c zUu+8bdNVAYUxN0+KyKGzZB4DlsK0v6kMPXxHShO3VKJ+SVeyTPUUaop8/kB0FxWww SnDoiqcq7786g== X-Virus-Scanned: amavis at datagile.ru Received: from owa.datagile.ru ([127.0.0.1]) by localhost (owa.datagile.ru [127.0.0.1]) (amavis, port 10026) with ESMTP id Ue1jVEXJEZc0 for ; Fri, 29 Sep 2023 13:36:28 +0000 (UTC) Received: from owa.datagile.ru (owa.datagile.ru [192.168.242.40]) by owa.datagile.ru (Postfix) with ESMTP id 03509154BB5 for ; Fri, 29 Sep 2023 13:36:28 +0000 (UTC) From: =?koi8-r?B?5dLPyMnOIOTFzsnTIPfMwcTJzcnSz9fJ3g==?= To: Subject: Fix receiving large legal tsvector from binary format Thread-Topic: Fix receiving large legal tsvector from binary format Date: Fri, 29 Sep 2023 13:36:27 +0000 (UTC) Message-ID: <009801d9f2d9$f29730c0$d7c59240$@datagile.ru> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="----=_NextPart_000_0099_01D9F2F3.17E4DDF0" X-Mailer: Microsoft Outlook 15.0 X-Mailer: Zimbra 10.0.2_GA_4565 (Zimbra-ZCO/9.0.0.1933 (10.0.19045 ru-RU) P2454 T860 R5729) Thread-Index: Adny2R7zVhjF930YRoCjF/yqI/y0Iw== Content-Language: ru X-Originating-IP: [192.168.242.40] List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk This is a multipart message in MIME format. ------=_NextPart_000_0099_01D9F2F3.17E4DDF0 Content-Type: multipart/alternative; boundary="----=_NextPart_001_009A_01D9F2F3.17E4DDF0" ------=_NextPart_001_009A_01D9F2F3.17E4DDF0 Content-Type: text/plain; charset="koi8-r" Content-Transfer-Encoding: 7bit Hello, There is a problem on receiving large tsvector from binary format with getting error "invalid tsvector: maximum total lexeme length exceeded". Required simple steps to reproduce problem: - Make table with one column of type 'tsvector' - Add row with large tsvector (900 Kb will be enougth) - Save table to file with "copy to with binary" - Restore table from file with "copy from with binary" At last step we can't restore a legal tsvector because there is error of required memory size calculation for tsvector during reading of its binary. It seems function "tsvectorrecv" can be improved: v1-0001-Fix-receiving-of-big-tsvector-from-binary - patch fixes problem of required size calculation for tsvector where wrong type is used (WordEntry instead of WordEntryPos). Size of wrong type is bigger than type of needed type therefore total size turns out larger than needed. So next test of that size fails during maximum size check. v1-0002-Replace-magic-one-in-tsvector-binary-receiving - patch removes magic ones from code. Those ones are used to add "sizeof(uint16)" to required size where number of WordEntryPos'es will be stored. Now it works only because size of WordEntryPos is equal to size of uint16. But it is not obviously during code reading and causes question "what for this magic one is?" v1-0003-Optimize-memory-allocation-during-tsvector-binary - patch makes memory allocation more accuracy and rarely. It seems memory to store tsvector's data is allocated redundant and reallocations happen more often than necessary. Best regards, Denis Erokhin Jatoba ------=_NextPart_001_009A_01D9F2F3.17E4DDF0 Content-Type: text/html; charset="koi8-r" Content-Transfer-Encoding: quoted-printable

Hello,

 

There is a = problem on receiving large tsvector from binary format with getting = error "invalid tsvector: maximum total lexeme length = exceeded". Required simple steps to reproduce = problem:

- Make table with one column of type = 'tsvector'

- Add row with large tsvector (900 Kb will be = enougth)

- Save table to file with "copy to with = binary"

- Restore table from file with = "copy from with binary"

 

At last step = we can't restore a legal tsvector because there is error of required = memory size calculation for tsvector during reading of its = binary.

 

It seems = function "tsvectorrecv" can be = improved:

v1-0001-Fix-receiving-of-big-tsvector-from-binary = - patch fixes problem of required size calculation for tsvector where = wrong type is used (WordEntry instead of WordEntryPos). Size of wrong = type is bigger than type of needed type therefore total size turns out = larger than needed. So next test of that size fails during maximum size = check.

v1-0002-Replace-magic-one-in-tsvector-binary-receiv= ing - patch removes magic ones from code. Those ones are used to add = "sizeof(uint16)" to required size where number of = WordEntryPos'es will be stored. Now it works only because size of = WordEntryPos is equal to size of uint16. But it is not obviously during = code reading and causes question "what for this magic one = is?"

v1-0003-Optimize-memory-allocation-during-tsvector-= binary - patch makes memory allocation more accuracy and rarely. It = seems memory to store tsvector's data is allocated redundant and = reallocations happen more often than necessary.

 

Best = regards,

Denis Erokhin

Jatoba

 

------=_NextPart_001_009A_01D9F2F3.17E4DDF0-- ------=_NextPart_000_0099_01D9F2F3.17E4DDF0 Content-Type: application/octet-stream; name="v1-0001-Fix-receiving-of-big-tsvector-from-binary.patch" Content-Transfer-Encoding: quoted-printable Content-Disposition: attachment; filename="v1-0001-Fix-receiving-of-big-tsvector-from-binary.patch"