Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qeLVi-006YVJ-Dc for pgsql-general@arkaria.postgresql.org; Thu, 07 Sep 2023 20:21:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1qeLVg-004Y1j-A7 for pgsql-general@arkaria.postgresql.org; Thu, 07 Sep 2023 20:21:52 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qeLVf-004Y1a-Qg for pgsql-general@lists.postgresql.org; Thu, 07 Sep 2023 20:21:51 +0000 Received: from sss.pgh.pa.us ([68.162.161.243]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1qeLVc-003JIe-8l for pgsql-general@lists.postgresql.org; Thu, 07 Sep 2023 20:21:50 +0000 Received: from sss1.sss.pgh.pa.us (localhost [127.0.0.1]) by sss.pgh.pa.us (8.15.2/8.15.2) with ESMTP id 387KLhn21755042; Thu, 7 Sep 2023 16:21:43 -0400 From: Tom Lane To: Erik Wienhold cc: Sai Teja , pgsql-general@lists.postgresql.org Subject: Re: Huge input lookup exception when trying to create the index for XML data type column in postgreSQL In-reply-to: <87357840.138645.1694115470979@office.mailbox.org> References: <1736511.1694113781@sss.pgh.pa.us> <87357840.138645.1694115470979@office.mailbox.org> Comments: In-reply-to Erik Wienhold message dated "Thu, 07 Sep 2023 21:37:50 +0200" MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <1755040.1694118103.1@sss.pgh.pa.us> Date: Thu, 07 Sep 2023 16:21:43 -0400 Message-ID: <1755041.1694118103@sss.pgh.pa.us> List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk Erik Wienhold writes: > On 07/09/2023 21:09 CEST Tom Lane wrote: >> There is no such string anywhere in the Postgres source code; >> furthermore, if someone tried to add such an error, it'd get rejected >> (I hope) as not conforming to our style guidelines. I thought maybe >> it's coming from libxml or the xpath code, but I couldn't get a match >> for it anywhere in Debian Code Search either. Is that the *exact* >> spelling of the message? > Looks like "Huge input lookup" as reported in [1] (also from Sai) and that > error is from libxml. Ah, thanks for the pointer. It looks like for the DOCUMENT case, we could maybe relax this restriction by passing the XML_PARSE_HUGE option to xmlCtxtReadDoc(). However, there are things to worry about: * Some of the other libxml functions we use don't seem to have an options argument, so it's not clear how to remove the limit in all code paths. * One of the first hits I got while googling for XML_PARSE_HUGE was CVE-2022-40303 [1] (libxml2: integer overflows with XML_PARSE_HUGE). It seems highly likely that not everybody's libxml is patched for that yet, meaning we'd be opening a lot of systems to security issues. * XML_PARSE_HUGE apparently also removes restrictions on nesting depth of XML documents. I wonder whether that creates a risk of stack-overflow crashes. On the whole, I'm not sure I want to mess with this. libxml2 is rickety enough already without taking off its training wheels. And, as noted by David J., we'd very possibly only be moving the bottleneck somewhere else. "Put many megabytes of data into one field" is an antipattern for successful SQL use, and probably always will be. regards, tom lane [1] https://bugzilla.redhat.com/show_bug.cgi?id=2136266