From: Rob Sargent <robjsargent@gmail.com>
To: Joe Carlson <jwcarlson@lbl.gov>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: TEXT column > 1Gb
Date: Wed, 12 Apr 2023 10:36:25 -0600
Message-ID: <d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com> (raw)
In-Reply-To: <0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov>
References: <85984fc4-6e08-06ad-78d7-d6f73a5c90a0@gmail.com>
<0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov>
On 4/12/23 08:59, Joe Carlson wrote:
> I’m curious what you learned. I’ve been tripping over the buffer
> allocation issue when either splitting input text into chunks or
> aggregating chunks in selects. I’ve decided that I need to move this
> to client side.
>
> The use case is genomics. Extracting substrings is common. So going to
> chunked storage makes sense.
>
> And we have a menagerie of code to deal with. Legacy Perl for loading.
> Clients in Perl, Java, node, python. A whole zoo.
>
> Thanks
My first stab was a java implementation based Selector class which
worked ok using a fixed buffer size but finding the end of last
transmission was hokey. I move to an embedded Tomcat implementation and
tomcat does the heavy lifting PERFECTLY. Easily slipped in to both
client and backend.
rjs
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients from your mail client; the link fills in
the headers below: reply via email
Some clients, webmail especially, drop In-Reply-To and References from
a link, and the reply then starts a new thread: use the mbox below.
To: pgsql-general@postgresql.org
Cc: robjsargent@gmail.com, jwcarlson@lbl.gov
Subject: Re: TEXT column > 1Gb
In-Reply-To: <d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com>
References: <85984fc4-6e08-06ad-78d7-d6f73a5c90a0@gmail.com> <0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov> <d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox