pg.ddx.io  pgsql-general@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: Rob Sargent <robjsargent@gmail.com>
To: Joe Carlson <jwcarlson@lbl.gov>
Cc: pgsql-general@lists.postgresql.org
Subject: Re: TEXT column > 1Gb
Date: Wed, 12 Apr 2023 10:36:25 -0600
Message-ID: <d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com> (raw)
In-Reply-To: <0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov>
References: <85984fc4-6e08-06ad-78d7-d6f73a5c90a0@gmail.com>
	<0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov>

On 4/12/23 08:59, Joe Carlson wrote:
> I’m curious what you learned. I’ve been tripping over the buffer 
> allocation issue when either splitting input text into chunks or 
> aggregating chunks in selects. I’ve decided that I need to move this 
> to client side.
>
> The use case is genomics. Extracting substrings is common. So going to 
> chunked storage makes sense.
>
> And we have a menagerie of code to deal with. Legacy Perl for loading. 
> Clients in Perl, Java, node, python. A whole zoo.
>
> Thanks

My first stab was a java implementation based Selector class which 
worked ok using a fixed buffer size but finding the end of last 
transmission was hokey.  I move to an embedded Tomcat implementation and 
tomcat does the heavy lifting PERFECTLY.  Easily slipped in to both 
client and backend.

rjs






view thread (15+ messages)  latest in thread

Message-ID: <d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com>
Permalink:  ../d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com/
Also on:    postgresql.org/message-id/d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-general@postgresql.org
  Cc: robjsargent@gmail.com, jwcarlson@lbl.gov
  Subject: Re: TEXT column > 1Gb
  In-Reply-To: <d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com>
  References: <85984fc4-6e08-06ad-78d7-d6f73a5c90a0@gmail.com> <0A8EB6F5-57BC-4519-8E9A-DBB809CC09D5@lbl.gov> <d84a153a-8ea6-ae4a-7d3e-c713908f97f6@gmail.com>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox