pg.ddx.io  pgsql-sql@postgresql.org mailing list archive  
help / color / mirror / Atom feed
From: wieck@debis.com
To: mitch@venux.net
Cc: pgsql-sql@postgreSQL.org
Subject: Re: [SQL] Dilema.
Date: Sat, 11 Dec 1999 01:03:35 +0100 (MET)
Message-ID: <m11wa0d-0003kGC@orion.SAPserv.Hamburg.dsh.de> (raw)
In-Reply-To: <00a201bf4356$a1640b80$0300000a@doot.org>

>   I have a problem that I'm hoping someone can help me with. I've built a
> web-pased application that is back-ended by PGsql.. It works great, I
> couldn't ask for better preformance.

    And  Bruce  just  said  (on the hackers list) that big tuples
    aren't an item of big pressure.  Reading this  less  than  24
    hours past tells another story.

> However, I just recently coded another module for it and this module needed
> to accept text files (or just large text blocks lets say). I ran into the 8k
> query limit barrier. I need these blocks of text to be searchable and am
> having to do this with a flat-file based solution right now, it's slow and
> cumbersome. I would much rather have the text stored in a PG field so I
> don't have to stray from the design of the search engine (which is designed
> to search only the pg database, but has been hacked to search flat files
> too).. I could break the texts up but they really all need to be in a single
> field, not broken up. I can't think up a way to do it with the 8k query
> limit.

    If  you  really  need  to  store  big  texts  into a table, a
    combination of a view, update  rules  and  PL  functions  can
    definitely  do  it. It might look a little complex first, but
    after understanding the trick, it's pretty neat.

    Look at this:

        pgsql=> \d bttest
              View "bttest"
         Attribute | Type | Extra
        -----------+------+-------
         key       | int4 |
         content   | text |
        View definition: SELECT bttest_shadow."key",
        bigtext_fetch(bttest_shadow.btid) AS content FROM
        bttest_shadow;

        pgsql=> select key, length(content) from bttest;
         key | length
        -----+--------
           1 |      3
           2 |      9
           4 | 135840
        (3 rows)

    Yes, the row with key=4 really contains a text of that  size.
    And I can get it back exactly as it was inserted.  And yes, I
    can do INSERT/UPDATE/DELETE on bttest. Anything behaves as if
    it where a regular table.

    But  it  needs  the  unlimited  querysize,  supported only in
    CURRENT development tree, so you'll  have  to  wait  for  7.0
    anyway.

> Also feel free to suggest some kind of indexing system for the flat files,
> the only problem with that will be integration into my existing application
> but I've left some hooks open so I can deal with it..

    That's another story, and Tom's comment on it says it all.


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#========================================= wieck@debis.com (Jan Wieck) #





view thread (5+ messages)  latest in thread

Message-ID: <m11wa0d-0003kGC@orion.SAPserv.Hamburg.dsh.de>
Permalink:  ../m11wa0d-0003kGC@orion.SAPserv.Hamburg.dsh.de/
Also on:    postgresql.org/message-id/m11wa0d-0003kGC@orion.SAPserv.Hamburg.dsh.de

 · 

reply

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Reply to all the recipients from your mail client; the link fills in
  the headers below: reply via email
  Some clients, webmail especially, drop In-Reply-To and References from
  a link, and the reply then starts a new thread: use the mbox below.

  To: pgsql-sql@postgresql.org
  Cc: wieck@debis.com, mitch@venux.net
  Subject: Re: [SQL] Dilema.
  In-Reply-To: <m11wa0d-0003kGC@orion.SAPserv.Hamburg.dsh.de>
  References: <00a201bf4356$a1640b80$0300000a@doot.org> <m11wa0d-0003kGC@orion.SAPserv.Hamburg.dsh.de>

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox