pg.ddx.io  pgsql-sql@postgresql.org mailing list archive  
help / color / mirror / Atom feed
8K Limit, whats the best strategy?
7+ messages / 5 participants
[nested] [flat]

* 8K Limit, whats the best strategy?
@ 2000-08-15 11:29  Keith Wong <keith@e-magine.com.au>
  1 sibling, 1 reply; 7+ messages in thread

From: Keith Wong @ 2000-08-15 11:29 UTC (permalink / raw)
  To: pgsql-sql

Hi everyone,

Just wondering what strategies people have used to get around the 8K row 
limit in Postgres.
If anyone has been troubled by this limitation before and has a nice 
solution around it, I would love to hear it.

My application is a web-based system in which needs to store large amounts 
of text.
The text stored needs to searchable as well.

Cheers,
Keith.




^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: 8K Limit, whats the best strategy?
@ 2000-08-15 15:37  Thomas Swan <tswan@olemiss.edu>
  1 sibling, 1 reply; 7+ messages in thread

From: Thomas Swan @ 2000-08-15 15:37 UTC (permalink / raw)
  To: Keith Wong <keith@e-magine.com.au>; pgsql-sql

At 06:29 AM 8/15/2000, Keith Wong wrote:
>Hi everyone,
>
>Just wondering what strategies people have used to get around the 8K row 
>limit in Postgres.
>If anyone has been troubled by this limitation before and has a nice 
>solution around it, I would love to hear it.
>
>My application is a web-based system in which needs to store large amounts 
>of text.
>The text stored needs to searchable as well.

You can you the type LZTEXT which is compressed text so your mileage will vary.

The other option is to go in and recompile for a larger row size.  Most of 
the people I have talked to had no problems with this route.  I move the 
size limit on an x86 Linux box to 32K (the max) and have not had any 
problems.   In the next release, I don't believe the row size limit will 
exist (correct me if I'm wrong).

Let me know the system your on, and I will try to help if I can.

-
- Thomas Swan
- Graduate Student  - Computer Science
- The University of Mississippi
-
- "People can be categorized into two fundamental
- groups, those that divide people into two groups
- and those that don't."

^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: 8K Limit, whats the best strategy?
@ 2000-08-15 15:48  Adam Lang <aalang@rutgersinsurance.com>
  parent: Thomas Swan <tswan@olemiss.edu>
  0 siblings, 0 replies; 7+ messages in thread

From: Adam Lang @ 2000-08-15 15:48 UTC (permalink / raw)
  To: pgsql-sql

I believe you are correct.  Version 7.1 will supposedly eliminate the row
size.

Adam Lang
Systems Engineer
Rutgers Casualty Insurance Company
----- Original Message -----
From: "Thomas Swan" <tswan@olemiss.edu>
To: "Keith Wong" <keith@e-magine.com.au>; <pgsql-sql@postgresql.org>
Sent: Tuesday, August 15, 2000 11:37 AM
Subject: Re: [SQL] 8K Limit, whats the best strategy?


> At 06:29 AM 8/15/2000, Keith Wong wrote:
> >Hi everyone,
> >
> >Just wondering what strategies people have used to get around the 8K row
> >limit in Postgres.
> >If anyone has been troubled by this limitation before and has a nice
> >solution around it, I would love to hear it.
> >
> >My application is a web-based system in which needs to store large
amounts
> >of text.
> >The text stored needs to searchable as well.
>
> You can you the type LZTEXT which is compressed text so your mileage will
vary.
>
> The other option is to go in and recompile for a larger row size.  Most of
> the people I have talked to had no problems with this route.  I move the
> size limit on an x86 Linux box to 32K (the max) and have not had any
> problems.   In the next release, I don't believe the row size limit will
> exist (correct me if I'm wrong).
>
> Let me know the system your on, and I will try to help if I can.
>
> -
> - Thomas Swan
> - Graduate Student  - Computer Science
> - The University of Mississippi
> -
> - "People can be categorized into two fundamental
> - groups, those that divide people into two groups
> - and those that don't."




^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: 8K Limit, whats the best strategy?
@ 2000-08-15 17:49  Poul L. Christiansen <plc@faroenet.fo>
  parent: Keith Wong <keith@e-magine.com.au>
  0 siblings, 1 reply; 7+ messages in thread

From: Poul L. Christiansen @ 2000-08-15 17:49 UTC (permalink / raw)
  To: Keith Wong <keith@e-magine.com.au>; +Cc: pgsql-sql

I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
only a viable solution if your text is less than a couple of 100K's.

You could try to be a daredevil and use the Toast code, even if it's beta. But
I don't know how far the Toast project has come.

Keith Wong wrote:

> Hi everyone,
>
> Just wondering what strategies people have used to get around the 8K row
> limit in Postgres.
> If anyone has been troubled by this limitation before and has a nice
> solution around it, I would love to hear it.
>
> My application is a web-based system in which needs to store large amounts
> of text.
> The text stored needs to searchable as well.
>
> Cheers,
> Keith.




^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: 8K Limit, whats the best strategy?
@ 2000-08-21 15:20  Poul L. Christiansen <plc@faroenet.fo>
  parent: Jan Wieck <janwieck@Yahoo.com>
  0 siblings, 1 reply; 7+ messages in thread

From: Poul L. Christiansen @ 2000-08-21 15:20 UTC (permalink / raw)
  To: Jan Wieck <janwieck@Yahoo.com>; +Cc: Keith Wong <keith@e-magine.com.au>; pgsql-sql

Jan Wieck wrote:

> Poul L. Christiansen wrote:
> > I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
> > only a viable solution if your text is less than a couple of 100K's.
> >
> > You could try to be a daredevil and use the Toast code, even if it's beta. But
> > I don't know how far the Toast project has come.
>
>     TOAST  is  finished  and will be shipped with 7.1. It's not a
>     solution for huge items, but medium sized  text  up  to  some
>     hundred K works fine.

What do you mean by "..not a solution for huge items"? Does TOAST have a size limit?

>
>
> Jan
>
> >
> > Keith Wong wrote:
> >
> > > Hi everyone,
> > >
> > > Just wondering what strategies people have used to get around the 8K row
> > > limit in Postgres.
> > > If anyone has been troubled by this limitation before and has a nice
> > > solution around it, I would love to hear it.
> > >
> > > My application is a web-based system in which needs to store large amounts
> > > of text.
> > > The text stored needs to searchable as well.
> > >
> > > Cheers,
> > > Keith.
> >
>
> --
>
> #======================================================================#
> # It's easier to get forgiveness for being wrong than for being right. #
> # Let's break this rule - forgive me.                                  #
> #================================================== JanWieck@Yahoo.com #




^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: 8K Limit, whats the best strategy?
@ 2000-08-21 15:41  Jan Wieck <janwieck@Yahoo.com>
  parent: Poul L. Christiansen <plc@faroenet.fo>
  0 siblings, 1 reply; 7+ messages in thread

From: Jan Wieck @ 2000-08-21 15:41 UTC (permalink / raw)
  To: Poul L. Christiansen <plc@faroenet.fo>; +Cc: Keith Wong <keith@e-magine.com.au>; pgsql-sql

Poul L. Christiansen wrote:
> I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
> only a viable solution if your text is less than a couple of 100K's.
>
> You could try to be a daredevil and use the Toast code, even if it's beta. But
> I don't know how far the Toast project has come.

    TOAST  is  finished  and will be shipped with 7.1. It's not a
    solution for huge items, but medium sized  text  up  to  some
    hundred K works fine.


Jan

>
> Keith Wong wrote:
>
> > Hi everyone,
> >
> > Just wondering what strategies people have used to get around the 8K row
> > limit in Postgres.
> > If anyone has been troubled by this limitation before and has a nice
> > solution around it, I would love to hear it.
> >
> > My application is a web-based system in which needs to store large amounts
> > of text.
> > The text stored needs to searchable as well.
> >
> > Cheers,
> > Keith.
>


--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #





^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: 8K Limit, whats the best strategy?
@ 2000-08-21 20:47  Jan Wieck <janwieck@Yahoo.com>
  parent: Poul L. Christiansen <plc@faroenet.fo>
  0 siblings, 0 replies; 7+ messages in thread

From: Jan Wieck @ 2000-08-21 20:47 UTC (permalink / raw)
  To: pgsql-sql; +Cc: Jan Wieck <janwieck@Yahoo.com>; Keith Wong <keith@e-magine.com.au>

Poul L. Christiansen wrote:
> Jan Wieck wrote:
>
> > Poul L. Christiansen wrote:
> > > I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
> > > only a viable solution if your text is less than a couple of 100K's.
> > >
> > > You could try to be a daredevil and use the Toast code, even if it's beta. But
> > > I don't know how far the Toast project has come.
> >
> >     TOAST  is  finished  and will be shipped with 7.1. It's not a
> >     solution for huge items, but medium sized  text  up  to  some
> >     hundred K works fine.
>
> What do you mean by "..not a solution for huge items"? Does TOAST have a size limit?
>

    Not  an explicit one. But imagine you really want to store an
    MP3 of - let's say 9M in the database.

    1.  Your client application must quote it somehow to  put  it
        into  an  INSERT querystring. The quoting makes it a 10MB
        thing (think positive).

    2.  The query is sent to  the  backend.  Now  you  have  this
        string a second time in memory.

    3.  The  query  is  parsed and a 10MB text datum is built for
        the querytrees Const node.

    4.  The query is executed, builds a 10MB tuple to store.

    5.  Toast jumps in and moves it out of the tuple again.

    So if the client and DB  are  running  on  the  same  system,
    you'll  have  the  MP3  four  times  in  memory.  Now  do  it
    concurrent with 20 backends and you'll need 800+ ... you  see
    why I said it's "not a solution for huge"?


Jan

--

#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me.                                  #
#================================================== JanWieck@Yahoo.com #





^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2000-08-21 20:47 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2000-08-15 11:29 ` Keith Wong <keith@e-magine.com.au>
2000-08-15 17:49   ` Poul L. Christiansen <plc@faroenet.fo>
2000-08-21 15:41     ` Jan Wieck <janwieck@Yahoo.com>
2000-08-21 15:20       ` Poul L. Christiansen <plc@faroenet.fo>
2000-08-21 20:47         ` Jan Wieck <janwieck@Yahoo.com>
2000-08-15 15:37 ` Thomas Swan <tswan@olemiss.edu>
2000-08-15 15:48   ` Adam Lang <aalang@rutgersinsurance.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox