pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feed8K Limit, whats the best strategy?
7+ messages / 5 participants
[nested] [flat]
* 8K Limit, whats the best strategy?
@ 2000-08-15 11:29 Keith Wong <keith@e-magine.com.au>
1 sibling, 1 reply; 7+ messages in thread
From: Keith Wong @ 2000-08-15 11:29 UTC (permalink / raw)
To: pgsql-sql
Hi everyone,
Just wondering what strategies people have used to get around the 8K row
limit in Postgres.
If anyone has been troubled by this limitation before and has a nice
solution around it, I would love to hear it.
My application is a web-based system in which needs to store large amounts
of text.
The text stored needs to searchable as well.
Cheers,
Keith.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: 8K Limit, whats the best strategy?
@ 2000-08-15 15:37 Thomas Swan <tswan@olemiss.edu>
1 sibling, 1 reply; 7+ messages in thread
From: Thomas Swan @ 2000-08-15 15:37 UTC (permalink / raw)
To: Keith Wong <keith@e-magine.com.au>; pgsql-sql
At 06:29 AM 8/15/2000, Keith Wong wrote:
>Hi everyone,
>
>Just wondering what strategies people have used to get around the 8K row
>limit in Postgres.
>If anyone has been troubled by this limitation before and has a nice
>solution around it, I would love to hear it.
>
>My application is a web-based system in which needs to store large amounts
>of text.
>The text stored needs to searchable as well.
You can you the type LZTEXT which is compressed text so your mileage will vary.
The other option is to go in and recompile for a larger row size. Most of
the people I have talked to had no problems with this route. I move the
size limit on an x86 Linux box to 32K (the max) and have not had any
problems. In the next release, I don't believe the row size limit will
exist (correct me if I'm wrong).
Let me know the system your on, and I will try to help if I can.
-
- Thomas Swan
- Graduate Student - Computer Science
- The University of Mississippi
-
- "People can be categorized into two fundamental
- groups, those that divide people into two groups
- and those that don't."
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: 8K Limit, whats the best strategy?
@ 2000-08-15 15:48 Adam Lang <aalang@rutgersinsurance.com>
parent: Thomas Swan <tswan@olemiss.edu>
0 siblings, 0 replies; 7+ messages in thread
From: Adam Lang @ 2000-08-15 15:48 UTC (permalink / raw)
To: pgsql-sql
I believe you are correct. Version 7.1 will supposedly eliminate the row
size.
Adam Lang
Systems Engineer
Rutgers Casualty Insurance Company
----- Original Message -----
From: "Thomas Swan" <tswan@olemiss.edu>
To: "Keith Wong" <keith@e-magine.com.au>; <pgsql-sql@postgresql.org>
Sent: Tuesday, August 15, 2000 11:37 AM
Subject: Re: [SQL] 8K Limit, whats the best strategy?
> At 06:29 AM 8/15/2000, Keith Wong wrote:
> >Hi everyone,
> >
> >Just wondering what strategies people have used to get around the 8K row
> >limit in Postgres.
> >If anyone has been troubled by this limitation before and has a nice
> >solution around it, I would love to hear it.
> >
> >My application is a web-based system in which needs to store large
amounts
> >of text.
> >The text stored needs to searchable as well.
>
> You can you the type LZTEXT which is compressed text so your mileage will
vary.
>
> The other option is to go in and recompile for a larger row size. Most of
> the people I have talked to had no problems with this route. I move the
> size limit on an x86 Linux box to 32K (the max) and have not had any
> problems. In the next release, I don't believe the row size limit will
> exist (correct me if I'm wrong).
>
> Let me know the system your on, and I will try to help if I can.
>
> -
> - Thomas Swan
> - Graduate Student - Computer Science
> - The University of Mississippi
> -
> - "People can be categorized into two fundamental
> - groups, those that divide people into two groups
> - and those that don't."
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: 8K Limit, whats the best strategy?
@ 2000-08-15 17:49 Poul L. Christiansen <plc@faroenet.fo>
parent: Keith Wong <keith@e-magine.com.au>
0 siblings, 1 reply; 7+ messages in thread
From: Poul L. Christiansen @ 2000-08-15 17:49 UTC (permalink / raw)
To: Keith Wong <keith@e-magine.com.au>; +Cc: pgsql-sql
I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
only a viable solution if your text is less than a couple of 100K's.
You could try to be a daredevil and use the Toast code, even if it's beta. But
I don't know how far the Toast project has come.
Keith Wong wrote:
> Hi everyone,
>
> Just wondering what strategies people have used to get around the 8K row
> limit in Postgres.
> If anyone has been troubled by this limitation before and has a nice
> solution around it, I would love to hear it.
>
> My application is a web-based system in which needs to store large amounts
> of text.
> The text stored needs to searchable as well.
>
> Cheers,
> Keith.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: 8K Limit, whats the best strategy?
@ 2000-08-21 15:20 Poul L. Christiansen <plc@faroenet.fo>
parent: Jan Wieck <janwieck@Yahoo.com>
0 siblings, 1 reply; 7+ messages in thread
From: Poul L. Christiansen @ 2000-08-21 15:20 UTC (permalink / raw)
To: Jan Wieck <janwieck@Yahoo.com>; +Cc: Keith Wong <keith@e-magine.com.au>; pgsql-sql
Jan Wieck wrote:
> Poul L. Christiansen wrote:
> > I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
> > only a viable solution if your text is less than a couple of 100K's.
> >
> > You could try to be a daredevil and use the Toast code, even if it's beta. But
> > I don't know how far the Toast project has come.
>
> TOAST is finished and will be shipped with 7.1. It's not a
> solution for huge items, but medium sized text up to some
> hundred K works fine.
What do you mean by "..not a solution for huge items"? Does TOAST have a size limit?
>
>
> Jan
>
> >
> > Keith Wong wrote:
> >
> > > Hi everyone,
> > >
> > > Just wondering what strategies people have used to get around the 8K row
> > > limit in Postgres.
> > > If anyone has been troubled by this limitation before and has a nice
> > > solution around it, I would love to hear it.
> > >
> > > My application is a web-based system in which needs to store large amounts
> > > of text.
> > > The text stored needs to searchable as well.
> > >
> > > Cheers,
> > > Keith.
> >
>
> --
>
> #======================================================================#
> # It's easier to get forgiveness for being wrong than for being right. #
> # Let's break this rule - forgive me. #
> #================================================== JanWieck@Yahoo.com #
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: 8K Limit, whats the best strategy?
@ 2000-08-21 15:41 Jan Wieck <janwieck@Yahoo.com>
parent: Poul L. Christiansen <plc@faroenet.fo>
0 siblings, 1 reply; 7+ messages in thread
From: Jan Wieck @ 2000-08-21 15:41 UTC (permalink / raw)
To: Poul L. Christiansen <plc@faroenet.fo>; +Cc: Keith Wong <keith@e-magine.com.au>; pgsql-sql
Poul L. Christiansen wrote:
> I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
> only a viable solution if your text is less than a couple of 100K's.
>
> You could try to be a daredevil and use the Toast code, even if it's beta. But
> I don't know how far the Toast project has come.
TOAST is finished and will be shipped with 7.1. It's not a
solution for huge items, but medium sized text up to some
hundred K works fine.
Jan
>
> Keith Wong wrote:
>
> > Hi everyone,
> >
> > Just wondering what strategies people have used to get around the 8K row
> > limit in Postgres.
> > If anyone has been troubled by this limitation before and has a nice
> > solution around it, I would love to hear it.
> >
> > My application is a web-based system in which needs to store large amounts
> > of text.
> > The text stored needs to searchable as well.
> >
> > Cheers,
> > Keith.
>
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: 8K Limit, whats the best strategy?
@ 2000-08-21 20:47 Jan Wieck <janwieck@Yahoo.com>
parent: Poul L. Christiansen <plc@faroenet.fo>
0 siblings, 0 replies; 7+ messages in thread
From: Jan Wieck @ 2000-08-21 20:47 UTC (permalink / raw)
To: pgsql-sql; +Cc: Jan Wieck <janwieck@Yahoo.com>; Keith Wong <keith@e-magine.com.au>
Poul L. Christiansen wrote:
> Jan Wieck wrote:
>
> > Poul L. Christiansen wrote:
> > > I've just split the text up (in 8 pieces), so it fits into 8K rows. But thats
> > > only a viable solution if your text is less than a couple of 100K's.
> > >
> > > You could try to be a daredevil and use the Toast code, even if it's beta. But
> > > I don't know how far the Toast project has come.
> >
> > TOAST is finished and will be shipped with 7.1. It's not a
> > solution for huge items, but medium sized text up to some
> > hundred K works fine.
>
> What do you mean by "..not a solution for huge items"? Does TOAST have a size limit?
>
Not an explicit one. But imagine you really want to store an
MP3 of - let's say 9M in the database.
1. Your client application must quote it somehow to put it
into an INSERT querystring. The quoting makes it a 10MB
thing (think positive).
2. The query is sent to the backend. Now you have this
string a second time in memory.
3. The query is parsed and a 10MB text datum is built for
the querytrees Const node.
4. The query is executed, builds a 10MB tuple to store.
5. Toast jumps in and moves it out of the tuple again.
So if the client and DB are running on the same system,
you'll have the MP3 four times in memory. Now do it
concurrent with 20 backends and you'll need 800+ ... you see
why I said it's "not a solution for huge"?
Jan
--
#======================================================================#
# It's easier to get forgiveness for being wrong than for being right. #
# Let's break this rule - forgive me. #
#================================================== JanWieck@Yahoo.com #
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2000-08-21 20:47 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2000-08-15 11:29 ` Keith Wong <keith@e-magine.com.au>
2000-08-15 17:49 ` Poul L. Christiansen <plc@faroenet.fo>
2000-08-21 15:41 ` Jan Wieck <janwieck@Yahoo.com>
2000-08-21 15:20 ` Poul L. Christiansen <plc@faroenet.fo>
2000-08-21 20:47 ` Jan Wieck <janwieck@Yahoo.com>
2000-08-15 15:37 ` Thomas Swan <tswan@olemiss.edu>
2000-08-15 15:48 ` Adam Lang <aalang@rutgersinsurance.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox