pg.ddx.io pgsql-admin@postgresql.org mailing list archive
help / color / mirror / Atom feedexceeded MAX_ALLOCATED_DESCS while trying to open file
7+ messages / 5 participants
[nested] [flat]
* exceeded MAX_ALLOCATED_DESCS while trying to open file
@ 2013-05-28 15:28 Steve.Toutant@inspq.qc.ca
2013-05-28 15:48 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file ktm@rice.edu <ktm@rice.edu>
2013-06-09 17:49 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 7+ messages in thread
From: Steve.Toutant@inspq.qc.ca @ 2013-05-28 15:28 UTC (permalink / raw)
To: pgsql-admin
Hi,
Using postgres 9.2, I created a FOREIGN TABLE on a csv file. It works.
But I need to change the structure of the data and I created several sql
select using union, like
select .... from .foreign table... where...
union
select .... from .foreign table... where...
union
...
It works for the first 10 select but then I get
ERROR: exceeded MAX_ALLOCATED_DESCS while trying to open file
"/srv/www/data/antidote/registre.csv"
What is it MAX_ALLOCATED_DESCS? and how can I modify it without changing
the code and re-compile? I tried using postgref.conf but this parameters
is not recognize.
thanks
steve
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: exceeded MAX_ALLOCATED_DESCS while trying to open file
2013-05-28 15:28 exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
@ 2013-05-28 15:48 ` ktm@rice.edu <ktm@rice.edu>
2013-05-28 15:56 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file Igor Neyman <ineyman@perceptron.com>
2013-05-28 16:06 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
2013-05-28 16:33 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file Alvaro Herrera <alvherre@2ndquadrant.com>
1 sibling, 3 replies; 7+ messages in thread
From: ktm@rice.edu @ 2013-05-28 15:48 UTC (permalink / raw)
To: Steve.Toutant@inspq.qc.ca; +Cc: pgsql-admin
On Tue, May 28, 2013 at 11:28:02AM -0400, Steve.Toutant@inspq.qc.ca wrote:
> Hi,
> Using postgres 9.2, I created a FOREIGN TABLE on a csv file. It works.
> But I need to change the structure of the data and I created several sql
> select using union, like
> select .... from .foreign table... where...
> union
> select .... from .foreign table... where...
> union
> ...
> It works for the first 10 select but then I get
> ERROR: exceeded MAX_ALLOCATED_DESCS while trying to open file
> "/srv/www/data/antidote/registre.csv"
>
> What is it MAX_ALLOCATED_DESCS? and how can I modify it without changing
> the code and re-compile? I tried using postgref.conf but this parameters
> is not recognize.
> thanks
> steve
Hi Steve,
Each SELECT opens your FOREIGN TABLE so you are opening it 10 times. This is
hardcoded by a #define in ./src/backend/storage/file/fd.c during the build so
you would need to recompile the software yourself to change it. Can you re-
write your query using a single SELECT or possibly read the data into a
temporary table for processing?
Regards,
Ken
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: exceeded MAX_ALLOCATED_DESCS while trying to open file
2013-05-28 15:28 exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
2013-05-28 15:48 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file ktm@rice.edu <ktm@rice.edu>
@ 2013-05-28 15:56 ` Igor Neyman <ineyman@perceptron.com>
2 siblings, 0 replies; 7+ messages in thread
From: Igor Neyman @ 2013-05-28 15:56 UTC (permalink / raw)
To: ktm@rice.edu <ktm@rice.edu>; Steve.Toutant@inspq.qc.ca <Steve.Toutant@inspq.qc.ca>; +Cc: pgsql-admin
>
> Hi Steve,
>
> Each SELECT opens your FOREIGN TABLE so you are opening it 10 times.
> This is hardcoded by a #define in ./src/backend/storage/file/fd.c
> during the build so you would need to recompile the software yourself
> to change it. Can you re- write your query using a single SELECT or
> possibly read the data into a temporary table for processing?
>
> Regards,
> Ken
>
Hi Ken,
So, it means that number of (different) foreign tables joined in single select cannot exceed 10, right?
Regards,
Igor Neyman
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: exceeded MAX_ALLOCATED_DESCS while trying to open file
2013-05-28 15:28 exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
2013-05-28 15:48 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file ktm@rice.edu <ktm@rice.edu>
@ 2013-05-28 16:06 ` Steve.Toutant@inspq.qc.ca
2 siblings, 0 replies; 7+ messages in thread
From: Steve.Toutant@inspq.qc.ca @ 2013-05-28 16:06 UTC (permalink / raw)
To: ktm@rice.edu <ktm@rice.edu>; +Cc: pgsql-admin
The csv has 34 columns and I need to create a crosstab (using 8 columns so
I can't use built in crosstab functions). That is why I need to make
several sql select.
I'm using foreign table so my client can update is csv file without having
to run the ETL to update a table in postgres.
Using temporary table means I got to know when the csv is updated...unless
there is something I misunderstood.
Perhaps the solution is to automate the process to copy the csv in a
table.....
"ktm@rice.edu" <ktm
2013-05-28 11:48
A
Steve.Toutant@inspq.qc.ca
cc
pgsql-admin@postgresql.org
Objet
Re: [ADMIN] exceeded MAX_ALLOCATED_DESCS while trying to open file
On Tue, May 28, 2013 at 11:28:02AM -0400, Steve.Toutant@inspq.qc.ca wrote:
> Hi,
> Using postgres 9.2, I created a FOREIGN TABLE on a csv file. It works.
> But I need to change the structure of the data and I created several sql
> select using union, like
> select .... from .foreign table... where...
> union
> select .... from .foreign table... where...
> union
> ...
> It works for the first 10 select but then I get
> ERROR: exceeded MAX_ALLOCATED_DESCS while trying to open file
> "/srv/www/data/antidote/registre.csv"
>
> What is it MAX_ALLOCATED_DESCS? and how can I modify it without changing
> the code and re-compile? I tried using postgref.conf but this parameters
> is not recognize.
> thanks
> steve
Hi Steve,
Each SELECT opens your FOREIGN TABLE so you are opening it 10 times. This
is
hardcoded by a #define in ./src/backend/storage/file/fd.c during the build
so
you would need to recompile the software yourself to change it. Can you
re-
write your query using a single SELECT or possibly read the data into a
temporary table for processing?
Regards,
Ken
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: exceeded MAX_ALLOCATED_DESCS while trying to open file
2013-05-28 15:28 exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
2013-05-28 15:48 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file ktm@rice.edu <ktm@rice.edu>
@ 2013-05-28 16:33 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2013-06-01 05:07 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file Tom Lane <tgl@sss.pgh.pa.us>
2 siblings, 1 reply; 7+ messages in thread
From: Alvaro Herrera @ 2013-05-28 16:33 UTC (permalink / raw)
To: ktm@rice.edu <ktm@rice.edu>; +Cc: Steve.Toutant@inspq.qc.ca; pgsql-admin
ktm@rice.edu wrote:
> Each SELECT opens your FOREIGN TABLE so you are opening it 10 times. This is
> hardcoded by a #define in ./src/backend/storage/file/fd.c during the build so
> you would need to recompile the software yourself to change it. Can you re-
> write your query using a single SELECT or possibly read the data into a
> temporary table for processing?
To me this sounds like a bug in whatever FDW is being used.
--
Álvaro Herrera http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training & Services
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: exceeded MAX_ALLOCATED_DESCS while trying to open file
2013-05-28 15:28 exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
2013-05-28 15:48 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file ktm@rice.edu <ktm@rice.edu>
2013-05-28 16:33 ` Re: exceeded MAX_ALLOCATED_DESCS while trying to open file Alvaro Herrera <alvherre@2ndquadrant.com>
@ 2013-06-01 05:07 ` Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 0 replies; 7+ messages in thread
From: Tom Lane @ 2013-06-01 05:07 UTC (permalink / raw)
To: Alvaro Herrera <alvherre@2ndquadrant.com>; +Cc: ktm@rice.edu <ktm@rice.edu>; Steve.Toutant@inspq.qc.ca; pgsql-admin
Alvaro Herrera <alvherre@2ndquadrant.com> writes:
> ktm@rice.edu wrote:
>> Each SELECT opens your FOREIGN TABLE so you are opening it 10 times. This is
>> hardcoded by a #define in ./src/backend/storage/file/fd.c during the build so
>> you would need to recompile the software yourself to change it. Can you re-
>> write your query using a single SELECT or possibly read the data into a
>> temporary table for processing?
> To me this sounds like a bug in whatever FDW is being used.
Yeah, we need to fix file_fdw so that it's using virtual rather than
physical file descriptors ...
regards, tom lane
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: exceeded MAX_ALLOCATED_DESCS while trying to open file
2013-05-28 15:28 exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
@ 2013-06-09 17:49 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 7+ messages in thread
From: Tom Lane @ 2013-06-09 17:49 UTC (permalink / raw)
To: Steve.Toutant@inspq.qc.ca; +Cc: pgsql-admin
Steve.Toutant@inspq.qc.ca writes:
> Using postgres 9.2, I created a FOREIGN TABLE on a csv file. It works.
> But I need to change the structure of the data and I created several sql
> select using union, like
> select .... from .foreign table... where...
> union
> select .... from .foreign table... where...
> union
> ...
> It works for the first 10 select but then I get
> ERROR: exceeded MAX_ALLOCATED_DESCS while trying to open file
> "/srv/www/data/antidote/registre.csv"
FYI, I've committed some changes that will relax this limit in future
releases.
regards, tom lane
--
Sent via pgsql-admin mailing list (pgsql-admin@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-admin
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2013-06-09 17:49 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2013-05-28 15:28 exceeded MAX_ALLOCATED_DESCS while trying to open file Steve.Toutant@inspq.qc.ca
2013-05-28 15:48 ` ktm@rice.edu <ktm@rice.edu>
2013-05-28 15:56 ` Igor Neyman <ineyman@perceptron.com>
2013-05-28 16:06 ` Steve.Toutant@inspq.qc.ca
2013-05-28 16:33 ` Alvaro Herrera <alvherre@2ndquadrant.com>
2013-06-01 05:07 ` Tom Lane <tgl@sss.pgh.pa.us>
2013-06-09 17:49 ` Tom Lane <tgl@sss.pgh.pa.us>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox