agora inbox for pgsql-interfaces@postgresql.org
help / color / mirror / Atom feedUsing threads in FDW for read-ahead
3+ messages / 3 participants
[nested] [flat]
* Using threads in FDW for read-ahead
@ 2014-06-10 19:10 Noah Watkins <noahwatkins@gmail.com>
0 siblings, 2 replies; 3+ messages in thread
From: Noah Watkins @ 2014-06-10 19:10 UTC (permalink / raw)
To: pgsql-interfaces
I have created a FDW for a storage backend and it is working well, and now
I would like to overlap processing with I/O by performing read-ahead. I
started by using a thread to do background I/O and this worked, but
problems started to arise when I tried to do predicate filtering in the
thread.
In particular, it seems as though `check_stack_depth` is built to assume a
single threaded environment (`stack_base_ptr` is global).
I'm wondering if there is a solution to this problem, or if there are
examples of overlapping tuple I/O and predicate filtering using
non-multithreading techniques?
Thanks,
Noah
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Using threads in FDW for read-ahead
@ 2014-06-10 19:21 Tom Lane <tgl@sss.pgh.pa.us>
parent: Noah Watkins <noahwatkins@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: Tom Lane @ 2014-06-10 19:21 UTC (permalink / raw)
To: Noah Watkins <noahwatkins@gmail.com>; +Cc: pgsql-interfaces
Noah Watkins <noahwatkins@gmail.com> writes:
> I have created a FDW for a storage backend and it is working well, and now
> I would like to overlap processing with I/O by performing read-ahead. I
> started by using a thread to do background I/O and this worked, but
> problems started to arise when I tried to do predicate filtering in the
> thread.
> In particular, it seems as though `check_stack_depth` is built to assume a
> single threaded environment (`stack_base_ptr` is global).
That is not even the tip of the iceberg of what will break if you try
to use multiple threads in a Postgres backend. It's not supported.
You might possibly manage to not break things if you keep the extra
threads sufficiently narrowly scoped --- which for starters would include
no use of palloc nor elog. Executing query predicates is right out.
regards, tom lane
--
Sent via pgsql-interfaces mailing list (pgsql-interfaces@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-interfaces
^ permalink raw reply [nested|flat] 3+ messages in thread
* Re: Using threads in FDW for read-ahead
@ 2014-06-10 19:23 Ian Pye <ianpye@gmail.com>
parent: Noah Watkins <noahwatkins@gmail.com>
1 sibling, 0 replies; 3+ messages in thread
From: Ian Pye @ 2014-06-10 19:23 UTC (permalink / raw)
To: Noah Watkins <noahwatkins@gmail.com>; +Cc: pgsql-interfaces
I've done this, but I had to resort to using middleware outside of the FWD.
On Tue, Jun 10, 2014 at 12:10 PM, Noah Watkins <noahwatkins@gmail.com>
wrote:
> I have created a FDW for a storage backend and it is working well, and now
> I would like to overlap processing with I/O by performing read-ahead. I
> started by using a thread to do background I/O and this worked, but
> problems started to arise when I tried to do predicate filtering in the
> thread.
>
> In particular, it seems as though `check_stack_depth` is built to assume a
> single threaded environment (`stack_base_ptr` is global).
>
> I'm wondering if there is a solution to this problem, or if there are
> examples of overlapping tuple I/O and predicate filtering using
> non-multithreading techniques?
>
> Thanks,
> Noah
>
^ permalink raw reply [nested|flat] 3+ messages in thread
end of thread, other threads:[~2014-06-10 19:23 UTC | newest]
Thread overview: 3+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2014-06-10 19:10 Using threads in FDW for read-ahead Noah Watkins <noahwatkins@gmail.com>
2014-06-10 19:21 ` Tom Lane <tgl@sss.pgh.pa.us>
2014-06-10 19:23 ` Ian Pye <ianpye@gmail.com>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox