Received: from actarg.com (host25.actarg.com [209.180.91.25] (may be forged)) by mail.postgresql.org (8.11.1/8.11.1) with ESMTP id eBDFSZp12094 for ; Wed, 13 Dec 2000 10:28:35 -0500 (EST) (envelope-from kyle@actarg.com) Received: from chi.actarg.com (IDENT:root@chi.actarg.com [192.168.2.16]) by actarg.com (8.9.3/8.9.3) with ESMTP id IAA20086; Wed, 13 Dec 2000 08:29:01 -0700 Received: from actarg.com (IDENT:kyle@localhost [127.0.0.1]) by chi.actarg.com (8.9.3/8.9.3) with ESMTP id IAA21016; Wed, 13 Dec 2000 08:29:00 -0700 Message-ID: <3A3795BC.44C91514@actarg.com> Date: Wed, 13 Dec 2000 08:29:00 -0700 From: Kyle Organization: Action Target Inc. X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: pgsql-sql@postgresql.org CC: JanWieck@Yahoo.com Subject: Strange slow behavior in backend Content-Type: multipart/mixed; boundary="------------2AF57863506E9D5EAB861653" X-Archive-Number: 200012/106 X-Sequence-Number: 440 This is a multi-part message in MIME format. --------------2AF57863506E9D5EAB861653 Content-Type: multipart/alternative; boundary="------------68D3014F1402D0E7CCDB4EC0" --------------68D3014F1402D0E7CCDB4EC0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit I'm using 7.0.1 with a TCL frontend. I have a schema that splits large files into tuple-sized bites and stores them in a table. This was done before TOAST in order to store large files. I have a backend TCL function that re-assembles the file like this: -- Fetch the specified document data, reassembling the bits back together -- in the right order. -- Calling sequence: cont_doc_fetch(crt_by,crt_date,ctype) create function cont_doc_fetch(int4,timestamp,varchar) returns text as ' set odata {} spi_exec -array d "select data from cont_doc_data where crt_by = \'$1\' and crt_date = \'$2\' and ctype = \'[quote $3]\' order by seq" { append odata $d(data) } return $odata ' LANGUAGE 'pltcl'; This worked great until I put a real big file in (about 5M). Then, when I tried to fetch the file, it seemed really slow (about 60 seconds). I tried reassembling the file in the frontend instead and my time dropped to about 6 seconds using this TCL fragment (mpg::qlist is an interface to pg_exec that returns a list of tuples): set data {} set tuple_list [mpg::qlist "select data from $ca(prefix)_doc_data where crt_by = $crt_by and crt_date = '$crt_date' and ctype = '$ctype' order by seq"] foreach rec $tuple_list { append data [lindex $rec 0] } The only difference I can identify is whether the re-assembly TCL code is running as a procedural language (backend) or in the frontend. Anyone have any idea why the difference is so dramatic? Jan: Is this the difference between old TCL and new TCL (with multi-port objects)? Or is there something else about the way the backend handles large chunks of data that would mark the difference? --------------68D3014F1402D0E7CCDB4EC0 Content-Type: text/html; charset=us-ascii Content-Transfer-Encoding: 7bit I'm using 7.0.1 with a TCL frontend.

I have a schema that splits large files into tuple-sized bites and stores them in a table.  This was done before TOAST in order to store large files.

I have a backend TCL function that re-assembles the file like this:

-- Fetch the specified document data, reassembling the bits back together
-- in the right order.
-- Calling sequence: cont_doc_fetch(crt_by,crt_date,ctype)
create function cont_doc_fetch(int4,timestamp,varchar) returns text as '
    set odata {}
    spi_exec -array d "select data from cont_doc_data where crt_by = \'$1\' and crt_date = \'$2\' and ctype = \'[quote $3]\' order by seq" {
        append odata $d(data)
    }
    return $odata
    ' LANGUAGE 'pltcl';

This worked great until I put a real big file in (about 5M).  Then, when I tried to fetch the file, it seemed really slow (about 60 seconds).  I tried reassembling the file in the frontend instead and my time dropped to about 6 seconds using this TCL fragment (mpg::qlist is an interface to pg_exec that returns a list of tuples):

    set data {}
    set tuple_list [mpg::qlist "select data from $ca(prefix)_doc_data where crt_by = $crt_by and crt_date = '$crt_date' and ctype = '$ctype' order by seq"]
    foreach rec $tuple_list {
        append data [lindex $rec 0]
    }

The only difference I can identify is whether the re-assembly TCL code is running as a procedural language (backend) or in the frontend.  Anyone have any idea why the difference is so dramatic?

Jan:
Is this the difference between old TCL and new TCL (with multi-port objects)?  Or is there something else about the way the backend handles large chunks of data that would mark the difference?

--------------68D3014F1402D0E7CCDB4EC0-- --------------2AF57863506E9D5EAB861653 Content-Type: text/x-vcard; charset=us-ascii; name="kyle.vcf" Content-Transfer-Encoding: 7bit Content-Description: Card for Kyle Content-Disposition: attachment; filename="kyle.vcf" begin:vcard n:Bateman;Kyle x-mozilla-html:FALSE org:Action Target Inc. adr:;;;;;; version:2.1 email;internet:kyle@actarg.com title:President x-mozilla-cpt:;0 fn:Kyle Bateman end:vcard --------------2AF57863506E9D5EAB861653--