Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtp (Exim 4.80) (envelope-from ) id 1ZkALw-00085T-D6 for pgsql-performance@arkaria.postgresql.org; Thu, 08 Oct 2015 12:31:20 +0000 Received: from localhost ([127.0.0.1] helo=postgresql.org) by malur.postgresql.org with smtp (Exim 4.84) (envelope-from ) id 1ZkALv-0001Hi-QZ for pgsql-performance@arkaria.postgresql.org; Thu, 08 Oct 2015 12:31:19 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZkAKF-0007oV-SJ for pgsql-performance@postgresql.org; Thu, 08 Oct 2015 12:29:36 +0000 Received: from mail.tivas.info ([85.234.217.139]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.84) (envelope-from ) id 1ZkAKB-0004me-LT for pgsql-performance@postgresql.org; Thu, 08 Oct 2015 12:29:34 +0000 Received: from localhost.localdomain ([192.168.3.104]) by mail.tivas.info (8.14.7/8.14.7) with ESMTP id t98CTSK0001479 for ; Thu, 8 Oct 2015 14:29:28 +0200 (CEST) (envelope-from bram@diomedia.be) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=diomedia.be; s=mail; t=1444307369; bh=uS7jg0q/4jyq7TsfiGJGPLETP0cA6xfmTfbo/F53xWw=; h=Date:From:To:Subject:References:In-Reply-To; b=Jm5mbJ9DQVaPL0erKtF0MxdU4OBWAV4vCh4VBShRaBC53OdO/nuFh6RPzIp+6ak3U zKaT4Qcs63xBIWjub2UnmffJP/VJjLFxRawWIdNfVnB55EbkDvMI5fz6Ev1OvGL737 AIcoAxVhHmjGbLelri6tvXveVjckrbJb9K/1YUuo= Message-ID: <561661A8.90301@diomedia.be> Date: Thu, 08 Oct 2015 14:29:28 +0200 From: Bram Van Steenlandt User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:31.0) Gecko/20100101 Thunderbird/31.6.0 MIME-Version: 1.0 To: pgsql-performance@postgresql.org Subject: Re: large object write performance References: <561634BD.2000309@diomedia.be> <3D2F06B8-297D-48CF-AA8C-47FD70925863@skogoglandskap.no> <56164825.6050209@diomedia.be> <5616588B.5000308@diomedia.be> <68E43830-5272-4AE7-AAF5-3887E2EB3C01@skogoglandskap.no> In-Reply-To: <68E43830-5272-4AE7-AAF5-3887E2EB3C01@skogoglandskap.no> Content-Type: multipart/alternative; boundary="------------070304080805070706050409" X-MessageSniffer-RulebaseUTC: 20150811174000 X-MessageSniffer-Identifier: <561661A8.90301@diomedia.be> X-GBUdb-Analysis: 0, 192.168.3.104, Ugly c=0.40914 p=-0.509434 Source Normal X-MessageSniffer-Scan-Result: 0 X-MessageSniffer-Rules: 0-0-0-13993-c X-Pg-Spam-Score: -2.0 (--) List-Archive: List-Help: List-ID: List-Owner: List-Post: List-Subscribe: List-Unsubscribe: X-Mailing-List: pgsql-performance Precedence: bulk Sender: pgsql-performance-owner@postgresql.org This is a multi-part message in MIME format. --------------070304080805070706050409 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Op 08-10-15 om 14:10 schreef Graeme B. Bell: >> On 08 Oct 2015, at 13:50, Bram Van Steenlandt wrote: >>>> 1. The part is "fobj = lobject(db.db,0,"r",0,fpath)", I don't think there is anything there > Re: lobject > > http://initd.org/psycopg/docs/usage.html#large-objects > > "Psycopg large object support *efficient* import/export with file system files using the lo_import() and lo_export() libpq functions.” > > See * I was under the impression they meant that the lobject was using lo_import and lo_export. I can't seem to find how to use lo_import en export, I searched google and came to the conclusion the lobject was the way to go. >>> x.lo_import() Traceback (most recent call last): File "", line 1, in AttributeError: 'psycopg2._psycopg.connection' object has no attribute 'lo_import' >>> from psycopg2.extensions import lo_importTraceback (most recent call last): File "", line 1, in ImportError: cannot import name lo_import Also: http://initd.org/psycopg/docs/connection.html |lobject|([/oid/[, /mode/[, /new_oid/[, /new_file/[, /lobject_factory/]]]]]) Return a new database large object as a |lobject| instance. See Access to PostgreSQL large objects for an overview. Parameters: * *oid* – The OID of the object to read or write. 0 to create a new large object and and have its OID assigned automatically. * *mode* – Access mode to the object, see below. * *new_oid* – Create a new object using the specified OID. The function raises |OperationalError| if the OID is already in use. Default is 0, meaning assign a new one automatically. * *new_file* – The name of a file to be imported in the the database (using the |lo_import()| function) * *lobject_factory* – Subclass of |lobject| to be instantiated. > > lobject seems to default to string handling in Python > That’s going to be slow. > Try using lo_import / export? > > Graeme Bell > --------------070304080805070706050409 Content-Type: text/html; charset=windows-1252 Content-Transfer-Encoding: 8bit

Op 08-10-15 om 14:10 schreef Graeme B. Bell:

      

        
On 08 Oct 2015, at 13:50, Bram Van Steenlandt <bram@diomedia.be> wrote:


        

          

            
1. The part is "fobj = lobject(db.db,0,"r",0,fpath)", I don't think there is anything there


          

        

      

      
Re: lobject

http://initd.org/psycopg/docs/usage.html#large-objects

"Psycopg large object support *efficient* import/export with file system files using the lo_import() and lo_export() libpq functions.”

See *

    

    I was under the impression they meant that the lobject was using
    lo_import and lo_export.

    I can't seem to find how to use lo_import en export, I searched
    google and came to the conclusion the lobject was the way to go.

    >>> x.lo_import()

    Traceback (most recent call last):

      File "<stdin>", line 1, in <module>

    AttributeError: 'psycopg2._psycopg.connection' object has no
    attribute 'lo_import'

    

    >>> from psycopg2.extensions import lo_importTraceback
    (most recent call last):

      File "<stdin>", line 1, in <module>

    ImportError: cannot import name lo_import

    

    Also:

    http://initd.org/psycopg/docs/connection.html

    

      

        lobject([oid[,
        mode[, new_oid[, new_file[, lobject_factory]]]]])

      

        
Return a new database large object as a lobject
          instance.

        
See Access
              to PostgreSQL large objects for an overview.

        

          
            
          
            

              Parameters:
              
                
    
                  
  • oid – The OID of the object to
                    read or write. 0 to create
                    a new large object and and have its OID assigned
                    automatically.
    • 
                  
  • mode – Access mode to the object,
                    see below.
    • 
                  
  • new_oid – Create a new object
                    using the specified OID. The
                    function raises OperationalError
                    if the OID is already
                    in use. Default is 0, meaning assign a new one
                    automatically.
    • 
                  
  • new_file – The name of a file to
                    be imported in the the database
                    (using the lo_import()
                    function)
    • 
                  
  • lobject_factory – Subclass of
                    lobject to be
                    instantiated.
    • 
                

              		
            

          
        

      

    

    

    

      

lobject seems to default to string handling in Python
That’s going to be slow.
Try using lo_import / export?

Graeme Bell



    

    

  


--------------070304080805070706050409--