From: Ross J. Reedstrom <reedstrm@rice.edu>
To: pgsql-performance <pgsql-performance@postgresql.org>
Subject: Re: TCP network cost
Date: Mon, 23 Feb 2009 13:43:25 -0600
Message-ID: <20090223194325.GB14802@cooker> (raw)
In-Reply-To: <op.upln5emacigqcu@soyouz>
References: <20090217070403.GB2486@cooker>
<1ca1c1410902171013q478f2620n13fcd68cb8b3d911@mail.gmail.com>
<3FAF940B-9B0B-4DBA-8728-6096022129DF@infogears.com>
<alpine.DEB.1.10.0902170032440.25606@asgard.lang.hm>
<20090217200416.GA23707@cooker>
<B7A2E4DA-A137-46F5-8643-2EF3772F4D2C@infogears.com>
<20090217211454.GB23707@cooker>
<op.upln5emacigqcu@soyouz>
On Thu, Feb 19, 2009 at 02:09:04PM +0100, PFC wrote:
>
> >python w/ psycopg (or psycopg2), which wraps libpq. Same results w/
> >either version.
>
> I've seen psycopg2 saturate a 100 Mbps ethernet connection (direct
> connection with crossover cable) between postgres server and client during
> a benchmark... I had to change the benchmark to not retrieve a large TEXT
> column to remove this bottleneck... this was last year so versions are
> probably different, but I don't think this matters a lot...
Here's the core of the problem: I in fact need to transfer exactly that:
a large single field (bytea in my case). I suspect psycopg[12] is having
issues w/ memory allocation, but that's just an unsupported gut feeling.
The final upshot is that I need to restructure my config to use the
large-object API (and hence a snapshot of psycopg2) to get decent
throughput.
> You should test with sending a large (>100 MB) amount of data
> through Netcat. This should give you your maximum wire speed. Use
> /dev/null as the test file, and use "pv" (pipe viewer) to measure
> throughput :
>
> box 1 : pv < /dev/zero | nc -lp 12345
> box 2 : nc (ip) 12345 >/dev/null
>
> On gigabit lan you should get 100 MB/s, on 100BaseT about 10 MB/s.
112 MB/s, and 233 MB/s for localhost. Thanks for the pointer to pv:
looks like a nice tool. Investigating this problem has lead me to a
number of nice 'old school' tools: the other is tcptrace and xplot.org.
I've been hand reading tcpdump output, or clicking around in
ethereal/wireshark. I like tcptrace's approach.
Ross
--
Ross Reedstrom, Ph.D. reedstrm@rice.edu
Systems Engineer & Admin, Research Scientist phone: 713-348-6166
The Connexions Project http://cnx.org fax: 713-348-3665
Rice University MS-375, Houston, TX 77005
GPG Key fingerprint = F023 82C8 9B0E 2CC6 0D8E F888 D3AE 810E 88F0 BEDE
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-performance@postgresql.org
Cc: reedstrm@rice.edu
Subject: Re: TCP network cost
In-Reply-To: <20090223194325.GB14802@cooker>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox