From zhengyang4k@gmail.com Mon Mar 21 12:17:22 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 86D84133655C for ; Mon, 21 Mar 2011 09:17:22 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 52365-04 for ; Mon, 21 Mar 2011 12:17:15 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) by mail.postgresql.org (Postfix) with ESMTP id 192AF1337BDC for ; Mon, 21 Mar 2011 09:17:14 -0300 (ADT) Received: by iwn34 with SMTP id 34so6318896iwn.19 for ; Mon, 21 Mar 2011 05:17:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:from:mime-version:content-type:subject:date :in-reply-to:to:references:message-id:x-mailer; bh=CBipFmtHr+Tlsw4Fk60xkayJf4P28iUbi1cJPgdl5Hs=; b=phoJbYHIJ5ZMRBTwbM77MHsgsyVLgIHURRF9lYpAkz9uGw/Cf3ysM0lTSbzRi9Ds7U pP+KtSkVV0Qok2zjxjVIJG/hgoXCYMc3Xfc9rv0+/sVJhjc/6zRl2PPaB9AEds2xDdRZ bHoKEbLkawsqzDG9qFaj+UQGkso9r/xXeoKnw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=from:mime-version:content-type:subject:date:in-reply-to:to :references:message-id:x-mailer; b=Qqdi53AUNZJkegthm72We5h4WOZiZWu4ThXABjVCHfVFk8t17IfQqMkf6mjvGJtF6X gbfXImot3y4MvQF9wHYwNOzoZzOII0N3lCp7LaOcDlrvbwDrxHItFYMz66XFBRxn0p/O XXQfhXT9S8xC+sHrdsvEAe9IbjbT8Moxv6nho= Received: by 10.42.135.72 with SMTP id o8mr6493458ict.443.1300709835329; Mon, 21 Mar 2011 05:17:15 -0700 (PDT) Received: from [122.64.171.54] ([124.68.4.80]) by mx.google.com with ESMTPS id gx2sm1175ibb.26.2011.03.21.05.17.13 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 21 Mar 2011 05:17:14 -0700 (PDT) From: Zheng Yang Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/alternative; boundary=Apple-Mail-1094-280082286 Subject: foreign data wrappers Date: Mon, 21 Mar 2011 20:17:09 +0800 In-Reply-To: To: pgsql-students@postgresql.org References: Message-Id: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.588 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, T_TO_NO_BRKTS_FREEMAIL=0.01 X-Spam-Level: X-Archive-Number: 201103/4 X-Sequence-Number: 145 --Apple-Mail-1094-280082286 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi! in the GSOC wiki, there is a potential project idea mentioning "Write = Foreign Data Wrappers for several external data sources (ODBC, SQL Server, Oracle, MySQL, = CouchDB, Redis, etc.)" Can anyone help explain how these wrapper are going to be like? Thanks! ZHENG Yang= --Apple-Mail-1094-280082286 Content-Transfer-Encoding: 7bit Content-Type: text/html; charset=us-ascii Hi!

in the GSOC wiki, there is a potential project idea mentioning "Write Foreign Data Wrappers
 for several external data sources (ODBC, SQL Server, Oracle, MySQL, CouchDB, Redis, etc.)"

Can anyone help explain how these wrapper are going to be like?

Thanks!

ZHENG Yang
--Apple-Mail-1094-280082286-- From selena@chesnok.com Mon Mar 21 18:33:54 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 8FC2D1337ECD for ; Mon, 21 Mar 2011 15:33:54 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 39770-01-4 for ; Mon, 21 Mar 2011 18:33:47 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from homiemail-a64.g.dreamhost.com (caiajhbdcaid.dreamhost.com [208.97.132.83]) by mail.postgresql.org (Postfix) with ESMTP id 0E82D1337F7E for ; Mon, 21 Mar 2011 13:57:18 -0300 (ADT) Received: from homiemail-a64.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a64.g.dreamhost.com (Postfix) with ESMTP id 8CAED438082 for ; Mon, 21 Mar 2011 09:57:17 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=chesnok.com; h=mime-version :in-reply-to:references:date:message-id:subject:from:to:cc :content-type:content-transfer-encoding; q=dns; s=chesnok.com; b=s16/2QA6Omwrf3Tow/7dE979WKaGOoh6zWjKlO0IJDIDnHDcxRsgUQ8iwfcaC CjcBTcocAcPmD9HUFBDJPBxzJtt3KZNh7oWP5glWZHACsDPOqzr1qIHIfYwFHVdo bGXSrc56418l5lgsfTuYoTNHJyqIcoBRfqHqUI5sACVkTM= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=chesnok.com; h= mime-version:in-reply-to:references:date:message-id:subject:from :to:cc:content-type:content-transfer-encoding; s=chesnok.com; bh=0dmb75r37iIA4d7BUXLaYPY857o=; b=WbkVwV1C1HK07fwSM8scPOuI4k7x mw0/XzS5Qp873OZ7ReID/5q9gaEdeznRSUtaHpGs7a/EWdfAIkGtTt5EeYeCqogw fj4v4tyPr/O4St6pY9xfdY+VtWmNTSo6YZkXl+Aq5+AIDHU7RSQR7Eg+QRXOQYps HAyO6fCauBUEsJg= Received: from mail-yi0-f46.google.com (mail-yi0-f46.google.com [209.85.218.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: selena@chesnok.com) by homiemail-a64.g.dreamhost.com (Postfix) with ESMTPSA id 618D243807E for ; Mon, 21 Mar 2011 09:57:17 -0700 (PDT) Received: by yia27 with SMTP id 27so2505588yia.19 for ; Mon, 21 Mar 2011 09:57:16 -0700 (PDT) MIME-Version: 1.0 Received: by 10.42.157.193 with SMTP id e1mr6082034icx.178.1300726625973; Mon, 21 Mar 2011 09:57:05 -0700 (PDT) Received: by 10.231.201.199 with HTTP; Mon, 21 Mar 2011 09:57:05 -0700 (PDT) In-Reply-To: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> Date: Mon, 21 Mar 2011 09:57:05 -0700 Message-ID: Subject: Re: foreign data wrappers From: Selena Deckelmann To: Zheng Yang Cc: pgsql-students@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201103/5 X-Sequence-Number: 146 Hi! On Mon, Mar 21, 2011 at 5:17 AM, Zheng Yang wrote: > in the GSOC wiki, there is a potential project idea mentioning "Write > Foreign Data Wrappers > =A0for several external data sources (ODBC, SQL Server, Oracle, MySQL, > CouchDB, Redis, etc.)" > Can anyone help explain how these wrapper are going to be like? Have a look at http://wiki.postgresql.org/images/4/4c/SQLMED-FOSDEM2009.pdf for starters. See if you can review that and come back with more specific questions. There are a few community members interested in seeing this move forward, so this project is likely a great one for GSoC. My suggestion would be to pick just one or two for this summer's project. -selena --=20 http://chesnok.com From andrew@dunslane.net Mon Mar 21 19:33:44 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 0BB181337EF8 for ; Mon, 21 Mar 2011 16:33:44 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 62485-02-2 for ; Mon, 21 Mar 2011 19:33:37 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.123]) by mail.postgresql.org (Postfix) with ESMTP id 5DE281337C40 for ; Mon, 21 Mar 2011 15:45:38 -0300 (ADT) X-Authority-Analysis: v=1.1 cv=r4yJ8ACLDmU9N8MfnU6qGSvboKzSN9UnPAeXToqJDNE= c=1 sm=0 a=YOqqrsh--ucA:10 a=r6Jiw2x_onsA:10 a=8nJEP1OIZ-IA:10 a=qCTYjJJ2g1SgQhsHZAUATw==:17 a=pGLkceISAAAA:8 a=epTmVMiNAAAA:8 a=HjYBxie0M1rm7r0VM04A:9 a=Kg5k_ADc8ylYZVRJYj-uysvFersA:4 a=wPNLvfGTeEIA:10 a=p2bXo2kYDEUA:10 a=MSl-tDqOz04A:10 a=qCTYjJJ2g1SgQhsHZAUATw==:117 X-Cloudmark-Score: 0 X-Originating-IP: 98.27.48.207 Received: from [98.27.48.207] ([98.27.48.207:56637] helo=[192.168.10.104]) by cdptpa-oedge01.mail.rr.com (envelope-from ) (ecelerity 2.2.3.46 r()) with ESMTP id 28/86-09483-1DC978D4; Mon, 21 Mar 2011 18:45:37 +0000 Message-ID: <4D879CD0.5030803@dunslane.net> Date: Mon, 21 Mar 2011 14:45:36 -0400 From: Andrew Dunstan User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc13 Lightning/1.0b3pre Thunderbird/3.1.7 MIME-Version: 1.0 To: Selena Deckelmann CC: Zheng Yang , pgsql-students@postgresql.org Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201103/6 X-Sequence-Number: 147 On 03/21/2011 12:57 PM, Selena Deckelmann wrote: > Hi! > > On Mon, Mar 21, 2011 at 5:17 AM, Zheng Yang wrote: > >> in the GSOC wiki, there is a potential project idea mentioning "Write >> Foreign Data Wrappers >> for several external data sources (ODBC, SQL Server, Oracle, MySQL, >> CouchDB, Redis, etc.)" >> Can anyone help explain how these wrapper are going to be like? > Have a look at http://wiki.postgresql.org/images/4/4c/SQLMED-FOSDEM2009.pdf > for starters. See if you can review that and come back with more > specific questions. > > There are a few community members interested in seeing this move > forward, so this project is likely a great one for GSoC. > > My suggestion would be to pick just one or two for this summer's project. > Yes, but 9.1 *has* actual FDWs. The best thing to do (other than attending my talk on Thursday :-) ) is to look at the file_fdw module in git head, to see an actual working example. cheers andrew From selena@chesnok.com Mon Mar 21 20:07:49 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id EDE691337E99 for ; Mon, 21 Mar 2011 17:07:49 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 70972-01-4 for ; Mon, 21 Mar 2011 20:07:40 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from homiemail-a16.g.dreamhost.com (caiajhbdccac.dreamhost.com [208.97.132.202]) by mail.postgresql.org (Postfix) with ESMTP id BFED11337C18 for ; Mon, 21 Mar 2011 16:50:21 -0300 (ADT) Received: from homiemail-a16.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a16.g.dreamhost.com (Postfix) with ESMTP id 1B0BA508063 for ; Mon, 21 Mar 2011 12:50:21 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=chesnok.com; h=mime-version :in-reply-to:references:date:message-id:subject:from:to:cc: content-type; q=dns; s=chesnok.com; b=Owct1YhseDSK/Aokn1nC1CbyRv fZj2nwFwj3NB9KFunbw9vHIIChnMxxlXgtbr+gVPSQ++QRHIkCoW11ModzC2gMaD KLqf8oE/7xmBioh6aZW72fB+Vp5423r5PvUvxW9oEmbSc9donuuadFm680IFwlQ8 NN+NJ9dPnNVY5ObIE= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=chesnok.com; h= mime-version:in-reply-to:references:date:message-id:subject:from :to:cc:content-type; s=chesnok.com; bh=52wvy1pKtA9ZGWuH+0TW4ZqRl SQ=; b=P1HGTfSW59uckeM67z33AxLdP8sPRo0tiM3WK7uGB93rp5H0Yzy6CbVl/ AilYJpyP+NHIjRgS0zVog7Iwj/P4Usl8+23pZ3H1C5CwO3tJ3df+kwBM/U62DvsU iF3F9cAusy/LIZoYUmW0mosrIa7PQaXc3fDvse9Y5wPQU99jm0= Received: from mail-fx0-f46.google.com (mail-fx0-f46.google.com [209.85.161.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: selena@chesnok.com) by homiemail-a16.g.dreamhost.com (Postfix) with ESMTPSA id B61EB50805B for ; Mon, 21 Mar 2011 12:50:20 -0700 (PDT) Received: by fxm17 with SMTP id 17so5798642fxm.19 for ; Mon, 21 Mar 2011 12:50:19 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.112.83 with SMTP id v19mr5360736fap.122.1300736983628; Mon, 21 Mar 2011 12:49:43 -0700 (PDT) Received: by 10.223.79.14 with HTTP; Mon, 21 Mar 2011 12:49:42 -0700 (PDT) In-Reply-To: <4D879CD0.5030803@dunslane.net> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> Date: Mon, 21 Mar 2011 12:49:42 -0700 Message-ID: Subject: Re: foreign data wrappers From: Selena Deckelmann To: Andrew Dunstan Cc: Zheng Yang , pgsql-students@postgresql.org Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201103/7 X-Sequence-Number: 148 On Mon, Mar 21, 2011 at 11:45 AM, Andrew Dunstan wrote: > Yes, but 9.1 *has* actual FDWs. The best thing to do (other than attending > my talk on Thursday :-) ) :) I'll be around, so hopefully I can attend your talk and post up some notes! > is to look at the file_fdw module in git head, to > see an actual working example. I figured if I posted some wrong information someone would respond! :) Zheng - here is a link to the existing code: http://git.postgresql.org/gitweb?p=postgresql.git;a=tree;f=contrib/file_fdw -selena -- http://chesnok.com From zhengyang4k@gmail.com Tue Mar 22 10:51:35 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 196491337B93 for ; Tue, 22 Mar 2011 07:51:35 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 92039-08 for ; Tue, 22 Mar 2011 10:51:13 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mail.postgresql.org (Postfix) with ESMTP id EBBD91336FA7 for ; Tue, 22 Mar 2011 07:51:12 -0300 (ADT) Received: by iyb26 with SMTP id 26so7319143iyb.19 for ; Tue, 22 Mar 2011 03:51:13 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=l/NGXS13Gvz/DqrmzLKKEl10P5P/eITLIcuYZ5UeJCQ=; b=kfRHGdp8ShZbjxKhEWauDuEiBxs0jPxDNhrXKvyNgPpMvZzNE6ZeqS4ZFJTm7DVOAa YzyDJsoNsu2aM5bsEmgO+lsvzcJLA2KsArf/nY+w0c/15jM8/A6xGoIo4y2Ft3c6AAP3 rkXJLUEh+eR6jbW+mS44pDoh911rNnAY5ltgc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=vySAYA36wSsWc/7+xZZxN79Rxwd9t57uQwu7A47ZAMPjn/tIIIFHjJ9r8Nz8M7H5xJ sxMyzaTopbd7jIgcOZEWlV4Bf1HFoMmfT0kZfA321fGFGpZb/CkvK3BNnob32E3NG0Cd wk0JqaNNbKi8ZGR7/CZl4EYigS292LzrULmv8= Received: by 10.42.149.200 with SMTP id x8mr8606025icv.195.1300791071484; Tue, 22 Mar 2011 03:51:11 -0700 (PDT) Received: from [122.64.165.81] ([61.158.124.164]) by mx.google.com with ESMTPS id xi12sm750668icb.18.2011.03.22.03.51.08 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 22 Mar 2011 03:51:10 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Zheng Yang In-Reply-To: Date: Tue, 22 Mar 2011 18:51:04 +0800 Cc: pgsql-students@postgresql.org Content-Transfer-Encoding: quoted-printable Message-Id: <945917CD-6B5D-4D5C-B02A-41891A1C4755@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> To: Selena Deckelmann , Andrew Dunstan X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201103/8 X-Sequence-Number: 149 Thanks! I will take a look! ZY On 22-Mar-2011, at 3:49 AM, Selena Deckelmann wrote: > On Mon, Mar 21, 2011 at 11:45 AM, Andrew Dunstan = wrote: >=20 >> Yes, but 9.1 *has* actual FDWs. The best thing to do (other than = attending >> my talk on Thursday :-) ) >=20 > :) I'll be around, so hopefully I can attend your talk and post up = some notes! >=20 >> is to look at the file_fdw module in git head, to >> see an actual working example. >=20 > I figured if I posted some wrong information someone would respond! :) >=20 > Zheng - here is a link to the existing code: > = http://git.postgresql.org/gitweb?p=3Dpostgresql.git;a=3Dtree;f=3Dcontrib/f= ile_fdw >=20 > -selena >=20 > --=20 > http://chesnok.com From zhengyang4k@gmail.com Thu Mar 24 05:40:05 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id A77A21337B69 for ; Thu, 24 Mar 2011 02:40:05 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 26800-01 for ; Thu, 24 Mar 2011 05:39:56 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) by mail.postgresql.org (Postfix) with ESMTP id 3D3681337B83 for ; Thu, 24 Mar 2011 02:39:55 -0300 (ADT) Received: by iwn34 with SMTP id 34so9314112iwn.19 for ; Wed, 23 Mar 2011 22:39:55 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to:x-mailer; bh=3AnvduFWk8uiAfZ3hyAzIdEuh9AnajQS0Gg9lputlCg=; b=pPrioNAXjHY4MLWR8be4Sn84alPoTF6zpze3DJeJcAG2nNS1VhR6en7cnA27tNsJGp 3SgIHT+6fZltVYzX5gwHkcY6RvHqMSCjiVjWT6tJxJy9ZveWY9u3192/2dnUsYsZWSuI 8iNyu9Tp2l4gxLsYY7QOA9Q8Y025wfxFp7X5g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; b=paKH3W9uNpvpJ2SINqT3sx7ltW8TW83tcyU4eGxNOL2vlglK3fiEr6e9np7z05YTWR weSWfMM2i8wJaf5PDYz0MqkcS/6gS0ffr+kM0PTVxytMLW1ZkuGMRvs2n0NLPf+qZRCt U9eJzagBW87SKU8pqkZPCJd0CCLxMRooMrpAo= Received: by 10.42.150.65 with SMTP id z1mr12129417icv.503.1300945195195; Wed, 23 Mar 2011 22:39:55 -0700 (PDT) Received: from [122.64.167.70] ([124.68.4.221]) by mx.google.com with ESMTPS id uf10sm5740356icb.17.2011.03.23.22.39.50 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 23 Mar 2011 22:39:53 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/alternative; boundary=Apple-Mail-233-515438408 From: Zheng Yang In-Reply-To: Date: Thu, 24 Mar 2011 13:39:45 +0800 Cc: pgsql-students@postgresql.org Message-Id: <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> To: Selena Deckelmann , Andrew Dunstan X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.598 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201103/19 X-Sequence-Number: 160 --Apple-Mail-233-515438408 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=windows-1252 Hi guys, I've figured out what a FDW does. It's quite clear from the doc that a = FDW is an interface postgresql database used to communicate with other data sources either it be a file, another DBMS or even weblinks = (please correct me if i am wrong). I have a few more specific questions on FDW. hope you guys can help! 1) by right the FDW should be implemented by the vendor of the = datasource, is that correct?=20 2) what are the FDW that has been implemented? I've seen an plain file = FWD module from the link provided by selena. 3) in one of the examples, when declaring an FDW, "..LIBRARY = =92foosql_fdw.so=92 LANGUAGE C;" the LANGUAGE option is going to be C = only? 4) Is theFDW something similar to the Mysql storage engine architecture? 5) for the datalink, the doc has mentioned about it's going to be OS = dependent. but i dun quite understand how does the kernel module and = LD_PRELOAD will work in this case, esp.LD_PRELOAD. That's the question so far. Thanks very much! Regards, Zheng Yang On 22-Mar-2011, at 3:49 AM, Selena Deckelmann wrote: > On Mon, Mar 21, 2011 at 11:45 AM, Andrew Dunstan = wrote: >=20 >> Yes, but 9.1 *has* actual FDWs. The best thing to do (other than = attending >> my talk on Thursday :-) ) >=20 > :) I'll be around, so hopefully I can attend your talk and post up = some notes! >=20 >> is to look at the file_fdw module in git head, to >> see an actual working example. >=20 > I figured if I posted some wrong information someone would respond! :) >=20 > Zheng - here is a link to the existing code: > = http://git.postgresql.org/gitweb?p=3Dpostgresql.git;a=3Dtree;f=3Dcontrib/f= ile_fdw >=20 > -selena >=20 > --=20 > http://chesnok.com --Apple-Mail-233-515438408 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=windows-1252 Hi = guys,

I've figured out what a FDW does. It's quite = clear from the doc that a FDW is an interface postgresql database used = to communicate with
other data sources either it be a file, = another DBMS or even weblinks  (please correct me if i am = wrong).

I have a few more specific questions on = FDW. hope you guys can help!

1) by right the = FDW should be implemented by the vendor of the datasource, is that = correct? 

2) what are the FDW that has = been implemented? I've seen an plain file FWD module from the link = provided by selena.

3) in one of the examples, = when declaring an FDW,  "..LIBRARY =92foosql_fdw.so=92 LANGUAGE = C;"  the LANGUAGE option is going to be C = only?

4) Is theFDW something similar to the = Mysql storage engine architecture?

5) for the = datalink, the doc has mentioned about it's going to be OS dependent. but = i dun quite understand how does the kernel module and LD_PRELOAD will = work in this case, esp.LD_PRELOAD.

That's the = question so far. Thanks very = much!

Regards,
Zheng = Yang



On = 22-Mar-2011, at 3:49 AM, Selena Deckelmann wrote:

On = Mon, Mar 21, 2011 at 11:45 AM, Andrew Dunstan <andrew@dunslane.net> = wrote:

Yes, but 9.1 *has* actual FDWs. = The best thing to do (other than attending
my talk on Thursday :-) )

:) I'll be = around, so hopefully I can attend your talk and post up some = notes!

is to look at the file_fdw = module in git head, to
see an = actual working example.

I figured if I posted some = wrong information someone would respond! :)

Zheng - here is a = link to the existing code:
http://git.postgresql.org/gitweb?p=3Dpostgresql.git;a=3Dt= ree;f=3Dcontrib/file_fdw

-selena

-- =
http://chesnok.com

= --Apple-Mail-233-515438408-- From guillaume@lelarge.info Thu Mar 24 07:54:52 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id AA9B11337B96 for ; Thu, 24 Mar 2011 04:54:52 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 73519-02 for ; Thu, 24 Mar 2011 07:54:43 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sd-17937.dedibox.fr (sd-17937.dedibox.fr [88.191.100.74]) by mail.postgresql.org (Postfix) with ESMTP id 9F8511337B83 for ; Thu, 24 Mar 2011 04:54:43 -0300 (ADT) Received: from [192.168.10.3] (ASte-Genev-Bois-153-1-85-140.w81-48.abo.wanadoo.fr [81.48.35.140]) by sd-17937.dedibox.fr (Postfix) with ESMTPA id BA2D43B8F5; Thu, 24 Mar 2011 08:54:42 +0100 (CET) Message-ID: <4D8AF8C2.3000007@lelarge.info> Date: Thu, 24 Mar 2011 08:54:42 +0100 From: Guillaume Lelarge User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); fr; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Zheng Yang CC: Selena Deckelmann , Andrew Dunstan , pgsql-students@postgresql.org Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> In-Reply-To: <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201103/20 X-Sequence-Number: 161 Hi, Le 24/03/2011 06:39, Zheng Yang a écrit : > [...] > I've figured out what a FDW does. It's quite clear from the doc that a FDW is an interface postgresql database used to communicate with > other data sources either it be a file, another DBMS or even weblinks (please correct me if i am wrong). > You're right. > I have a few more specific questions on FDW. hope you guys can help! > > 1) by right the FDW should be implemented by the vendor of the datasource, is that correct? > If you mean that an Oracle FDW should be implemented by Oracle, and an Excel FDW should be implemented by Microsoft, then no, I don't think so. AFAICS, anyone can write any FDW. > 2) what are the FDW that has been implemented? I've seen an plain file FWD module from the link provided by selena. > Yeah, that's the one provided as a contrib module. I've also seen a postgres one and a twitter one on pgsql-hackers. Andrew wrote two other FDWs he'll talk about today at PGEast (http://people.planetpostgresql.org/andrew/index.php?/archives/163-First-extension.html). I hope he'll publish his slides somewhere really soon :) > 3) in one of the examples, when declaring an FDW, "..LIBRARY ’foosql_fdw.so’ LANGUAGE C;" the LANGUAGE option is going to be C only? > According to http://developer.postgresql.org/pgdocs/postgres/fdwhandler.html, yes. > 4) Is theFDW something similar to the Mysql storage engine architecture? > Not really. > 5) for the datalink, the doc has mentioned about it's going to be OS dependent. but i dun quite understand how does the kernel module and LD_PRELOAD will work in this case, esp.LD_PRELOAD. > No idea. Writing some FDWs would make a really good GSoC project. -- Guillaume http://www.postgresql.fr http://dalibo.com From zhengyang4k@gmail.com Thu Mar 24 08:49:38 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 662481337B96 for ; Thu, 24 Mar 2011 05:49:38 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 72509-08 for ; Thu, 24 Mar 2011 08:49:29 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-pz0-f46.google.com (mail-pz0-f46.google.com [209.85.210.46]) by mail.postgresql.org (Postfix) with ESMTP id 9AAFE1337B49 for ; Thu, 24 Mar 2011 05:49:27 -0300 (ADT) Received: by pzk9 with SMTP id 9so1145902pzk.19 for ; Thu, 24 Mar 2011 01:49:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=IJGzsn0WZGArpaGeZJJtzenP4dPn+js3pE/cj1EO72M=; b=u0kI+OWcxVtJ6q+IcYPJWedbgnnzzi9nyQFRiwSVa5vPRTWa2hzXNbdcOWqhR1wlsH GwF/f/l5K4KbsHctabu/QBCC9CxyeGq6YiM5Tyfg0EXTVecZ+P8pgEm3MEaJ64f3iUPs Qt6g0xXEuslCV9gmf0IjJPibNTIaTvuOhvOOs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=swOJ5vUla0t5kC6760WDZvOFGzq20G24v/oS8dgyOt+EpJfqzpNWjWIEIsRWFGXkz7 jaYOwHAMSos81JA+SiMT3RQTtf9d9IPfOcafW1qTukB46wZ9j+py6Vpia9ZyzEp7ZZf1 9jz74cbONYqcVWNPs+0VRomYtPRUEVzboSZYA= Received: by 10.142.196.17 with SMTP id t17mr5864701wff.55.1300956566853; Thu, 24 Mar 2011 01:49:26 -0700 (PDT) Received: from [122.64.202.118] ([124.68.4.33]) by mx.google.com with ESMTPS id n4sm11212009wfl.14.2011.03.24.01.49.22 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 24 Mar 2011 01:49:25 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=iso-8859-1 From: Zheng Yang In-Reply-To: <4D8AF8C2.3000007@lelarge.info> Date: Thu, 24 Mar 2011 16:49:19 +0800 Cc: Selena Deckelmann , Andrew Dunstan , pgsql-students@postgresql.org Content-Transfer-Encoding: quoted-printable Message-Id: <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> To: Guillaume Lelarge X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001 X-Spam-Level: X-Archive-Number: 201103/21 X-Sequence-Number: 162 Hi Guillaume, Thanks very much! Your answers help a lot! >=20 > Le 24/03/2011 06:39, Zheng Yang a =E9crit : >=20 >> 4) Is theFDW something similar to the Mysql storage engine = architecture? >>=20 >=20 > Not really. AFAIK,there is a CSV storage engine in MySql that serves as a wrapper, = enable executing SQL queries directly on the file. This makes me think that the CSV FDW was doing similar things.=20 >=20 >> 5) for the datalink, the doc has mentioned about it's going to be OS = dependent. but i dun quite understand how does the kernel module and = LD_PRELOAD will work in this case, esp.LD_PRELOAD. >>=20 >=20 > No idea. >=20 I am quite new to postgresql, but slowly picking up :) In general = context, storing images directly inside DBMS fields as BLOBs was not = considered as a good practice.=20 However, storing file directories or links may cause inconsistency. So = I am quite interested in how this datalink concept can be implemented!=20= > Writing some FDWs would make a really good GSoC project. Haha, agree! I believe that's also a great starting point for learning = pgsql internals. cheers ZY From alvherre@commandprompt.com Thu Mar 24 16:49:27 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 912D01337990 for ; Thu, 24 Mar 2011 13:49:27 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 02154-02 for ; Thu, 24 Mar 2011 16:49:10 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from spinlock.commandprompt.com (host-215.commandprompt.net [207.173.203.215]) by mail.postgresql.org (Postfix) with ESMTP id D7E3713364DE for ; Thu, 24 Mar 2011 13:49:09 -0300 (ADT) Received: from localhost (localhost [127.0.0.1]) by spinlock.commandprompt.com (Postfix) with ESMTP id E099D348024; Thu, 24 Mar 2011 09:49:08 -0700 (PDT) X-Virus-Scanned: Debian amavisd-new at commandprompt.com Received: from spinlock.commandprompt.com ([127.0.0.1]) by localhost (spinlock.commandprompt.com.commandprompt.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id aJ2F8L3mw-KU; Thu, 24 Mar 2011 09:49:06 -0700 (PDT) Received: from perhan.alvh.no-ip.org (unknown [190.95.20.220]) by spinlock.commandprompt.com (Postfix) with ESMTPSA id A008D348022; Thu, 24 Mar 2011 09:49:06 -0700 (PDT) Received: by perhan.alvh.no-ip.org (Postfix, from userid 1000) id 0DF206EA7B; Thu, 24 Mar 2011 13:49:02 -0300 (CLST) Content-Type: text/plain; charset=UTF-8 Cc: Guillaume Lelarge , Selena Deckelmann , Andrew Dunstan , pgsql-students Subject: Re: foreign data wrappers From: Alvaro Herrera To: Zheng Yang In-reply-to: <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> Date: Thu, 24 Mar 2011 13:49:02 -0300 Message-Id: <1300985156-sup-3058@alvh.no-ip.org> User-Agent: Sup/git Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-0.386 tagged_above=-10 required=5 tests=BAYES_00=-1.9, URIBL_RHS_DOB=1.514 X-Spam-Level: X-Archive-Number: 201103/22 X-Sequence-Number: 163 Excerpts from Zheng Yang's message of jue mar 24 05:49:19 -0300 2011: > I am quite new to postgresql, but slowly picking up :) In general context, storing images directly inside DBMS fields as BLOBs was not considered as a good practice. > However, storing file directories or links may cause inconsistency. So I am quite interested in how this datalink concept can be implemented! I'm not really sure how different would be handling the file linking in an FDW. For example, if a transaction runs that deletes a file through the FDW, and the transaction rolls back, how are you going to restore the file to life? (It sounds like you're trying to have a FDW that would present a directory as a table, and each file in the dir as a row. Maybe it's not a bad idea but it needs a lot more thought.) -- Ã�lvaro Herrera The PostgreSQL Company - Command Prompt, Inc. PostgreSQL Replication, Consulting, Custom Development, 24x7 support From andrew@dunslane.net Thu Mar 24 17:00:20 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id BC9BF1337990 for ; Thu, 24 Mar 2011 14:00:20 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 08748-01 for ; Thu, 24 Mar 2011 17:00:01 +0000 (UTC) X-Greylist: delayed 00:05:10.832203 by SQLgrey-1.7.6 Received: from alexis.jtlnet.com (alexis.jtlnet.com [69.36.9.70]) by mail.postgresql.org (Postfix) with ESMTP id 6BC2713364DE for ; Thu, 24 Mar 2011 14:00:01 -0300 (ADT) Received: from dunslane.net ([::ffff:127.0.0.1]) by alexis.jtlnet.com with esmtp; Thu, 24 Mar 2011 12:55:13 -0400 id 00027C83.4D8B7771.00003BDC Received: from 208.91.122.102 (SquirrelMail authenticated user andrew@dunslane.net) by dunslane.net with HTTP; Thu, 24 Mar 2011 12:55:13 -0400 (EDT) Message-ID: <47676.208.91.122.102.1300985713.squirrel@dunslane.net> In-Reply-To: <1300985156-sup-3058@alvh.no-ip.org> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> Date: Thu, 24 Mar 2011 12:55:13 -0400 (EDT) Subject: Re: foreign data wrappers From: "Andrew Dunstan" To: "Alvaro Herrera" Cc: "Zheng Yang" , "Guillaume Lelarge" , "Selena Deckelmann" , "Andrew Dunstan" , "pgsql-students" User-Agent: SquirrelMail/1.4.10a Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: 7bit X-Priority: 3 (Normal) Importance: Normal X-Mime-Autoconverted: from 8bit to 7bit by courier 0.64 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201103/23 X-Sequence-Number: 164 On Thu, March 24, 2011 12:49 pm, Alvaro Herrera wrote: > Excerpts from Zheng Yang's message of jue mar 24 05:49:19 -0300 2011: > >> I am quite new to postgresql, but slowly picking up :) In general >> context, storing images directly inside DBMS fields as BLOBs was not >> considered as a good practice. >> However, storing file directories or links may cause inconsistency. So >> I am quite interested in how this datalink concept can be implemented! > > I'm not really sure how different would be handling the file linking in > an FDW. For example, if a transaction runs that deletes a file through > the FDW, and the transaction rolls back, how are you going to restore > the file to life? > > (It sounds like you're trying to have a FDW that would present a > directory as a table, and each file in the dir as a row. Maybe it's not > a bad idea but it needs a lot more thought.) > Currently FDWs can't store anything. They are read-only. But I don't really buy this stuff about not storing images in the database. I've done numbers of apps that do exactly that with great success. If the images are huge that's another matter, but for small images it works just fine. cheers andrew From selena@chesnok.com Thu Mar 24 20:59:52 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 60A0A1337BDD for ; Thu, 24 Mar 2011 17:59:52 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 39789-02 for ; Thu, 24 Mar 2011 20:59:44 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from homiemail-a73.g.dreamhost.com (caiajhbdcaid.dreamhost.com [208.97.132.83]) by mail.postgresql.org (Postfix) with ESMTP id 839DD1337BE8 for ; Thu, 24 Mar 2011 17:59:31 -0300 (ADT) Received: from homiemail-a73.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a73.g.dreamhost.com (Postfix) with ESMTP id 211DF1F0086 for ; Thu, 24 Mar 2011 13:59:30 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=chesnok.com; h=mime-version :in-reply-to:references:date:message-id:subject:from:to:cc: content-type; q=dns; s=chesnok.com; b=LRIDFQMWKCVjoha43zqMK0a+eV JvziLGHAOjgQV1gTQwMw9p5z4c4vV73leqHoasTLqoVuieEszSPYPbCR0gQGkLXN MyacBru1BtSOdetyfRRKrdb2UIH5rGrZlkLvDUOvMTUcBCXuJPAX3MTH4q73uHM7 3k1WK66KltAnagph4= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=chesnok.com; h= mime-version:in-reply-to:references:date:message-id:subject:from :to:cc:content-type; s=chesnok.com; bh=quzmcb2y11KZykTXsAU/jEX8F t4=; b=lMY+9PxbujKsD4VTPX4oOIdOALi+mF6mb6gR1Rofq0P2X3/7HSPjJVh9K yhVgD0ic7J22aymGllVJZLhYELi7v983796D+VhBeGz9JaJdjviX7sk/8wQR4p/3 b3OtcEb2PH5lotZfK0RrOBP6glJomu8rDugHtH/ademaWqZhMo= Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: selena@chesnok.com) by homiemail-a73.g.dreamhost.com (Postfix) with ESMTPSA id DCA741F0084 for ; Thu, 24 Mar 2011 13:59:29 -0700 (PDT) Received: by iyb26 with SMTP id 26so356866iyb.19 for ; Thu, 24 Mar 2011 13:59:29 -0700 (PDT) MIME-Version: 1.0 Received: by 10.42.74.194 with SMTP id x2mr7827280icj.113.1301000369435; Thu, 24 Mar 2011 13:59:29 -0700 (PDT) Received: by 10.42.213.137 with HTTP; Thu, 24 Mar 2011 13:59:29 -0700 (PDT) In-Reply-To: <47676.208.91.122.102.1300985713.squirrel@dunslane.net> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> Date: Thu, 24 Mar 2011 13:59:29 -0700 Message-ID: Subject: Re: foreign data wrappers From: Selena Deckelmann To: Andrew Dunstan Cc: Alvaro Herrera , Zheng Yang , Guillaume Lelarge , pgsql-students Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-0.386 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001, URIBL_RHS_DOB=1.514 X-Spam-Level: X-Archive-Number: 201103/24 X-Sequence-Number: 165 And, Andrew's slides from his talk today are now available: http://people.planetpostgresql.org/andrew/uploads/fdw2.pdf We really think the FDW projects would be great ones for GSoC. I know we said that already. :) -selena From rama.theone@gmail.com Fri Mar 25 00:53:38 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 6A0461337BDC for ; Thu, 24 Mar 2011 21:53:38 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 32272-06 for ; Fri, 25 Mar 2011 00:53:30 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-gw0-f46.google.com (mail-gw0-f46.google.com [74.125.83.46]) by mail.postgresql.org (Postfix) with ESMTP id 4ABAD1337BD3 for ; Thu, 24 Mar 2011 21:53:30 -0300 (ADT) Received: by gwaa18 with SMTP id a18so269305gwa.19 for ; Thu, 24 Mar 2011 17:53:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=0jO1uKu2kvoYK86iIrsljlmj8HCRr/9h0BiL9uj1Hyk=; b=pjE1psXVGbdKID1oOEaZTj95CgaVQ7fG9qkAKcv91ScCN4TZDd8R2odWMDfbvZUP1y uQ7VkNpJg8lR/5lmvUR9BFxFwqDw60h+4vnau6hcGOY6QzwtHFs2wA1K3ief8o7RX1QF cliDfOlIHyua9ARTFmZvHCcA+ghWEdFOrGuoM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=eL6+0WWBibIldsoJDI4vKkoaVJR8guokawdXgrFzuUGFAx1m0Cz/kIkqYUwH/qbDX7 3vz02B5qZkqT/f6qjiMnxkSAl3elFUaDP6UJSxBF9wL850BKdN89XgnQRZ2ar7mhp6DC Udf/3qRNmjtA1kmZDys+T3Cc0xtWBFYrgMtI0= Received: by 10.236.63.133 with SMTP id a5mr151867yhd.274.1301014410098; Thu, 24 Mar 2011 17:53:30 -0700 (PDT) MIME-Version: 1.0 Received: by 10.236.102.143 with HTTP; Thu, 24 Mar 2011 17:53:10 -0700 (PDT) In-Reply-To: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> From: Shiv Date: Fri, 25 Mar 2011 08:53:10 +0800 Message-ID: Subject: Re: foreign data wrappers To: Selena Deckelmann Cc: Andrew Dunstan , Alvaro Herrera , Zheng Yang , Guillaume Lelarge , pgsql-students Content-Type: multipart/alternative; boundary=00248c768ebac2fa66049f4405f1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.084 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, URIBL_RHS_DOB=1.514 X-Spam-Level: X-Archive-Number: 201103/25 X-Sequence-Number: 166 --00248c768ebac2fa66049f4405f1 Content-Type: text/plain; charset=ISO-8859-1 Thank you for the slides. And yes FDW seems like a good project to get involved in especially if one (and by one I mean me) is knew to the Postgres codebase. Regards, Shiv On Fri, Mar 25, 2011 at 4:59 AM, Selena Deckelmann wrote: > And, Andrew's slides from his talk today are now available: > http://people.planetpostgresql.org/andrew/uploads/fdw2.pdf > > We really think the FDW projects would be great ones for GSoC. I know > we said that already. :) > > -selena > > -- > Sent via pgsql-students mailing list (pgsql-students@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-students > --00248c768ebac2fa66049f4405f1 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Thank you for the slides. And yes FDW seems like a good project to get invo= lved in especially if one (and by one I mean me) is knew to the Postgres co= debase.
Regards,
Shiv



--00248c768ebac2fa66049f4405f1-- From zhengyang4k@gmail.com Fri Mar 25 02:07:35 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id E520D1337B8B for ; Thu, 24 Mar 2011 23:07:35 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 47872-05 for ; Fri, 25 Mar 2011 02:07:27 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mail.postgresql.org (Postfix) with ESMTP id 33BC91337990 for ; Thu, 24 Mar 2011 23:07:25 -0300 (ADT) Received: by iyb14 with SMTP id 14so21616iyb.19 for ; Thu, 24 Mar 2011 19:07:25 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to:x-mailer; bh=Rp/ccUpojcLsRFMf4AEyGxvKp6KZeSEy6rL3cbvIfbU=; b=M0OMJZT+WuuY+jjCikwCiFXgXJ6d7LPtaCUy6DzmZpuHLtLxLZ764CSHhVx4glvD2R aAQJ1nN3IYVeLTenbn5PY84CbEBRaWm+NTPr8WzmkHpn+Ke7kL+bW7I4zo0avQpw3Yd4 +KcrkeO8YMK3q2hlKcLRnJnKVYU4D+Oef0Qao= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; b=HfjUykl4+0m/U9fT70R1IC5Dnoecu/KMZsQL6r9FqLObwM5kbdFxG5MeeSuMrA7up2 cofQAp6fpEvhCwBg8L7VdK5PUdS1CLzdUoj1//ulV+epq5WCzkn9k5vPMpOTDa1aOtF8 Vn9tH2H33mB0En5H3u1PV+fFPKfZoY/L0bVKM= Received: by 10.42.147.134 with SMTP id n6mr313548icv.216.1301018845477; Thu, 24 Mar 2011 19:07:25 -0700 (PDT) Received: from [122.64.182.61] ([61.158.124.105]) by mx.google.com with ESMTPS id xa13sm264374icb.15.2011.03.24.19.07.15 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 24 Mar 2011 19:07:19 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/alternative; boundary=Apple-Mail-772-589084906 From: Zheng Yang In-Reply-To: Date: Fri, 25 Mar 2011 10:07:11 +0800 Cc: Selena Deckelmann , Andrew Dunstan , Alvaro Herrera , Guillaume Lelarge , pgsql-students Message-Id: <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> To: Shiv X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.898 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 201103/26 X-Sequence-Number: 167 --Apple-Mail-772-589084906 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Thanks for the slides! As what's mentioned on one of the slides, the = datasource of FDW can be virtually everything. So if there were a FDW for flickr API, the examples will be something = like: 1) CREATE FOREIGN TABLE flickr_table (photo_id INT, ownerid INT, photo = BLOB, text TEXT) SERVER flick_fdw OPTIONS (api_key 'AAAA'); =20 p.s. the text field is used as the keyword for free text search. when query from the table: 2) SELECT photo FROM flickr_table WHERE text =3D 'panda' LIMIT 0, 10; This will return top ten photos relevant to 'panda'. Is my understanding = correct? I've briefly gone through the slides. Regarding the 6 callbacks, is that = correct to say that a full table scan will always be performed = irregardless of the sql statement, the FDW is blind to the sql query performed, right? And can anyone help explain what a planner is? What does this "provide = cost estimates to planner" mean? Thanks! Hi Shiv, I am also from National University of Singapore! cheers, ZY On 25-Mar-2011, at 8:53 AM, Shiv wrote: > Thank you for the slides. And yes FDW seems like a good project to get = involved in especially if one (and by one I mean me) is knew to the = Postgres codebase. > Regards, > Shiv >=20 >=20 > On Fri, Mar 25, 2011 at 4:59 AM, Selena Deckelmann = wrote: > And, Andrew's slides from his talk today are now available: > http://people.planetpostgresql.org/andrew/uploads/fdw2.pdf >=20 > We really think the FDW projects would be great ones for GSoC. I know > we said that already. :) >=20 > -selena >=20 > -- > Sent via pgsql-students mailing list (pgsql-students@postgresql.org) > To make changes to your subscription: > http://www.postgresql.org/mailpref/pgsql-students >=20 --Apple-Mail-772-589084906 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii

Hi Shiv, I am also from National = University of = Singapore!

cheers,
ZY

On 25-Mar-2011, at 8:53 AM, Shiv = wrote:

Thank you for the slides. And yes FDW seems like a good = project to get involved in especially if one (and by one I mean me) is = knew to the Postgres codebase.
Regards,
Shiv


On Fri, Mar 25, 2011 at 4:59 AM, = Selena Deckelmann <selena@chesnok.com> = wrote:
And, Andrew's slides from his talk today are now available:
http://people.planetpostgresql.org/andrew/uploads/fdw2.p= df

We really think the FDW projects would be great ones for GSoC. I = know
we said that already. :)

-selena

--
Sent via pgsql-students mailing list (pgsql-students@postgresql.or= g)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-students


= --Apple-Mail-772-589084906-- From rama.theone@gmail.com Fri Mar 25 04:57:20 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 908B91337B4E for ; Fri, 25 Mar 2011 01:57:20 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 71307-04 for ; Fri, 25 Mar 2011 04:57:01 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-yw0-f46.google.com (mail-yw0-f46.google.com [209.85.213.46]) by mail.postgresql.org (Postfix) with ESMTP id 521251337BDA for ; Fri, 25 Mar 2011 01:57:01 -0300 (ADT) Received: by ywj3 with SMTP id 3so319489ywj.19 for ; Thu, 24 Mar 2011 21:57:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=wMACmoMC9gzONbKSnb7gGtqH7uZ5c+w3kS8zxGQITZc=; b=rMG74ReF/ANOisdWslF/YZNU5Hs/RVNinJhmtiVPfOYRM2whTRdm+dybtzzLSh0hzF nEHCD+j2YoLQNUB46ucS+3JImNtFcg1itqjRxZtwNa8CSVXTyAUQX+cC5Q9TkFScx4Xf 8Petp3yj9agtPLmF2HCdxdT+ITpa+9/i6I63g= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=TKkNUpmz50e5M0PA2WNnHkIxfg8EDvv9JuJUlAJYR83jKogyNBp0N3jx5ykxgXwp04 NefjvmTpMBIdAyoj9nuJX7z5xPpnH9E/sJbMd/0BvfVQjPO4mo6EtWH7omjiHI3e/9if wmkTaBOZQloSzHhLtFmroJvUrfhDd82EmHqDo= Received: by 10.236.190.3 with SMTP id d3mr485806yhn.174.1301029020409; Thu, 24 Mar 2011 21:57:00 -0700 (PDT) MIME-Version: 1.0 Received: by 10.236.102.143 with HTTP; Thu, 24 Mar 2011 21:56:40 -0700 (PDT) In-Reply-To: <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> From: Shiv Date: Fri, 25 Mar 2011 12:56:40 +0800 Message-ID: Subject: Re: foreign data wrappers To: Zheng Yang Cc: Selena Deckelmann , Andrew Dunstan , Alvaro Herrera , Guillaume Lelarge , pgsql-students Content-Type: multipart/alternative; boundary=20cf305b161c9a9e26049f476c73 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.598 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201103/27 X-Sequence-Number: 168 --20cf305b161c9a9e26049f476c73 Content-Type: text/plain; charset=ISO-8859-1 Haha.. that's great! Geographical proximity is always nice! The Flickr example is a good question. I would be interested to know if that is allowable (under the specifications for FDW) Regards, Shiv On Fri, Mar 25, 2011 at 10:07 AM, Zheng Yang wrote: > Thanks for the slides! As what's mentioned on one of the slides, the > datasource of FDW can be virtually everything. > > So if there were a FDW for flickr API, the examples will be something like: > > 1) CREATE FOREIGN TABLE flickr_table (photo_id INT, ownerid INT, photo > BLOB, text TEXT) SERVER flick_fdw OPTIONS (api_key 'AAAA'); > p.s. the text field is used as the keyword for free text search. > > when query from the table: > 2) SELECT photo FROM flickr_table WHERE text = 'panda' LIMIT 0, 10; > > This will return top ten photos relevant to 'panda'. Is my understanding > correct? > > > I've briefly gone through the slides. Regarding the 6 callbacks, is that > correct to say that a full table scan will always be performed irregardless > of the sql statement, > the FDW is blind to the sql query performed, right? > > And can anyone help explain what a planner is? What does this "provide cost > estimates to planner" mean? Thanks! > > Hi Shiv, I am also from National University of Singapore! > > cheers, > ZY > > > On 25-Mar-2011, at 8:53 AM, Shiv wrote: > > Thank you for the slides. And yes FDW seems like a good project to get > involved in especially if one (and by one I mean me) is knew to the Postgres > codebase. > Regards, > Shiv > > > On Fri, Mar 25, 2011 at 4:59 AM, Selena Deckelmann wrote: > >> And, Andrew's slides from his talk today are now available: >> http://people.planetpostgresql.org/andrew/uploads/fdw2.pdf >> >> We really think the FDW projects would be great ones for GSoC. I know >> we said that already. :) >> >> -selena >> >> -- >> Sent via pgsql-students mailing list (pgsql-students@postgresql.org) >> To make changes to your subscription: >> http://www.postgresql.org/mailpref/pgsql-students >> > > > --20cf305b161c9a9e26049f476c73 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Haha.. that's great! Geographical proximity is always nice!

The Flickr example is a good question. I would be interested to kno= w if that is allowable (under the specifications for FDW)
Regards,
Shiv


On Fri, Mar 25, 2011 at 10:07 AM, Zheng = Yang <zhengya= ng4k@gmail.com> wrote:
Thanks for the slides! As what's me= ntioned on one of the slides, the datasource of FDW can be virtually everyt= hing.

So if there were a FDW for flickr API, the examples will= be something like:

1)=A0CREATE FOREIGN TABLE flickr_table (photo_id INT, o= wnerid INT, photo BLOB, text TEXT) SERVER flick_fdw OPTIONS (api_key 'A= AAA'); =A0
p.s. the text field is used as the keyword for fre= e text search.

when query from the table:
2)=A0SELECT photo = FROM=A0flickr_table=A0WHERE text =3D 'panda' LIMIT 0, 10;

This will return top ten photos relevant to 'panda'= . Is my understanding correct?


I've briefly gone through the slides= . Regarding the 6 callbacks, is that correct to say that a full table scan = will always be performed irregardless of the sql statement,
the FDW is blind to the sql query performed, right?

And can anyone help explain what a planner is? What doe= s this "provide cost estimates to planner" mean? Thanks!

Hi Shiv, I am also from National University of Singapore!<= /div>

cheers,
ZY


On 25-Mar-2011, at 8:53 AM, Shi= v wrote:

Thank you for the slides. And y= es FDW seems like a good project to get involved in especially if one (and = by one I mean me) is knew to the Postgres codebase.
Regards,
Shiv





--20cf305b161c9a9e26049f476c73-- From selena@chesnok.com Fri Mar 25 15:38:22 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 3E3071337B6E for ; Fri, 25 Mar 2011 12:38:22 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 96622-08 for ; Fri, 25 Mar 2011 15:38:04 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from homiemail-a72.g.dreamhost.com (caiajhbdcagg.dreamhost.com [208.97.132.66]) by mail.postgresql.org (Postfix) with ESMTP id 130AD1337B61 for ; Fri, 25 Mar 2011 12:38:03 -0300 (ADT) Received: from homiemail-a72.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a72.g.dreamhost.com (Postfix) with ESMTP id 429EA6B0070 for ; Fri, 25 Mar 2011 08:38:02 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=chesnok.com; h=mime-version :in-reply-to:references:date:message-id:subject:from:to:cc :content-type:content-transfer-encoding; q=dns; s=chesnok.com; b=Q/uETkgkqvBJetKCJA2WBE+oHd7MOhWYuGpkG26c2Ok0nKqKUczaIwnx+MHgK UytK6C9HUBFTCmckruJAEgueqa0wwYHpDjxHBkXu9I9ow8cq88IVp3vJ60aLNFle IomTNxGvMo6xdfoeR2GEsKCFZTsqvxd7lWRpaNLo0OJuxM= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=chesnok.com; h= mime-version:in-reply-to:references:date:message-id:subject:from :to:cc:content-type:content-transfer-encoding; s=chesnok.com; bh=sZSGYQ/sFGQ0+fakN8v82Kya4X4=; b=Ot/yLwSoyIRZZhq9OT0WH5fV/Bjy bALhtb2J5AwEncz3kHcbTe0yE+NIyuzBOwxKmYmkHs1zTeQroeDC5xddtGk7vcpn kRSd2oaNFuM8JrJHlEJI2XbGfxoJ3PGgTtzGawWwKBhVMYJzpfGRGaS9FxPWdi+3 74Z7nNniskKOtpw= Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: selena@chesnok.com) by homiemail-a72.g.dreamhost.com (Postfix) with ESMTPSA id 2960B6B0059 for ; Fri, 25 Mar 2011 08:38:02 -0700 (PDT) Received: by iwn34 with SMTP id 34so1175563iwn.19 for ; Fri, 25 Mar 2011 08:38:01 -0700 (PDT) MIME-Version: 1.0 Received: by 10.231.113.98 with SMTP id z34mr870368ibp.188.1301067481704; Fri, 25 Mar 2011 08:38:01 -0700 (PDT) Received: by 10.231.201.199 with HTTP; Fri, 25 Mar 2011 08:38:01 -0700 (PDT) In-Reply-To: <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> Date: Fri, 25 Mar 2011 08:38:01 -0700 Message-ID: Subject: Re: foreign data wrappers From: Selena Deckelmann To: Zheng Yang Cc: Shiv , Andrew Dunstan , Alvaro Herrera , Guillaume Lelarge , pgsql-students Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201103/28 X-Sequence-Number: 169 Hi! On Thu, Mar 24, 2011 at 7:07 PM, Zheng Yang wrote: > Thanks for the slides! As what's mentioned on one of the slides, the > datasource of FDW can be virtually everything. > So if there were a FDW for flickr API, the examples will be something lik= e: > 1)=A0CREATE FOREIGN TABLE flickr_table (photo_id INT, ownerid INT, photo = BLOB, > text TEXT) SERVER flick_fdw OPTIONS (api_key 'AAAA'); > p.s. the text field is used as the keyword for free text search. > when query from the table: Yes, but perhaps we could name the 'text' column something like 'search' or ... someone else could help come up with a better name. :) > 2)=A0SELECT photo FROM=A0flickr_table=A0WHERE text =3D 'panda' LIMIT 0, 1= 0; > This will return top ten photos relevant to 'panda'. Is my understanding > correct? Yes, the first ten results. > I've briefly gone through the slides. Regarding the 6 callbacks, is that > correct to say that a full table scan will always be performed irregardle= ss > of the sql statement, > the FDW is blind to the sql query performed, right? That's correct. > And can anyone help explain what a planner is? What does this "provide co= st > estimates to planner" mean? Thanks! Have a look at this for a quick overview: http://www.postgresql.org/docs/9.0/static/planner-optimizer.html -selena --=20 http://chesnok.com From zhengyang4k@gmail.com Sat Mar 26 02:24:08 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 2281B1337B6E for ; Fri, 25 Mar 2011 23:24:08 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 34412-01 for ; Sat, 26 Mar 2011 02:23:59 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) by mail.postgresql.org (Postfix) with ESMTP id 3DFD21337B61 for ; Fri, 25 Mar 2011 23:23:57 -0300 (ADT) Received: by iwn34 with SMTP id 34so1666366iwn.19 for ; Fri, 25 Mar 2011 19:23:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to:x-mailer; bh=bT4NdYnmw+Zybz5Pl9Ykr3je4oYcNjoQU54afa+Dunk=; b=vhExcoDTw12NkVqQCdVUom/rSag1hm6BLEpMgkIY97ang0mre1XZJVPct+lPK/JqYJ DObQoyRBGjF75gJOe5o1LW/0c/7+rUeplL/aeN+ZJF+SN5TFKS0/arwcFkUS2aIbc6ZU NCOBWB6NY+J7KZHZYig+afa292TdxQgU7GhXg= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; b=pICcai12fmYNA5+OGo4F6K1ZGJx9d1AjoAWl61QN+pkKoVoNHIApTb1CCb0R46zvr9 3J6n8lOHVa5nykz9W/WlyaS2bWROQsszvhj48eFUUURY8K9Jh8vOSTle1k7AYojPP+s9 Skt1uqx8pgsqdN5FOa0RjsCRlCt0XL1Bly95M= Received: by 10.43.45.198 with SMTP id ul6mr2354440icb.449.1301106237732; Fri, 25 Mar 2011 19:23:57 -0700 (PDT) Received: from [122.64.185.49] ([61.158.124.225]) by mx.google.com with ESMTPS id wo11sm987742icb.8.2011.03.25.19.23.51 (version=TLSv1/SSLv3 cipher=OTHER); Fri, 25 Mar 2011 19:23:55 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/alternative; boundary=Apple-Mail-1451-676480803 From: Zheng Yang In-Reply-To: Date: Sat, 26 Mar 2011 10:23:47 +0800 Cc: Shiv , Andrew Dunstan , Alvaro Herrera , Guillaume Lelarge , pgsql-students Message-Id: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> To: Selena Deckelmann X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.898 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 201103/29 X-Sequence-Number: 170 --Apple-Mail-1451-676480803 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Thanks! Can anyone help on this: I wanted to check out an copy of pgsql source so that I can start = playing around with it.=20 I followed the working with git instructions here: http://wiki.postgresql.org/wiki/Working_with_Git but when I run this command: git clone git://git.postgresql.org/git/postgresql.git I get this error: Cloning into gitpostgresql... fatal: The remote end hung up unexpectedly ZY On 25-Mar-2011, at 11:38 PM, Selena Deckelmann wrote: > Hi! >=20 > On Thu, Mar 24, 2011 at 7:07 PM, Zheng Yang = wrote: >> Thanks for the slides! As what's mentioned on one of the slides, the >> datasource of FDW can be virtually everything. >> So if there were a FDW for flickr API, the examples will be something = like: >> 1) CREATE FOREIGN TABLE flickr_table (photo_id INT, ownerid INT, = photo BLOB, >> text TEXT) SERVER flick_fdw OPTIONS (api_key 'AAAA'); >> p.s. the text field is used as the keyword for free text search. >> when query from the table: >=20 > Yes, but perhaps we could name the 'text' column something like > 'search' or ... someone else could help come up with a better name. :) >=20 >> 2) SELECT photo FROM flickr_table WHERE text =3D 'panda' LIMIT 0, 10; >> This will return top ten photos relevant to 'panda'. Is my = understanding >> correct? >=20 > Yes, the first ten results. >=20 >> I've briefly gone through the slides. Regarding the 6 callbacks, is = that >> correct to say that a full table scan will always be performed = irregardless >> of the sql statement, >> the FDW is blind to the sql query performed, right? >=20 > That's correct. >=20 >> And can anyone help explain what a planner is? What does this = "provide cost >> estimates to planner" mean? Thanks! >=20 > Have a look at this for a quick overview: > http://www.postgresql.org/docs/9.0/static/planner-optimizer.html >=20 > -selena >=20 > --=20 > http://chesnok.com --Apple-Mail-1451-676480803 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii http://wiki.post= gresql.org/wiki/Working_with_Git

but when I = run this command:
git clone git://git.postgresql.org/git/po=
stgresql.git

I get this error:
Cloning = into gitpostgresql...
fatal: The remote end hung up = unexpectedly


ZY

On 25-Mar-2011, at 11:38 PM, Selena = Deckelmann wrote:

Hi!

On Thu, Mar 24, 2011 at 7:07 PM, Zheng = Yang <zhengyang4k@gmail.com> = wrote:
Thanks for the slides! As what's = mentioned on one of the slides, the
datasource of FDW can be virtually = everything.
So if there were a = FDW for flickr API, the examples will be something = like:
1) CREATE FOREIGN = TABLE flickr_table (photo_id INT, ownerid INT, photo = BLOB,
text TEXT) SERVER = flick_fdw OPTIONS (api_key 'AAAA');
p.s. the text field is used as the keyword for free text = search.
when query from the = table:

Yes, but perhaps we could name the 'text' = column something like
'search' or ... someone else could help come up = with a better name. :)

2) SELECT = photo FROM flickr_table WHERE text =3D 'panda' LIMIT 0, = 10;
This will return top ten = photos relevant to 'panda'. Is my = understanding
correct?

Yes, the first ten = results.

I've briefly gone through the = slides. Regarding the 6 callbacks, is that
correct to say that a full table scan will always be = performed irregardless
of the = sql statement,
the FDW is = blind to the sql query performed, right?

That's = correct.

And can anyone help explain = what a planner is? What does this "provide = cost
estimates to planner" = mean? Thanks!

Have a look at this for a quick = overview:
= http://www.postgresql.org/docs/9.0/static/planner-optimizer.html
-selena

-- =
http://chesnok.com

= --Apple-Mail-1451-676480803-- From dpage@pgadmin.org Sat Mar 26 08:11:19 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 600331337B4E for ; Sat, 26 Mar 2011 05:11:19 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 24845-06 for ; Sat, 26 Mar 2011 08:11:09 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mail.postgresql.org (Postfix) with ESMTP id 26784133798E for ; Sat, 26 Mar 2011 05:11:07 -0300 (ADT) Received: by iyb14 with SMTP id 14so1286608iyb.19 for ; Sat, 26 Mar 2011 01:11:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.43.59.80 with SMTP id wn16mr2982264icb.67.1301127066906; Sat, 26 Mar 2011 01:11:06 -0700 (PDT) Received: by 10.42.165.135 with HTTP; Sat, 26 Mar 2011 01:11:06 -0700 (PDT) In-Reply-To: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> Date: Sat, 26 Mar 2011 08:11:06 +0000 Message-ID: Subject: Re: foreign data wrappers From: Dave Page To: Zheng Yang Cc: Selena Deckelmann , Shiv , Andrew Dunstan , Alvaro Herrera , Guillaume Lelarge , pgsql-students Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201103/30 X-Sequence-Number: 171 On Sat, Mar 26, 2011 at 2:23 AM, Zheng Yang wrote: > Thanks! > Can anyone help on this: > I wanted to check out an copy of pgsql source so that I can start playing > around with it. > I followed the working with git instructions here: > http://wiki.postgresql.org/wiki/Working_with_Git > but when I run this command: > > git clone git://git.postgresql.org/git/postgresql.git > > I get this error: > Cloning into gitpostgresql... > fatal: The remote end hung up unexpectedly > Try cloning over http: git clone http://git.postgresql.org/git/postgresql.git -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company From zhengyang4k@gmail.com Sat Mar 26 12:46:51 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 08C951337B96 for ; Sat, 26 Mar 2011 09:46:51 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 05784-04 for ; Sat, 26 Mar 2011 12:46:43 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) by mail.postgresql.org (Postfix) with ESMTP id BAE8A133798E for ; Sat, 26 Mar 2011 09:46:43 -0300 (ADT) Received: by iwn34 with SMTP id 34so1970042iwn.19 for ; Sat, 26 Mar 2011 05:46:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=lyvCdDEB1Wq0c0vLEoRcTgwbJD+r5Qpv2W5oMjmDU24=; b=BArL0IwTaPW66NWpE9vWsbHoAUcMNXvzugQMIVNRlNAWzknz4053L7fl4FYsnY8FlX a0rsbgxMVoxXU/tv3Au4sCQjTxyV6s8BLQM95UdXDiUPoqGXev0sBZsDmcBAKdEd309X cqM0lVTXDMy7GLOhmWsPF7j/uroH9SXESBMoo= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=Tn1fKka5liI+rZKo2w3FpMgvWtvto/ihkwdGsKStqxfIet64RWf/j1Ek5eDY+GeXCL dMlN8+EB6PPuPzV9uMRrITSpNEJJiKnYW1POWolz008gl3xcAC4/TyFFGOQ2w14YPrk3 S3DD73hGMy1EoijzNGvx8hScoWjYcULTohDO8= Received: by 10.43.69.132 with SMTP id yc4mr3218477icb.221.1301143604037; Sat, 26 Mar 2011 05:46:44 -0700 (PDT) Received: from [122.64.185.49] ([61.158.124.225]) by mx.google.com with ESMTPS id wu1sm1316571icb.10.2011.03.26.05.46.37 (version=TLSv1/SSLv3 cipher=OTHER); Sat, 26 Mar 2011 05:46:42 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Zheng Yang In-Reply-To: Date: Sat, 26 Mar 2011 20:46:28 +0800 Cc: Selena Deckelmann , Shiv , Andrew Dunstan , Alvaro Herrera , Guillaume Lelarge , pgsql-students Content-Transfer-Encoding: 7bit Message-Id: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> To: Dave Page X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.599 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201103/31 X-Sequence-Number: 172 Hi, On 26-Mar-2011, at 4:11 PM, Dave Page wrote: > > Try cloning over http: > > git clone http://git.postgresql.org/git/postgresql.git > It seems hanging there forever. $git clone http://git.postgresql.org/git/postgresql.git Cloning into postgresql... ZY From rama.theone@gmail.com Sat Mar 26 12:49:26 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id B30E41337B4E for ; Sat, 26 Mar 2011 09:49:26 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 12166-10 for ; Sat, 26 Mar 2011 12:49:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-gx0-f174.google.com (mail-gx0-f174.google.com [209.85.161.174]) by mail.postgresql.org (Postfix) with ESMTP id B0CFB133798E for ; Sat, 26 Mar 2011 09:49:08 -0300 (ADT) Received: by gxk21 with SMTP id 21so750684gxk.19 for ; Sat, 26 Mar 2011 05:49:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc:content-type; bh=WPE6afWXFT+N2E4vV5AcFT952LRXmUEkGSQBRm7pdKo=; b=EWlW0zeF/1qGKeDDOl00+qEryiSpSC2zr+AMUcdDU/Tc+BRlrCbPXJoOR7hExU5u/u /KSJ727RKk0cx4MNAfqGCowsN5R4uFrzTmKjyb67idJpLG7bcmDc9rGKiInDUj24/QO1 yc63qAjYIl8Dn82BPxLme+remnweM8idN3M8s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; b=Ba9rptkI+IhFAQtTrQa7Jk45Loa45P0QaNDcw1pxpU5tNvICmN7XQm/+zXNknTn0tZ 8BvMECM6NgXFpKXDZgcmVEPPx9yz0pMAFnicR4eTL3A0m7wskjoW9M4OoCR4/qZbfXJF 8hnHsiwtSH2YfS7MOxHG9agTknbOWbaLX0ilA= Received: by 10.236.134.147 with SMTP id s19mr647165yhi.24.1301143748071; Sat, 26 Mar 2011 05:49:08 -0700 (PDT) MIME-Version: 1.0 Received: by 10.236.102.143 with HTTP; Sat, 26 Mar 2011 05:48:48 -0700 (PDT) In-Reply-To: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> From: Shiv Date: Sat, 26 Mar 2011 20:48:48 +0800 Message-ID: Subject: Re: foreign data wrappers To: Zheng Yang Cc: Dave Page , Selena Deckelmann , Andrew Dunstan , Alvaro Herrera , Guillaume Lelarge , pgsql-students Content-Type: multipart/alternative; boundary=20cf303a3631e7d07d049f6222fc X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-2.598 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7 X-Spam-Level: X-Archive-Number: 201103/32 X-Sequence-Number: 173 --20cf303a3631e7d07d049f6222fc Content-Type: text/plain; charset=ISO-8859-1 Hey Zheng, I don't seem to be having any issues. Something else might be going wrong. Regards, Shiv On Sat, Mar 26, 2011 at 8:46 PM, Zheng Yang wrote: > Hi, > > On 26-Mar-2011, at 4:11 PM, Dave Page wrote: > > > > Try cloning over http: > > > > git clone http://git.postgresql.org/git/postgresql.git > > > > It seems hanging there forever. > > $git clone http://git.postgresql.org/git/postgresql.git > Cloning into postgresql... > > > ZY > --20cf303a3631e7d07d049f6222fc Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Hey Zheng,
=A0I don't seem to be having any issues. Something else = might be going wrong.
Regards,
Shiv


On Sat, Mar 26, 2011 at 8:46 PM, Zheng Y= ang <zhengyan= g4k@gmail.com> wrote:
Hi,

On 26-Mar-2011, at 4:11 PM, Dave Page wrote:
>
> Try cloning over http:
>
> git clone http://git.postgresql.org/git/postgresql.git
>

It seems hanging there forever.

$git clone http://git.postgresql.org/git/postgresql.git
Cloning into postgresql...


ZY

--20cf303a3631e7d07d049f6222fc-- From andrew@dunslane.net Sat Mar 26 15:07:26 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id CB7C41337990 for ; Sat, 26 Mar 2011 12:07:26 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 29180-09 for ; Sat, 26 Mar 2011 15:07:19 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.122]) by mail.postgresql.org (Postfix) with ESMTP id 7E6BD133740B for ; Sat, 26 Mar 2011 12:07:19 -0300 (ADT) X-Authority-Analysis: v=1.1 cv=2pE2Kh9Ye2ywHyyFZnC5ZQ1FvuPrdOtuPO5uN4ysVDU= c=1 sm=0 a=YOqqrsh--ucA:10 a=r6Jiw2x_onsA:10 a=8nJEP1OIZ-IA:10 a=qCTYjJJ2g1SgQhsHZAUATw==:17 a=epTmVMiNAAAA:8 a=RbiJVo_cxWiHys3y7R4A:9 a=Z6IrFThzw18ycmL1GgoA:7 a=FgQUcE5K0gZGX6yk4gk8tQ0g4IIA:4 a=wPNLvfGTeEIA:10 a=H47jIOvrWu8A:10 a=qCTYjJJ2g1SgQhsHZAUATw==:117 X-Cloudmark-Score: 0 X-Originating-IP: 98.27.48.207 Received: from [98.27.48.207] ([98.27.48.207:56303] helo=[192.168.10.104]) by cdptpa-oedge03.mail.rr.com (envelope-from ) (ecelerity 2.2.3.46 r()) with ESMTP id D9/C1-05159-5210E8D4; Sat, 26 Mar 2011 15:07:18 +0000 Message-ID: <4D8E0125.9060905@dunslane.net> Date: Sat, 26 Mar 2011 11:07:17 -0400 From: Andrew Dunstan User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc13 Lightning/1.0b3pre Thunderbird/3.1.7 MIME-Version: 1.0 To: Zheng Yang CC: Shiv , Selena Deckelmann , Alvaro Herrera , Guillaume Lelarge , pgsql-students Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> In-Reply-To: <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201103/33 X-Sequence-Number: 174 On 03/24/2011 10:07 PM, Zheng Yang wrote: > > > I've briefly gone through the slides. Regarding the 6 callbacks, is > that correct to say that a full table scan will always be performed > irregardless of the sql statement, > the FDW is blind to the sql query performed, right? Yes, fairly much. If the feed is large you need some way to pass a limit to the foreign side, possibly via table options. I'm fairly sure you won't be able to get it via the SELECT statement. > > And can anyone help explain what a planner is? What does this "provide > cost estimates to planner" mean? Thanks! See If you're going to work on PostgreSQL code you need to read the docs. cheers andrew From zhengyang4k@gmail.com Sun Mar 27 12:52:00 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 1499E1337B43 for ; Sun, 27 Mar 2011 09:52:00 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 99699-05 for ; Sun, 27 Mar 2011 12:51:41 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) by mail.postgresql.org (Postfix) with ESMTP id 63A31133798E for ; Sun, 27 Mar 2011 09:51:40 -0300 (ADT) Received: by iwn34 with SMTP id 34so2688446iwn.19 for ; Sun, 27 Mar 2011 05:51:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to:x-mailer; bh=8AD9GJBdy/vNTVMKCEZwgYux7XmI9jcilpw6rj+Asx4=; b=LM1UtBxTebwrRpvDEYK4fmh4dNYTkFQTm0jwOfTLg56j3yfUVByFHqtXSOSrL/qRHd xne8B9eev3uUxb5kMdAhe0jVRtp1bVn5sLzyt2dxVmi3DrcR1omZVAuorHtBtyd9cjIJ oepGDo5L7KeZf69CHXP024tQASgyUzYawDy44= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; b=nYiqxXyKlvKZFzsWQdaR05f0NEYRlZ7/QJw72CMAxeIF5D2ETKyOWy0gR4a+PemSZu bLPJB+2doPaNP1vuHpgfxmuu6CP9ZhQDFppSjhl+r61f9ryDLLDxFg1OHOdw6eHTiH6O WUoPljorKmiC59dNHaZubWo2OpDk1e6j3LV6Y= Received: by 10.42.221.196 with SMTP id id4mr4340086icb.141.1301230298926; Sun, 27 Mar 2011 05:51:38 -0700 (PDT) Received: from [172.21.24.1] (nusnet-228-5.dynip.nus.edu.sg [137.132.228.5]) by mx.google.com with ESMTPS id gy41sm2171660ibb.39.2011.03.27.05.51.29 (version=TLSv1/SSLv3 cipher=OTHER); Sun, 27 Mar 2011 05:51:36 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/alternative; boundary=Apple-Mail-2669-800534858 From: Zheng Yang In-Reply-To: <4D8E0125.9060905@dunslane.net> Date: Sun, 27 Mar 2011 20:51:21 +0800 Cc: Shiv , Selena Deckelmann , Alvaro Herrera , Guillaume Lelarge , pgsql-students Message-Id: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> To: Andrew Dunstan X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.898 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 201103/34 X-Sequence-Number: 175 --Apple-Mail-2669-800534858 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi, I managed to clone a copy of 9.1devel and compiled an binary from it. I = also did a make in the 'contrib/file_fdw/' folder and a file_fdw.so file = is generated. But the following error occurs when I was trying to experiment with the = file FDW: postgres=3D# CREATE FUNCTION file_fdw_handler() RETURNS fdw_handler AS = 'file_fdw' LANGUAGE C STRICT; ERROR: could not access file "file_fdw": No such file or directory STATEMENT: CREATE FUNCTION file_fdw_handler() RETURNS fdw_handler AS = 'file_fdw' LANGUAGE C STRICT; ERROR: could not access file "file_fdw": No such file or directory Did I miss anything? ZY On 26-Mar-2011, at 11:07 PM, Andrew Dunstan wrote: >=20 >=20 > On 03/24/2011 10:07 PM, Zheng Yang wrote: >>=20 >>=20 >> I've briefly gone through the slides. Regarding the 6 callbacks, is = that correct to say that a full table scan will always be performed = irregardless of the sql statement, >> the FDW is blind to the sql query performed, right? >=20 > Yes, fairly much. If the feed is large you need some way to pass a = limit to the foreign side, possibly via table options. I'm fairly sure = you won't be able to get it via the SELECT statement. >=20 >>=20 >> And can anyone help explain what a planner is? What does this = "provide cost estimates to planner" mean? Thanks! >=20 >=20 > See = >=20 > If you're going to work on PostgreSQL code you need to read the docs. >=20 > cheers >=20 > andrew >=20 --Apple-Mail-2669-800534858 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii ERROR:  could not access file "file_fdw": No such = file or directory
STATEMENT:  CREATE FUNCTION = file_fdw_handler() RETURNS = fdw_handler AS 'file_fdw' LANGUAGE C = STRICT;
ERROR:  could not access file "file_fdw": No such = file or = directory


Did = I miss = anything?


ZY

<= div>

On 26-Mar-2011, at 11:07 PM, Andrew = Dunstan wrote:



On 03/24/2011 10:07 PM, Zheng Yang = wrote:


I've briefly = gone through the slides. Regarding the 6 callbacks, is that correct to = say that a full table scan will always be performed irregardless of the = sql statement,
the FDW is = blind to the sql query performed, right?

Yes, fairly = much. If the feed is large you need some way to pass a limit to the = foreign side, possibly via table options. I'm fairly sure you won't be = able to get it via the SELECT statement.


And can anyone = help explain what a planner is? What does this "provide cost estimates = to planner" mean? Thanks!


See <http://www.postgresql.org/docs/current/static/planner-optimizer.html>

If you're going to work on PostgreSQL code you need to = read the = docs.

cheers

andrew


= --Apple-Mail-2669-800534858-- From guillaume@lelarge.info Mon Mar 28 07:28:16 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 781FF1337BC4 for ; Mon, 28 Mar 2011 04:28:16 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 79671-08 for ; Mon, 28 Mar 2011 07:28:06 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sd-17937.dedibox.fr (sd-17937.dedibox.fr [88.191.100.74]) by mail.postgresql.org (Postfix) with ESMTP id B00BE1337B61 for ; Mon, 28 Mar 2011 04:28:05 -0300 (ADT) Received: from [192.168.10.3] (ASte-Genev-Bois-153-1-85-140.w81-48.abo.wanadoo.fr [81.48.35.140]) by sd-17937.dedibox.fr (Postfix) with ESMTPA id 3517A3B8F5; Mon, 28 Mar 2011 09:28:05 +0200 (CEST) Message-ID: <4D903884.8070405@lelarge.info> Date: Mon, 28 Mar 2011 09:28:04 +0200 From: Guillaume Lelarge User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); fr; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Zheng Yang CC: Andrew Dunstan , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9 X-Spam-Level: X-Archive-Number: 201103/35 X-Sequence-Number: 176 Le 27/03/2011 14:51, Zheng Yang a écrit : > Hi, > > I managed to clone a copy of 9.1devel and compiled an binary from it. I also did a make in the 'contrib/file_fdw/' folder and a file_fdw.so file is generated. > > But the following error occurs when I was trying to experiment with the file FDW: > > postgres=# CREATE FUNCTION file_fdw_handler() RETURNS fdw_handler AS 'file_fdw' LANGUAGE C STRICT; > ERROR: could not access file "file_fdw": No such file or directory > STATEMENT: CREATE FUNCTION file_fdw_handler() RETURNS fdw_handler AS 'file_fdw' LANGUAGE C STRICT; > ERROR: could not access file "file_fdw": No such file or directory > > > Did I miss anything? > make install? -- Guillaume http://www.postgresql.fr http://dalibo.com From zhengyang4k@gmail.com Mon Mar 28 09:50:19 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 3D27E1337B49 for ; Mon, 28 Mar 2011 06:50:19 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 22711-09 for ; Mon, 28 Mar 2011 09:50:09 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) by mail.postgresql.org (Postfix) with ESMTP id 7951C1337B47 for ; Mon, 28 Mar 2011 06:50:08 -0300 (ADT) Received: by iwn34 with SMTP id 34so3477265iwn.19 for ; Mon, 28 Mar 2011 02:50:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=W3Io615lQVC0tQt1g57HgZNFKySx8tVNwRgeit33xKY=; b=ir1kmQ1Kxi42GsJlj5hHGwgjk6smzqspok0uMdCdfxMuLcrwx62VgoIQOv/xs+vMRb irF/V3sKzxRT/r1IBEPkvu1NlLSe5MdIPk/utrUfCaxfUCAGzcE0oDpIrWILVQ6wHMsG pDyk13Z2dP0eJzzG7VKPqI/DkfNxloiP7xjRA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=L3olIzgkm+nImdw9RHyloT291p4CiNksiROclA/dZD7YuKKfKEDsOV61egM8r6zJ9q t0E7qQHHXzYnRqw6icfHJeoZ3xePdgDMaNKsqrVY8UJcVXRxvuJmnnIvi6Fx+5ATfHIg Y4dxPbTi8z8sCoZ44pqAoG65TzLK6GBLRZJoQ= Received: by 10.42.138.129 with SMTP id c1mr6721063icu.492.1301305808388; Mon, 28 Mar 2011 02:50:08 -0700 (PDT) Received: from [122.64.174.55] ([61.158.124.241]) by mx.google.com with ESMTPS id 13sm2846589ibo.8.2011.03.28.02.50.03 (version=TLSv1/SSLv3 cipher=OTHER); Mon, 28 Mar 2011 02:50:07 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=iso-8859-1 From: Zheng Yang In-Reply-To: <4D903884.8070405@lelarge.info> Date: Mon, 28 Mar 2011 17:49:59 +0800 Cc: Andrew Dunstan , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Content-Transfer-Encoding: quoted-printable Message-Id: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <4D903884.8070405@lelarge.info> To: Guillaume Lelarge X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001 X-Spam-Level: X-Archive-Number: 201103/36 X-Sequence-Number: 177 Haha.. Thanks, Guilaume. Forgot to install. It works now! ZY On 28-Mar-2011, at 3:28 PM, Guillaume Lelarge wrote: > Le 27/03/2011 14:51, Zheng Yang a =E9crit : >> Hi, >>=20 >> I managed to clone a copy of 9.1devel and compiled an binary from it. = I also did a make in the 'contrib/file_fdw/' folder and a file_fdw.so = file is generated. >>=20 >> But the following error occurs when I was trying to experiment with = the file FDW: >>=20 >> postgres=3D# CREATE FUNCTION file_fdw_handler() RETURNS fdw_handler = AS 'file_fdw' LANGUAGE C STRICT; >> ERROR: could not access file "file_fdw": No such file or directory >> STATEMENT: CREATE FUNCTION file_fdw_handler() RETURNS fdw_handler AS = 'file_fdw' LANGUAGE C STRICT; >> ERROR: could not access file "file_fdw": No such file or directory >>=20 >>=20 >> Did I miss anything? >>=20 >=20 > make install? >=20 >=20 > --=20 > Guillaume > http://www.postgresql.fr > http://dalibo.com From zhengyang4k@gmail.com Tue Mar 29 11:29:20 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 1E6651337999 for ; Tue, 29 Mar 2011 08:29:20 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 79167-02 for ; Tue, 29 Mar 2011 11:29:11 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iw0-f174.google.com (mail-iw0-f174.google.com [209.85.214.174]) by mail.postgresql.org (Postfix) with ESMTP id 71EC41337BE2 for ; Tue, 29 Mar 2011 08:29:09 -0300 (ADT) Received: by iwn34 with SMTP id 34so65315iwn.19 for ; Tue, 29 Mar 2011 04:29:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=NjJDIPXZe6N1hjxkIOow6GJvDbzw4WWnfYXXk4VrHuA=; b=xbyFYhUaE3Fj6TMBVIkj9xdkj7OqNk02qsgaoRX4Iu3uAoIYrvQKuIHDV8RduKxnEb UfcBHdQX92zghflAQUZOdPNXAy4BqcREHfNykiOp6MlopYqIaPYwuSpXloNXVfDPPBo7 owgMQ5fVj/dgRZzIZ4L7gTRX6GSDH3Menkd6Y= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=vgr0a6Sb8y734BUiaOnWxbkoTl3L9y2F9J3QD6UQYGdI1Ta2MeonvyayONE0FvSaEH rtkcu2TlNY1Tdli2LC98JHnKvTlwINMyXW4qkzU2q2IPrmScHDNYRV9prcfoARzrytF+ Vc/SAtVT5OYpFsd1nJqswuaaJbQ+orilDpA9o= Received: by 10.42.117.3 with SMTP id r3mr8001238icq.429.1301398149758; Tue, 29 Mar 2011 04:29:09 -0700 (PDT) Received: from [172.16.36.3] ([74.117.56.62]) by mx.google.com with ESMTPS id g4sm3443129ick.23.2011.03.29.04.29.03 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 29 Mar 2011 04:29:07 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Zheng Yang In-Reply-To: <4D8E0125.9060905@dunslane.net> Date: Tue, 29 Mar 2011 19:28:57 +0800 Cc: Shiv , Selena Deckelmann , Alvaro Herrera , Guillaume Lelarge , pgsql-students Content-Transfer-Encoding: quoted-printable Message-Id: <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> To: Andrew Dunstan X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001 X-Spam-Level: X-Archive-Number: 201103/37 X-Sequence-Number: 178 Hi guys, >>=20 >> I've briefly gone through the slides. Regarding the 6 callbacks, is = that correct to say that a full table scan will always be performed = irregardless of the sql statement, >> the FDW is blind to the sql query performed, right? >=20 > Yes, fairly much. If the feed is large you need some way to pass a = limit to the foreign side, possibly via table options. I'm fairly sure = you won't be able to get it via the SELECT statement. >=20 Regarding the previous flickr example, I'm wondering how this 'free text = search' function can be done if the FDW is blind to the SELECT = statement. For instance, the following query is to retrieve a photo relevant to = 'panda': SELECT photo FROM flickr_table WHERE search LIKE '%panda%'; In this case, the FDW can only open a connection to flickr web service = and return the next 'row' .=20 The problem is that there are a huge number of photos in flickr server = and retrieving them sequentially is not realistic.=20 Any ideas on how this can be done? ZY= From guillaume@lelarge.info Tue Mar 29 15:49:16 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id 04F191337B69 for ; Tue, 29 Mar 2011 12:49:16 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 74085-06 for ; Tue, 29 Mar 2011 15:48:58 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sd-17937.dedibox.fr (sd-17937.dedibox.fr [88.191.100.74]) by mail.postgresql.org (Postfix) with ESMTP id 1E2B51337B67 for ; Tue, 29 Mar 2011 12:48:57 -0300 (ADT) Received: from [192.168.42.162] (LCaen-151-93-7-26.w80-13.abo.wanadoo.fr [80.13.230.26]) by sd-17937.dedibox.fr (Postfix) with ESMTPA id 7CA443B8F5; Tue, 29 Mar 2011 17:48:56 +0200 (CEST) Message-ID: <4D91FF65.2090401@lelarge.info> Date: Tue, 29 Mar 2011 17:48:53 +0200 From: Guillaume Lelarge User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); fr; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Zheng Yang CC: Andrew Dunstan , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> In-Reply-To: <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.324 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_NJABL_PROXY=2.224 X-Spam-Level: X-Archive-Number: 201103/38 X-Sequence-Number: 179 Le 29/03/2011 13:28, Zheng Yang a écrit : > Hi guys, > >>> >>> I've briefly gone through the slides. Regarding the 6 callbacks, is that correct to say that a full table scan will always be performed irregardless of the sql statement, >>> the FDW is blind to the sql query performed, right? >> >> Yes, fairly much. If the feed is large you need some way to pass a limit to the foreign side, possibly via table options. I'm fairly sure you won't be able to get it via the SELECT statement. >> > > > Regarding the previous flickr example, I'm wondering how this 'free text search' function can be done if the FDW is blind to the SELECT statement. > > For instance, the following query is to retrieve a photo relevant to 'panda': > > SELECT photo FROM flickr_table WHERE search LIKE '%panda%'; > > In this case, the FDW can only open a connection to flickr web service and return the next 'row' . > The problem is that there are a huge number of photos in flickr server and retrieving them sequentially is not realistic. > Any ideas on how this can be done? > It probably means that flickr is not a good example of a nice fdw. -- Guillaume http://www.postgresql.fr http://dalibo.com From andrew@dunslane.net Tue Mar 29 16:32:23 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 964E61337B93 for ; Tue, 29 Mar 2011 13:32:23 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 98039-10 for ; Tue, 29 Mar 2011 16:32:16 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.122]) by mail.postgresql.org (Postfix) with ESMTP id 173CA1337B47 for ; Tue, 29 Mar 2011 13:32:15 -0300 (ADT) X-Authority-Analysis: v=1.1 cv=r4yJ8ACLDmU9N8MfnU6qGSvboKzSN9UnPAeXToqJDNE= c=1 sm=0 a=YOqqrsh--ucA:10 a=r6Jiw2x_onsA:10 a=8nJEP1OIZ-IA:10 a=qCTYjJJ2g1SgQhsHZAUATw==:17 a=BqNE0HiyaE8oqZRXOSIA:9 a=JyLTqDN3k_78U1L8zi4A:7 a=NvpOu8K9y6aAxB4EisWLrFBcDKMA:4 a=wPNLvfGTeEIA:10 a=Vhc9UbeUoiuuE2Nq:21 a=hrFKkLdy3on51tkx:21 a=qCTYjJJ2g1SgQhsHZAUATw==:117 X-Cloudmark-Score: 0 X-Originating-IP: 98.27.48.207 Received: from [98.27.48.207] ([98.27.48.207:53024] helo=[192.168.10.104]) by cdptpa-oedge01.mail.rr.com (envelope-from ) (ecelerity 2.2.3.46 r()) with ESMTP id CA/5F-09483-E89029D4; Tue, 29 Mar 2011 16:32:15 +0000 Message-ID: <4D92098D.6040409@dunslane.net> Date: Tue, 29 Mar 2011 12:32:13 -0400 From: Andrew Dunstan User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc13 Lightning/1.0b3pre Thunderbird/3.1.7 MIME-Version: 1.0 To: Guillaume Lelarge CC: Zheng Yang , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> <4D91FF65.2090401@lelarge.info> In-Reply-To: <4D91FF65.2090401@lelarge.info> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201103/39 X-Sequence-Number: 180 On 03/29/2011 11:48 AM, Guillaume Lelarge wrote: > Le 29/03/2011 13:28, Zheng Yang a écrit : >> Hi guys, >> >>>> I've briefly gone through the slides. Regarding the 6 callbacks, is that correct to say that a full table scan will always be performed irregardless of the sql statement, >>>> the FDW is blind to the sql query performed, right? >>> Yes, fairly much. If the feed is large you need some way to pass a limit to the foreign side, possibly via table options. I'm fairly sure you won't be able to get it via the SELECT statement. >>> >> >> Regarding the previous flickr example, I'm wondering how this 'free text search' function can be done if the FDW is blind to the SELECT statement. >> >> For instance, the following query is to retrieve a photo relevant to 'panda': >> >> SELECT photo FROM flickr_table WHERE search LIKE '%panda%'; >> >> In this case, the FDW can only open a connection to flickr web service and return the next 'row' . >> The problem is that there are a huge number of photos in flickr server and retrieving them sequentially is not realistic. >> Any ideas on how this can be done? >> > It probably means that flickr is not a good example of a nice fdw. Neither of you are being very creative. As I mentioned above, you need to embed this sort of stuff in table options. so you would have something like: create foreign table panda_flickr (photo bytea, ...) server flickr_server options (searchterm 'panda', maxrows '50'); select photo from panda_flickr; cheers andrew From guillaume@lelarge.info Tue Mar 29 16:35:31 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id DA0E51337B47 for ; Tue, 29 Mar 2011 13:35:31 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 06308-10 for ; Tue, 29 Mar 2011 16:35:13 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from sd-17937.dedibox.fr (sd-17937.dedibox.fr [88.191.100.74]) by mail.postgresql.org (Postfix) with ESMTP id 9A73A133655C for ; Tue, 29 Mar 2011 13:35:13 -0300 (ADT) Received: from [192.168.42.162] (LCaen-151-93-7-26.w80-13.abo.wanadoo.fr [80.13.230.26]) by sd-17937.dedibox.fr (Postfix) with ESMTPA id C785A3B8F5; Tue, 29 Mar 2011 18:35:12 +0200 (CEST) Message-ID: <4D920A3D.8010004@lelarge.info> Date: Tue, 29 Mar 2011 18:35:09 +0200 From: Guillaume Lelarge User-Agent: Mozilla/5.0 (X11; U; Linux i686 (x86_64); fr; rv:1.9.2.15) Gecko/20110303 Thunderbird/3.1.9 MIME-Version: 1.0 To: Andrew Dunstan CC: Zheng Yang , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> <4D91FF65.2090401@lelarge.info> <4D92098D.6040409@dunslane.net> In-Reply-To: <4D92098D.6040409@dunslane.net> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=0.324 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_NJABL_PROXY=2.224 X-Spam-Level: X-Archive-Number: 201103/40 X-Sequence-Number: 181 Le 29/03/2011 18:32, Andrew Dunstan a écrit : > > > On 03/29/2011 11:48 AM, Guillaume Lelarge wrote: >> Le 29/03/2011 13:28, Zheng Yang a écrit : >>> Hi guys, >>> >>>>> I've briefly gone through the slides. Regarding the 6 callbacks, is >>>>> that correct to say that a full table scan will always be performed >>>>> irregardless of the sql statement, >>>>> the FDW is blind to the sql query performed, right? >>>> Yes, fairly much. If the feed is large you need some way to pass a >>>> limit to the foreign side, possibly via table options. I'm fairly >>>> sure you won't be able to get it via the SELECT statement. >>>> >>> >>> Regarding the previous flickr example, I'm wondering how this 'free >>> text search' function can be done if the FDW is blind to the SELECT >>> statement. >>> >>> For instance, the following query is to retrieve a photo relevant to >>> 'panda': >>> >>> SELECT photo FROM flickr_table WHERE search LIKE '%panda%'; >>> >>> In this case, the FDW can only open a connection to flickr web >>> service and return the next 'row' . >>> The problem is that there are a huge number of photos in flickr >>> server and retrieving them sequentially is not realistic. >>> Any ideas on how this can be done? >>> >> It probably means that flickr is not a good example of a nice fdw. > > > Neither of you are being very creative. As I mentioned above, you need > to embed this sort of stuff in table options. > > so you would have something like: > > create foreign table panda_flickr (photo bytea, ...) > server flickr_server > options (searchterm 'panda', maxrows '50'); > select photo from panda_flickr; > This would work but means you need to create a new foreign table to search something else. So, yeah, it works, but it's not convenient. -- Guillaume http://www.postgresql.fr http://dalibo.com From andrew@dunslane.net Tue Mar 29 16:52:05 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 07FEC1337B93 for ; Tue, 29 Mar 2011 13:52:05 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 57625-03 for ; Tue, 29 Mar 2011 16:51:57 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from cdptpa-omtalb.mail.rr.com (cdptpa-omtalb.mail.rr.com [75.180.132.123]) by mail.postgresql.org (Postfix) with ESMTP id 85E741337BB7 for ; Tue, 29 Mar 2011 13:51:57 -0300 (ADT) X-Authority-Analysis: v=1.1 cv=r4yJ8ACLDmU9N8MfnU6qGSvboKzSN9UnPAeXToqJDNE= c=1 sm=0 a=YOqqrsh--ucA:10 a=r6Jiw2x_onsA:10 a=8nJEP1OIZ-IA:10 a=qCTYjJJ2g1SgQhsHZAUATw==:17 a=oYZMiAjE7Ods0IdXa1MA:9 a=KxfJEybeSvD3fsaYOsMA:7 a=FIUuaVk_In_6XzoHzP_PsAIWBCEA:4 a=wPNLvfGTeEIA:10 a=UvA89RsQ9BjyT1jR:21 a=phsvAHy9l1VyBep8:21 a=qCTYjJJ2g1SgQhsHZAUATw==:117 X-Cloudmark-Score: 0 X-Originating-IP: 98.27.48.207 Received: from [98.27.48.207] ([98.27.48.207:39735] helo=[192.168.10.104]) by cdptpa-oedge01.mail.rr.com (envelope-from ) (ecelerity 2.2.3.46 r()) with ESMTP id A3/30-09483-B2E029D4; Tue, 29 Mar 2011 16:51:57 +0000 Message-ID: <4D920E29.2030706@dunslane.net> Date: Tue, 29 Mar 2011 12:51:53 -0400 From: Andrew Dunstan User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101209 Fedora/3.1.7-0.35.b3pre.fc13 Lightning/1.0b3pre Thunderbird/3.1.7 MIME-Version: 1.0 To: Guillaume Lelarge CC: Zheng Yang , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Subject: Re: foreign data wrappers References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> <4D91FF65.2090401@lelarge.info> <4D92098D.6040409@dunslane.net> <4D920A3D.8010004@lelarge.info> In-Reply-To: <4D920A3D.8010004@lelarge.info> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201103/41 X-Sequence-Number: 182 On 03/29/2011 12:35 PM, Guillaume Lelarge wrote: > Le 29/03/2011 18:32, Andrew Dunstan a écrit : >> >> On 03/29/2011 11:48 AM, Guillaume Lelarge wrote: >>> Le 29/03/2011 13:28, Zheng Yang a écrit : >>>> Hi guys, >>>> >>>>>> I've briefly gone through the slides. Regarding the 6 callbacks, is >>>>>> that correct to say that a full table scan will always be performed >>>>>> irregardless of the sql statement, >>>>>> the FDW is blind to the sql query performed, right? >>>>> Yes, fairly much. If the feed is large you need some way to pass a >>>>> limit to the foreign side, possibly via table options. I'm fairly >>>>> sure you won't be able to get it via the SELECT statement. >>>>> >>>> Regarding the previous flickr example, I'm wondering how this 'free >>>> text search' function can be done if the FDW is blind to the SELECT >>>> statement. >>>> >>>> For instance, the following query is to retrieve a photo relevant to >>>> 'panda': >>>> >>>> SELECT photo FROM flickr_table WHERE search LIKE '%panda%'; >>>> >>>> In this case, the FDW can only open a connection to flickr web >>>> service and return the next 'row' . >>>> The problem is that there are a huge number of photos in flickr >>>> server and retrieving them sequentially is not realistic. >>>> Any ideas on how this can be done? >>>> >>> It probably means that flickr is not a good example of a nice fdw. >> >> Neither of you are being very creative. As I mentioned above, you need >> to embed this sort of stuff in table options. >> >> so you would have something like: >> >> create foreign table panda_flickr (photo bytea, ...) >> server flickr_server >> options (searchterm 'panda', maxrows '50'); >> select photo from panda_flickr; >> > This would work but means you need to create a new foreign table to > search something else. > > So, yeah, it works, but it's not convenient. The other possibility is that you can dig down into the ForiegnScanState object. The FDW routines are passed a ForeignScanState object which contains a ScanState object which in turn contains a PlanState object which has a list of quals. You probably need to dig quite a bit further but that's a start. cheers andrew From zhengyang4k@gmail.com Wed Mar 30 03:00:50 2011 Received: from maia.hub.org (maia-5.hub.org [200.46.204.29]) by mail.postgresql.org (Postfix) with ESMTP id 968E3133655C for ; Wed, 30 Mar 2011 00:00:50 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.29]) (amavisd-maia, port 10024) with ESMTP id 42336-01 for ; Wed, 30 Mar 2011 03:00:40 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mail.postgresql.org (Postfix) with ESMTP id AB92A1337B47 for ; Wed, 30 Mar 2011 00:00:39 -0300 (ADT) Received: by iyb14 with SMTP id 14so775954iyb.19 for ; Tue, 29 Mar 2011 20:00:38 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:message-id:references:to:x-mailer; bh=nsou/5Dy5qZXQQ+0UbfhVolKYR+khPo6FBka7wUorT4=; b=jl1r5oQZZSxpYsq2ALZxflWTLWh66mc0i6Txs2+IclEj4zvSg2tHA44CwBaCNvKIxb 6tn3ia+uFwUjhGMDCE2w56W9S55n0lvEg0w3r9rACtbhcPvEftOuSD8LxFnTph4BHs5P jIahud9Cir65FW7wQsZ39Ptqihwd9rp9afIxw= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer; b=bcsyfxlVKWd19e5TkyH7j56Fm3sae+B70uwYlIjAhLysHqRmE3j69B2zYcLe1svxLE A7PRpV7uV/Ra72hArGTHIGghW1YB7SWwH62P1AcPvlGDLXEpOf7CDQ/EvFE7wg3vz8qH so7DfcQTCE7KkFodSwMIcFRbl8xtFzeu+zBXM= Received: by 10.42.156.199 with SMTP id a7mr448296icx.417.1301454038496; Tue, 29 Mar 2011 20:00:38 -0700 (PDT) Received: from [172.21.24.1] (nusnet-228-5.dynip.nus.edu.sg [137.132.228.5]) by mx.google.com with ESMTPS id wo11sm3734956icb.20.2011.03.29.20.00.29 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 29 Mar 2011 20:00:35 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: multipart/alternative; boundary=Apple-Mail-1116-1024277115 From: Zheng Yang In-Reply-To: <4D920E29.2030706@dunslane.net> Date: Wed, 30 Mar 2011 11:00:24 +0800 Cc: Guillaume Lelarge , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Message-Id: References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> <4D91FF65.2090401@lelarge.info> <4D92098D.6040409@dunslane.net> <4D920A3D.8010004@lelarge.info> <4D920E29.2030706@dunslane.net> To: Andrew Dunstan X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.898 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001, HTML_MESSAGE=0.001 X-Spam-Level: X-Archive-Number: 201103/42 X-Sequence-Number: 183 --Apple-Mail-1116-1024277115 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=iso-8859-1 Hi Andrew and Guillaume, >=20 >=20 > On 03/29/2011 12:35 PM, Guillaume Lelarge wrote: >> Le 29/03/2011 18:32, Andrew Dunstan a =E9crit : >>>=20 >>> On 03/29/2011 11:48 AM, Guillaume Lelarge wrote: >>>> Le 29/03/2011 13:28, Zheng Yang a =E9crit : >>>>> Hi guys, >>>>>=20 >>>>>>> I've briefly gone through the slides. Regarding the 6 callbacks, = is >>>>>>> that correct to say that a full table scan will always be = performed >>>>>>> irregardless of the sql statement, >>>>>>> the FDW is blind to the sql query performed, right? >>>>>> Yes, fairly much. If the feed is large you need some way to pass = a >>>>>> limit to the foreign side, possibly via table options. I'm fairly >>>>>> sure you won't be able to get it via the SELECT statement. >>>>>>=20 >>>>> Regarding the previous flickr example, I'm wondering how this = 'free >>>>> text search' function can be done if the FDW is blind to the = SELECT >>>>> statement. >>>>>=20 >>>>> For instance, the following query is to retrieve a photo relevant = to >>>>> 'panda': >>>>>=20 >>>>> SELECT photo FROM flickr_table WHERE search LIKE '%panda%'; >>>>>=20 >>>>> In this case, the FDW can only open a connection to flickr web >>>>> service and return the next 'row' . >>>>> The problem is that there are a huge number of photos in flickr >>>>> server and retrieving them sequentially is not realistic. >>>>> Any ideas on how this can be done? >>>>>=20 >>>> It probably means that flickr is not a good example of a nice fdw. >>>=20 >>> Neither of you are being very creative. As I mentioned above, you = need >>> to embed this sort of stuff in table options. >>>=20 >>> so you would have something like: >>>=20 >>> create foreign table panda_flickr (photo bytea, ...) >>> server flickr_server >>> options (searchterm 'panda', maxrows '50'); >>> select photo from panda_flickr; >>>=20 >> This would work but means you need to create a new foreign table to >> search something else. >>=20 >> So, yeah, it works, but it's not convenient. >=20 > The other possibility is that you can dig down into the = ForiegnScanState object. The FDW routines are passed a ForeignScanState = object which contains a ScanState object which in turn contains a = PlanState object which has a list of quals. You probably need to dig = quite a bit further but that's a start. I think this is a common issue for all FDWs that need to access remote = resources over a network. For example, if there were a Mysql FDW, a full = table scan implies the whole table will be transferred over.=20 it is not quite efficient for large tables.=20 If a table size is 1GB, iterating the whole table row by row means those = 1GB of data needs to be transferred over. This may take hours even if = for an sql statement as simple as =20 SELECT * from table where id =3D 1; cheers, ZY --Apple-Mail-1116-1024277115 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=iso-8859-1


On 03/29/2011 12:35 PM, Guillaume Lelarge = wrote:
Le 29/03/2011 18:32, Andrew Dunstan = a =E9crit :

On 03/29/2011 11:48 AM, = Guillaume Lelarge wrote:
Le = 29/03/2011 13:28, Zheng Yang a =E9crit = :
Hi = guys,

I've briefly gone through the = slides. Regarding the 6 callbacks, = is
that correct to say that a full = table scan will always be = performed
irregardless of the sql = statement,
the = FDW is blind to the sql query performed, = right?
Yes, fairly much. If the feed is = large you need some way to pass = a
limit = to the foreign side, possibly via table options. I'm = fairly
sure = you won't be able to get it via the SELECT = statement.

Regarding the previous flickr = example, I'm wondering how this = 'free
text search' function can be = done if the FDW is blind to the = SELECT
statement.

For instance, the following = query is to retrieve a photo relevant = to
'panda':

    SELECT = photo FROM flickr_table WHERE search LIKE = '%panda%';

In this case, the FDW can only = open a connection to flickr = web
service and return the next = 'row' = .
The problem is that there are a = huge number of photos in = flickr
server and retrieving them = sequentially is not = realistic.
Any ideas on how this can be = done?

It probably means that flickr is not a good example of a = nice fdw.

Neither of you are being very = creative. As I mentioned above, you = need
to embed this sort of stuff in table = options.

so you would have something = like:

   create = foreign table panda_flickr (photo bytea, = ...)
        server = flickr_server
=         options (searchterm = 'panda', maxrows '50');
   select photo = from panda_flickr;

This= would work but means you need to create a new foreign table = to
search something = else.

So, yeah, it = works, but it's not convenient.

The other = possibility is that you can dig down into the ForiegnScanState object. = The FDW routines are passed a ForeignScanState object which contains a = ScanState object which in turn contains a PlanState object which has a = list of quals. You probably need to dig quite a bit further but that's a = start.

I think this is a common issue for all FDWs that need to access = remote resources over a network. For example, if there were a Mysql FDW, = a full table scan implies the whole table will be transferred = over. 
it is not quite efficient for large = tables. 

If a table size is 1GB, iterating = the whole table row by row means those 1GB of data needs to be = transferred over. This may take hours even if for an sql statement as = simple as  
SELECT * from table where id =3D = 1;

cheers,
ZY

=


= --Apple-Mail-1116-1024277115-- From zhengyang4k@gmail.com Thu Mar 31 08:55:50 2011 Received: from maia.hub.org (maia-3.hub.org [200.46.204.243]) by mail.postgresql.org (Postfix) with ESMTP id 25E631337BA3 for ; Thu, 31 Mar 2011 05:55:50 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.243]) (amavisd-maia, port 10024) with ESMTP id 40875-04 for ; Thu, 31 Mar 2011 08:55:39 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail-iy0-f174.google.com (mail-iy0-f174.google.com [209.85.210.174]) by mail.postgresql.org (Postfix) with ESMTP id B71BE1337B97 for ; Thu, 31 Mar 2011 05:55:39 -0300 (ADT) Received: by iyb14 with SMTP id 14so2148163iyb.19 for ; Thu, 31 Mar 2011 01:55:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:subject:mime-version:content-type:from :in-reply-to:date:cc:content-transfer-encoding:message-id:references :to:x-mailer; bh=sX1h8bgdHDdvbS0t9z2m2ytLnfC6OmubGWJDdVZE/Mw=; b=sHuZe9Y9f6HOVK9FAFN6UFjaLqw8edL9zMD9ar2zj778hEX0T5WnpvtMKas5U9XPh0 dDS1EwX+PZdPoZ56HC1VRV47E1aGrUDpDJNEy6k3yyPDckAUbKmqXfzNxOivRn0ShVb+ 7ANODwHJ/qe69Xy65rDkkhtTyGr1mkO2TG90w= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer; b=bOgIAz3SVR3Zx2+yX9TcBOyBCv1PBPa9i+doSE5S/KoUxMdYzocg1+pa8McxZAbGH5 ffY7EVrpg7us5K9UI1VgdXVanMYRDwFhKR3EI4oMFKb71X+wUzTn5jt0BPdDhxvOW2qD bqznpRA8TMt+7AU4KQjbl6ao0kNOCsjr/Ownk= Received: by 10.42.240.134 with SMTP id la6mr2870598icb.347.1301561739634; Thu, 31 Mar 2011 01:55:39 -0700 (PDT) Received: from [172.16.36.3] ([74.117.56.62]) by mx.google.com with ESMTPS id y10sm608984iba.63.2011.03.31.01.55.32 (version=TLSv1/SSLv3 cipher=OTHER); Thu, 31 Mar 2011 01:55:36 -0700 (PDT) Subject: Re: foreign data wrappers Mime-Version: 1.0 (Apple Message framework v1082) Content-Type: text/plain; charset=us-ascii From: Zheng Yang In-Reply-To: <4D920E29.2030706@dunslane.net> Date: Thu, 31 Mar 2011 16:55:29 +0800 Cc: Guillaume Lelarge , Shiv , Selena Deckelmann , Alvaro Herrera , pgsql-students Content-Transfer-Encoding: 7bit Message-Id: <78D7FF0C-EE85-479F-9C64-077B37E70B96@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> <4D91FF65.2090401@lelarge.info> <4D92098D.6040409@dunslane.net> <4D920A3D.8010004@lelarge.info> <4D920E29.2030706@dunslane.net> To: Andrew Dunstan X-Mailer: Apple Mail (2.1082) X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.899 tagged_above=-10 required=5 tests=BAYES_00=-1.9, FREEMAIL_FROM=0.001 X-Spam-Level: X-Archive-Number: 201103/44 X-Sequence-Number: 185 Hi guys, What does this 'quantifiable results' mean for proposals? Thanks! Regards, ZY From selena@chesnok.com Thu Mar 31 12:05:33 2011 Received: from maia.hub.org (maia-2.hub.org [200.46.204.251]) by mail.postgresql.org (Postfix) with ESMTP id ECED41337B93 for ; Thu, 31 Mar 2011 09:05:33 -0300 (ADT) Received: from mail.postgresql.org ([200.46.204.86]) by maia.hub.org (mx1.hub.org [200.46.204.251]) (amavisd-maia, port 10024) with ESMTP id 07704-06 for ; Thu, 31 Mar 2011 12:05:14 +0000 (UTC) X-Greylist: from auto-whitelisted by SQLgrey-1.7.6 Received: from homiemail-a35.g.dreamhost.com (mailbigip.dreamhost.com [208.97.132.5]) by mail.postgresql.org (Postfix) with ESMTP id 8D7641337B82 for ; Thu, 31 Mar 2011 09:05:14 -0300 (ADT) Received: from homiemail-a35.g.dreamhost.com (localhost [127.0.0.1]) by homiemail-a35.g.dreamhost.com (Postfix) with ESMTP id 682685406F for ; Thu, 31 Mar 2011 05:05:14 -0700 (PDT) DomainKey-Signature: a=rsa-sha1; c=nofws; d=chesnok.com; h=mime-version :in-reply-to:references:date:message-id:subject:from:to:cc: content-type; q=dns; s=chesnok.com; b=J0WQbL9r5ynh6T6WRfLnAk4/rz EAsNAwlWqy0+SJkLVtjbWrDN4K9h4jdFu438xTw1XINahimLsa8c04ABM5SJ4eAm mS+tH/d+ldoG1W1Xtnjt+MkBaID29Imjk7aqEv9yFmhpdxzSAhDZfBU/KWLVvYHq iNhvxFWXeOSlLIg6Y= DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=chesnok.com; h= mime-version:in-reply-to:references:date:message-id:subject:from :to:cc:content-type; s=chesnok.com; bh=lB8GWshbVZWdsFqH8b2P87loX 6M=; b=tjzDZC4ShO0npObO5ey/odgPJ6dIZe2pDWPN3PJFLc5qlLUGx8P9qVYgB Eiz51JlC1UhNI6iDWZ6OlpvWCVt7+4aBjUmNGjKSqhc4jsEKeJaNYXRoP0PKkP0l ykRV8uznH90rxx7gId1Bqij2b9vJ4sLuZyolDeWK5fRspwYGvE= Received: from mail-fx0-f46.google.com (mail-fx0-f46.google.com [209.85.161.46]) (using TLSv1 with cipher RC4-SHA (128/128 bits)) (No client certificate requested) (Authenticated sender: selena@chesnok.com) by homiemail-a35.g.dreamhost.com (Postfix) with ESMTPSA id 0D1FF54058 for ; Thu, 31 Mar 2011 05:05:13 -0700 (PDT) Received: by fxm17 with SMTP id 17so1776100fxm.19 for ; Thu, 31 Mar 2011 05:05:12 -0700 (PDT) MIME-Version: 1.0 Received: by 10.223.112.83 with SMTP id v19mr130517fap.122.1301573112657; Thu, 31 Mar 2011 05:05:12 -0700 (PDT) Received: by 10.223.126.7 with HTTP; Thu, 31 Mar 2011 05:05:12 -0700 (PDT) In-Reply-To: <78D7FF0C-EE85-479F-9C64-077B37E70B96@gmail.com> References: <690867AB-C50B-4354-91D0-8A0811639C4C@gmail.com> <4D879CD0.5030803@dunslane.net> <3EF9E4CC-CF42-4B45-86FF-4CB9D46D7442@gmail.com> <4D8AF8C2.3000007@lelarge.info> <4ACC318A-1B6F-49A5-9B02-49EEAF7B97C7@gmail.com> <1300985156-sup-3058@alvh.no-ip.org> <47676.208.91.122.102.1300985713.squirrel@dunslane.net> <603812A0-F464-44AE-8A90-DF9BD1DC2DB9@gmail.com> <4D8E0125.9060905@dunslane.net> <8DB4D3E9-DA21-45FD-A2A2-5C6D479AA565@gmail.com> <4D91FF65.2090401@lelarge.info> <4D92098D.6040409@dunslane.net> <4D920A3D.8010004@lelarge.info> <4D920E29.2030706@dunslane.net> <78D7FF0C-EE85-479F-9C64-077B37E70B96@gmail.com> Date: Thu, 31 Mar 2011 05:05:12 -0700 Message-ID: Subject: Re: foreign data wrappers From: Selena Deckelmann To: Zheng Yang Cc: pgsql-students Content-Type: text/plain; charset=ISO-8859-1 X-Virus-Scanned: Maia Mailguard 1.0.1 X-Spam-Status: No, hits=-1.9 tagged_above=-10 required=5 tests=BAYES_00=-1.9, RCVD_IN_DNSWL_NONE=-0.0001 X-Spam-Level: X-Archive-Number: 201103/46 X-Sequence-Number: 187 Hi! On Thu, Mar 31, 2011 at 1:55 AM, Zheng Yang wrote: > What does this 'quantifiable results' mean for proposals? Thanks! It means what will the project deliver in terms of a feature, or more than one feature, that can be evaluated. Like: * Implement a FDW for Drizzle that is capable of read-only access to tables with datatypes of int, float, varchar, char and timedate. etc. -selena -- http://chesnok.com