From: Etsuro Fujita <fujita.etsuro@lab.ntt.co.jp>
To: 'Shigeru HANADA' <shigeru.hanada@gmail.com>
Cc: pgsql-hackers@postgresql.org
Subject: Re: WIP: Collecting statistics on CSV file data
Date: Fri, 6 Apr 2012 14:16:08 +0900
Message-ID: <005a01cd13b4$5f890a30$1e9b1e90$@lab.ntt.co.jp> (raw)
In-Reply-To: <4F7E57D9.3000807@gmail.com>
References: <4E6DE18B.4010505@lab.ntt.co.jp>
<4E7861C1.2000607@gmail.com>
<4E8EC1D8.3080700@lab.ntt.co.jp>
<4E9C6573.8050507@gmail.com>
<4E9FF04B.3050504@lab.ntt.co.jp>
<4EB7C07A.5030205@gmail.com>
<4EC60883.2050905@lab.ntt.co.jp>
<4EC648CA.5050904@gmail.com>
<4EC9CBC5.3070500@lab.ntt.co.jp>
<4EE1FC38.5080707@lab.ntt.co.jp>
<4EE5D862.1080809@gmail.com>
<4EE74C5F.3000309@lab.ntt.co.jp>
<4F7D8BBF.4050004@gmail.com>
<4F7E57D9.3000807@gmail.com>
Thanks, Hanada-san!
Best regards,
Etsuro Fujita
-----Original Message-----
From: Shigeru HANADA [mailto:shigeru.hanada@gmail.com]
Sent: Friday, April 06, 2012 11:41 AM
To: Etsuro Fujita
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] WIP: Collecting statistics on CSV file data
(2012/04/05 21:10), Shigeru HANADA wrote:
> file_fdw
> ========
> This patch contains a use case of new handler function in
> contrib/file_fdw. Since file_fdw reads data from a flat file,
> fileAnalyzeForeignTable uses similar algorithm to ordinary tables; it
> samples first N rows first, and replaces them randomly with subsequent
> rows. Also file_fdw updates pg_class.relpages by calculating number
> of pages from size of the data file.
>
> To allow FDWs to implement sampling argorighm like this, several
> functions are exported from analyze.c, e.g. random_fract,
> init_selection_state, and get_next_S.
Just after my post, Fujita-san posted another v7 patch[1], so I merged
v7 patches into v8 patch.
[1] http://archives.postgresql.org/pgsql-hackers/2012-04/msg00212.php
Changes taken from Fujita-san's patch
=====================================
* Remove reporting validrows and deadrows at the end of acquire_sample_rows
of file_fdw. Thus, it doesn't validate NOT NULL constraints any more.
* Improve get_rel_size of file_fdw, which is used in GetForeignRelSize, to
estimate current # of tuples of the foreign table from these values.
- # of pages/tuples which are updated by last ANALYZE
- current file size
Additional Changes
==================
* Fix memory leak in acquire_sample_rows which caused by calling
NextCopyFrom repeatedly in one long-span memory context. I add per-record
temporary context and it's used during processing a record.
Main context is used to create heap tuples from sampled records, because
sample tuples must be valid after the function ends.
* Some cosmetic changes for document, e.g. remove line-break inside tagged
elements.
* Some cosmetic changes to make patch more readable by minimizing difference
from master branch.
Changes did *not* merged
========================
* Fujita-san moved document of AnalyzeForeignTable to the section "Foreign
Data Wrapper Helper Functions" from "Foreign Data Wrapper Callback
Routines". But I think analyze handler is one of callback functions, though
it's optional.
Please find attached a patch.
Regards,
--
Shigeru HANADA
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Reply to all the recipients using the --to and --cc options:
reply via email
To: pgsql-hackers@postgresql.org
Cc: fujita.etsuro@lab.ntt.co.jp, shigeru.hanada@gmail.com
Subject: Re: WIP: Collecting statistics on CSV file data
In-Reply-To: <005a01cd13b4$5f890a30$1e9b1e90$@lab.ntt.co.jp>
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox