pg.ddx.io pgsql-hackers@postgresql.org mailing list archive
help / color / mirror / Atom feedSolution of the file name problem of copy on windows.
12+ messages / 5 participants
[nested] [flat]
* Solution of the file name problem of copy on windows.
@ 2009-04-07 16:40 Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
0 siblings, 2 replies; 12+ messages in thread
From: Hiroshi Saito @ 2009-04-07 16:40 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
Hi Tom-san.
I want to solve one problem before the release of 8.4.
However, since it also seems to be the new feature,
if not enough for 8.4, you may suggest that it is 8.5.
In Japan, the local file name of a server is dealt with by SJIS.
The example present Postgres...
server_encoding = UTF-8
client_encoding = SJIS
At this time, a copy file name is UTF-8. It was troubled by handling.:-(
Then, I make this proposal patch.
regression test
=======================
All 120 tests passed.
=======================
as for database is UTF-8.
HIROSHI=# \l
$B%G!<%?%Y!<%90lMw(B
$BL>A0(B | $B=jM-<T(B | $B%(%s%3!<%G%#%s%0(B | Collation | Ctype | $B%"%/%;%98"(B
-----------+---------+------------------+-----------+-------+-------------------
--
HIROSHI | HIROSHI | UTF8 | C | C |
eucdb | HIROSHI | EUC_JP | C | C |
HIROSHI=# create table $BF|K\8l$F$9$H(B ($B$-!<(B text);
CREATE TABLE
HIROSHI=# insert into $BF|K\8l$F$9$H(B values('$B$o!<$$(B');
INSERT 0 1
HIROSHI=# copy $BF|K\8l$F$9$H(B to 'C:/tmp/$BF|K\8l(BUTF8.txt';
COPY 1
HIROSHI=# delete from $BF|K\8l$F$9$H(B;
DELETE 1
HIROSHI=# copy $BF|K\8l$F$9$H(B from 'C:/tmp/$BF|K\8l(BUTF8.txt';
COPY 1
HIROSHI=# select * from $BF|K\8l$F$9$H(B;
$B$-!<(B
--------
$B$o!<$$(B
(1 $B9T(B)
as for database is eucjp.
HIROSHI=# \c eucdb
psql (8.4devel)
$B%G!<%?%Y!<%9(B "eucdb" $B$K@\B3$7$^$7$?!#(B.
eucdb=# \d
$B%j%l!<%7%g%s$N0lMw(B
$B%9%-!<%^(B | $BL>A0(B | $B7?(B | $B=jM-<T(B
----------+--------------+-------+---------
public | $BF|K\8l$F$9$H(B | table | HIROSHI
(1 $B9T(B)
eucdb=# select * from $BF|K\8l$F$9$H(B;
$B$-!<(B
--------
$B$o!<$$(B
(1 $B9T(B)
eucdb=# copy $BF|K\8l$F$9$H(B to 'C:/tmp/$BF|K\8l(Beucdb.txt';
COPY 1
eucdb=# delete from $BF|K\8l$F$9$H(B;
DELETE 1
eucdb=# copy $BF|K\8l$F$9$H(B from 'C:/tmp/$BF|K\8l(Beucdb.txt';
COPY 1
eucdb=# select * from $BF|K\8l$F$9$H(B;
$B$-!<(B
--------
$B$o!<$$(B
(1 $B9T(B)
C:\tmp>dir $BF|K\8l(B*
$B%I%i%$%V(B C $B$N%\%j%e!<%`(B $B%i%Y%k$O(B SYS $B$G$9(B
$B%\%j%e!<%`(B $B%7%j%"%kHV9f$O(B 1433-2C7C $B$G$9(B
C:\tmp $B$N%G%#%l%/%H%j(B
2009/04/07 13:58 8 $BF|K\8l(Beucdb.txt
2009/04/07 13:58 8 $BF|K\8l(Butf8.txt
2 $B8D$N%U%!%$%k(B 16 $B%P%$%H(B
It seems that it is very comfortable. !!
What do you think?
Regards,
Hiroshi Saito
Attachments:
copy_patch3 (application/octet-stream, 3.1K) download
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-07 17:46 Tom Lane <tgl@sss.pgh.pa.us>
parent: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
1 sibling, 1 reply; 12+ messages in thread
From: Tom Lane @ 2009-04-07 17:46 UTC (permalink / raw)
To: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; +Cc: pgsql-hackers
"Hiroshi Saito" <z-saito@guitar.ocn.ne.jp> writes:
> I want to solve one problem before the release of 8.4.
> However, since it also seems to be the new feature,
> if not enough for 8.4, you may suggest that it is 8.5.
I'm not too clear on what this is really supposed to accomplish, but
we are hardly going to put code like that into every single file access
in Postgres, which is what seems to be the logical implication.
Shouldn't we just tell people to use a database encoding that matches
their system environment?
regards, tom lane
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-08 02:19 Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
parent: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
1 sibling, 2 replies; 12+ messages in thread
From: Itagaki Takahiro @ 2009-04-08 02:19 UTC (permalink / raw)
To: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi,
"Hiroshi Saito" <z-saito@guitar.ocn.ne.jp> wrote:
> At this time, a copy file name is UTF-8. It was troubled by handling.:-(
> Then, I make this proposal patch.
I think the problem is not only in Windows but also in all platforms
where the database encoding doesn't match their OS's encoding.
Instead of Windows specific codes, how about adding GetPlatformEncoding()
and convert all of *absolute* paths? It would be performed at the lowest
API layer; i.e, BasicOpenFile(). Standard database file accesses with
RelFileNode are not affected because is uses *relative* paths.
There are some issues:
* Is it possible to determine the platform encoding?
* The above cannot handle non-ascii path under $PGDATA.
Is it acceptable?
* In Windows, the native encoding is UTF-16, but we will use SJIS
if we take on the above method. Is the limitation acceptable?
Comments welcome.
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-08 10:31 Hiroshi Inoue <inoue@tpf.co.jp>
parent: Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 1 reply; 12+ messages in thread
From: Hiroshi Inoue @ 2009-04-08 10:31 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; pgsql-hackers
Tom Lane wrote:
> "Hiroshi Saito" <z-saito@guitar.ocn.ne.jp> writes:
>> I want to solve one problem before the release of 8.4.
>> However, since it also seems to be the new feature,
>> if not enough for 8.4, you may suggest that it is 8.5.
>
> I'm not too clear on what this is really supposed to accomplish, but
> we are hardly going to put code like that into every single file access
> in Postgres, which is what seems to be the logical implication.
> Shouldn't we just tell people to use a database encoding that matches
> their system environment?
Unfortunately (as usual) under Japanese Windows there's no database
encoding that matches the system environment.
As for the file name in COPY command, there's little meaning to
convert it to the server encoding because the file name is irrelevant
to the database. Because Windows is Unicode(UTF-16) based, it seems
natural to convert the file name to wide characters once.
regards,
Hiroshi Inoue
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-08 14:00 Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
parent: Hiroshi Inoue <inoue@tpf.co.jp>
0 siblings, 0 replies; 12+ messages in thread
From: Hiroshi Saito @ 2009-04-08 14:00 UTC (permalink / raw)
To: Hiroshi Inoue <inoue@tpf.co.jp>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
Hi.
----- Original Message -----
From: "Hiroshi Inoue" <inoue@tpf.co.jp>
> Tom Lane wrote:
>> "Hiroshi Saito" <z-saito@guitar.ocn.ne.jp> writes:
>>> I want to solve one problem before the release of 8.4.
>>> However, since it also seems to be the new feature,
>>> if not enough for 8.4, you may suggest that it is 8.5.
>>
>> I'm not too clear on what this is really supposed to accomplish, but
>> we are hardly going to put code like that into every single file access
>> in Postgres, which is what seems to be the logical implication.
>> Shouldn't we just tell people to use a database encoding that matches
>> their system environment?
>
> Unfortunately (as usual) under Japanese Windows there's no database
> encoding that matches the system environment.
> As for the file name in COPY command, there's little meaning to
> convert it to the server encoding because the file name is irrelevant
> to the database. Because Windows is Unicode(UTF-16) based, it seems
> natural to convert the file name to wide characters once.
Yes, If server encoding can be chosen by windows, the facilities
in good working order. It was not possible though it was regrettable.
Regards,
Hiroshi Saito
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-08 16:01 Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
parent: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
1 sibling, 1 reply; 12+ messages in thread
From: Hiroshi Saito @ 2009-04-08 16:01 UTC (permalink / raw)
To: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>; +Cc: Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Hi Itagaki-san.
Um, I had a focus in help the problem which is not avoided.
I am not sensitive to a problem being avoided depending on usage.
However, I will wish to work spontaneously, when it is help much.
Regards,
Hiroshi Saito
----- Original Message -----
From: "Itagaki Takahiro" <itagaki.takahiro@oss.ntt.co.jp>
> Hi,
>
> "Hiroshi Saito" <z-saito@guitar.ocn.ne.jp> wrote:
>
>> At this time, a copy file name is UTF-8. It was troubled by handling.:-(
>> Then, I make this proposal patch.
>
> I think the problem is not only in Windows but also in all platforms
> where the database encoding doesn't match their OS's encoding.
>
> Instead of Windows specific codes, how about adding GetPlatformEncoding()
> and convert all of *absolute* paths? It would be performed at the lowest
> API layer; i.e, BasicOpenFile(). Standard database file accesses with
> RelFileNode are not affected because is uses *relative* paths.
>
> There are some issues:
> * Is it possible to determine the platform encoding?
> * The above cannot handle non-ascii path under $PGDATA.
> Is it acceptable?
> * In Windows, the native encoding is UTF-16, but we will use SJIS
> if we take on the above method. Is the limitation acceptable?
>
> Comments welcome.
>
> Regards,
> ---
> ITAGAKI Takahiro
> NTT Open Source Software Center
>
>
>
> --
> Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
> To make changes to your subscription:
> http://www.postgresql.org/mailpref/pgsql-hackers
>
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-09 00:22 Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
parent: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
0 siblings, 1 reply; 12+ messages in thread
From: Itagaki Takahiro @ 2009-04-09 00:22 UTC (permalink / raw)
To: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; +Cc: pgsql-hackers
"Hiroshi Saito" <z-saito@guitar.ocn.ne.jp> wrote:
> Um, I had a focus in help the problem which is not avoided.
> I am not sensitive to a problem being avoided depending on usage.
> However, I will wish to work spontaneously, when it is help much.
I'll research whether encoding of filesystem path is affected by
locale settings or not in some platforms. Also, we need to research
where we should get the system encoding when the locale is set to "C",
which is popular in Japanese users.
I'll report to you the progress :)
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-13 10:13 Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
parent: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
0 siblings, 1 reply; 12+ messages in thread
From: Itagaki Takahiro @ 2009-04-13 10:13 UTC (permalink / raw)
To: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; pgsql-hackers
Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp> wrote:
> "Hiroshi Saito" <z-saito@guitar.ocn.ne.jp> wrote:
>
> > Um, I had a focus in help the problem which is not avoided.
> > I am not sensitive to a problem being avoided depending on usage.
> > However, I will wish to work spontaneously, when it is help much.
>
> I'll research whether encoding of filesystem path is affected by
> locale settings or not in some platforms. Also, we need to research
> where we should get the system encoding when the locale is set to "C",
> which is popular in Japanese users.
Here is a patch to implement GetPlatformEncoding() and convert absolute
file paths from database encoding to platform encoding. Since encoding
of paths are converted at AllocateFile() and BasicOpenFile(), not only
COPY TO/FROM but also almost of file operations are covered by the patch.
Callers of file access methods don't have to modify their codes.
Please test the patch in a variety of platforms. I tested it on Windows
and Linux, and then I found {PG_UTF8, "ANSI_X3.4-1968"} is required for
encoding_match_list in src/port/chklocale.c on Linux (FC6).
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
Attachments:
GetPlatformEncoding.patch (application/octet-stream, 6.3K) download
diff -cpr HEAD/src/backend/storage/file/fd.c GetPlatformEncoding/src/backend/storage/file/fd.c
*** HEAD/src/backend/storage/file/fd.c Thu Mar 5 13:14:27 2009
--- GetPlatformEncoding/src/backend/storage/file/fd.c Mon Apr 13 18:45:06 2009
***************
*** 49,54 ****
--- 49,55 ----
#include <sys/resource.h> /* for getrlimit */
#endif
+ #include "mb/pg_wchar.h"
#include "miscadmin.h"
#include "access/xact.h"
#include "catalog/pg_tablespace.h"
*************** static File OpenTemporaryFileInTablespac
*** 247,252 ****
--- 248,254 ----
static void AtProcExit_Files(int code, Datum arg);
static void CleanupTempFiles(bool isProcExit);
static void RemovePgTempFilesInDir(const char *tmpdirname);
+ static char *GetPlatformPath(const char *path);
/*
*************** int
*** 494,505 ****
BasicOpenFile(FileName fileName, int fileFlags, int fileMode)
{
int fd;
tryAgain:
! fd = open(fileName, fileFlags, fileMode);
if (fd >= 0)
return fd; /* success! */
if (errno == EMFILE || errno == ENFILE)
{
--- 496,514 ----
BasicOpenFile(FileName fileName, int fileFlags, int fileMode)
{
int fd;
+ char *path;
+
+ path = GetPlatformPath(fileName);
tryAgain:
! fd = open(path, fileFlags, fileMode);
if (fd >= 0)
+ {
+ if (path != fileName)
+ pfree(path);
return fd; /* success! */
+ }
if (errno == EMFILE || errno == ENFILE)
{
*************** tryAgain:
*** 514,519 ****
--- 523,530 ----
errno = save_errno;
}
+ if (path != fileName)
+ pfree(path);
return -1; /* failure */
}
*************** FILE *
*** 1336,1345 ****
--- 1347,1359 ----
AllocateFile(const char *name, const char *mode)
{
FILE *file;
+ char *path;
DO_DB(elog(LOG, "AllocateFile: Allocated %d (%s)",
numAllocatedDescs, name));
+ path = GetPlatformPath(name);
+
/*
* The test against MAX_ALLOCATED_DESCS prevents us from overflowing
* allocatedFiles[]; the test against max_safe_fds prevents AllocateFile
*************** AllocateFile(const char *name, const cha
*** 1351,1357 ****
elog(ERROR, "too many private files demanded");
TryAgain:
! if ((file = fopen(name, mode)) != NULL)
{
AllocateDesc *desc = &allocatedDescs[numAllocatedDescs];
--- 1365,1371 ----
elog(ERROR, "too many private files demanded");
TryAgain:
! if ((file = fopen(path, mode)) != NULL)
{
AllocateDesc *desc = &allocatedDescs[numAllocatedDescs];
*************** TryAgain:
*** 1359,1364 ****
--- 1373,1380 ----
desc->desc.file = file;
desc->create_subid = GetCurrentSubTransactionId();
numAllocatedDescs++;
+ if (path != name)
+ pfree(path);
return desc->desc.file;
}
*************** TryAgain:
*** 1375,1380 ****
--- 1391,1398 ----
errno = save_errno;
}
+ if (path != name)
+ pfree(path);
return NULL;
}
*************** RemovePgTempFilesInDir(const char *tmpdi
*** 1875,1877 ****
--- 1893,1905 ----
FreeDir(temp_dir);
}
+
+ static char *
+ GetPlatformPath(const char *path)
+ {
+ if (is_absolute_path(path))
+ return (char *) pg_do_encoding_conversion((unsigned char *) path,
+ strlen(path), GetDatabaseEncoding(), GetPlatformEncoding());
+ else
+ return (char *) path;
+ }
diff -cpr HEAD/src/backend/utils/mb/mbutils.c GetPlatformEncoding/src/backend/utils/mb/mbutils.c
*** HEAD/src/backend/utils/mb/mbutils.c Thu Apr 9 11:56:48 2009
--- GetPlatformEncoding/src/backend/utils/mb/mbutils.c Mon Apr 13 18:41:58 2009
*************** pg_client_encoding(PG_FUNCTION_ARGS)
*** 1009,1011 ****
--- 1009,1091 ----
Assert(ClientEncoding);
return DirectFunctionCall1(namein, CStringGetDatum(ClientEncoding->name));
}
+
+ #ifdef WIN32
+
+ struct codepage_to_encoding
+ {
+ UINT codepage;
+ enum pg_enc encoding;
+ };
+
+ static const struct codepage_to_encoding codepage2encoding[] =
+ {
+ {866, PG_WIN866},
+ {874, PG_WIN874},
+ {932, PG_SJIS},
+ {936, PG_GBK},
+ {950, PG_BIG5},
+ {1250, PG_WIN1250},
+ {1251, PG_WIN1251},
+ {1252, PG_WIN1252},
+ {1253, PG_WIN1253},
+ {1254, PG_WIN1254},
+ {1255, PG_WIN1255},
+ {1256, PG_WIN1256},
+ {1257, PG_WIN1257},
+ {1258, PG_WIN1258},
+ {1361, PG_JOHAB},
+ {20866, PG_KOI8R},
+ {20932, PG_EUC_JP},
+ {20936, PG_EUC_CN},
+ {21866, PG_KOI8U},
+ {28591, PG_LATIN1},
+ {28592, PG_LATIN2},
+ {28593, PG_LATIN3},
+ {28594, PG_LATIN4},
+ {28595, PG_ISO_8859_5},
+ {28596, PG_ISO_8859_6},
+ {28597, PG_ISO_8859_7},
+ {28598, PG_ISO_8859_8},
+ {28599, PG_LATIN5},
+ {28605, PG_LATIN9},
+ {51949, PG_EUC_KR}, /* or 20949 ? */
+ {54936, PG_GB18030},
+ {65001, PG_UTF8}
+ };
+
+ static int
+ compare_codepage(const void *v1, const void *v2)
+ {
+ UINT cp1 = ((const struct codepage_to_encoding *) v1)->codepage;
+ UINT cp2 = ((const struct codepage_to_encoding *) v2)->codepage;
+
+ if (cp1 < cp2)
+ return -1;
+ else if (cp1 > cp2)
+ return +1;
+ else
+ return 0;
+ }
+
+ int
+ GetPlatformEncoding(void)
+ {
+ struct codepage_to_encoding key;
+ const struct codepage_to_encoding *found;
+
+ key.codepage = GetACP();
+ found = bsearch(&key, codepage2encoding, lengthof(codepage2encoding),
+ sizeof(key), compare_codepage);
+ return found ? found->encoding : PG_SQL_ASCII;
+ }
+
+ #else
+
+ int
+ GetPlatformEncoding(void)
+ {
+ return pg_get_encoding_from_locale("");
+ }
+
+ #endif
diff -cpr HEAD/src/include/mb/pg_wchar.h GetPlatformEncoding/src/include/mb/pg_wchar.h
*** HEAD/src/include/mb/pg_wchar.h Thu Apr 9 11:56:48 2009
--- GetPlatformEncoding/src/include/mb/pg_wchar.h Mon Apr 13 18:41:58 2009
*************** extern void SetDatabaseEncoding(int enco
*** 392,397 ****
--- 392,398 ----
extern int GetDatabaseEncoding(void);
extern const char *GetDatabaseEncodingName(void);
extern void pg_bind_textdomain_codeset(const char *domainname);
+ extern int GetPlatformEncoding(void);
extern int pg_valid_client_encoding(const char *name);
extern int pg_valid_server_encoding(const char *name);
diff -cpr HEAD/src/port/chklocale.c GetPlatformEncoding/src/port/chklocale.c
*** HEAD/src/port/chklocale.c Fri Feb 13 09:58:01 2009
--- GetPlatformEncoding/src/port/chklocale.c Mon Apr 13 19:00:20 2009
*************** static const struct encoding_match encod
*** 73,78 ****
--- 73,79 ----
{PG_UTF8, "UTF-8"},
{PG_UTF8, "utf8"},
{PG_UTF8, "CP65001"},
+ {PG_UTF8, "ANSI_X3.4-1968"},
{PG_LATIN1, "ISO-8859-1"},
{PG_LATIN1, "ISO8859-1"},
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-13 14:14 Tom Lane <tgl@sss.pgh.pa.us>
parent: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
0 siblings, 2 replies; 12+ messages in thread
From: Tom Lane @ 2009-04-13 14:14 UTC (permalink / raw)
To: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>; +Cc: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; pgsql-hackers
Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:
> Here is a patch to implement GetPlatformEncoding() and convert absolute
> file paths from database encoding to platform encoding.
This seems like a fairly significant overhead added to solve a really
minor problem (if it's not minor why has it never come up before?).
I'm also not convinced by any of the details --- why are GetACP and
pg_get_encoding_from_locale the things to look at, and why is fd.c an
appropriate place to hook in? Surely if we need it here, we need it in
places like initdb as well. But really this is much too low a level to
be solving the problem at. If we have to convert path encodings in the
backend, we should be doing it once somewhere around the place where we
identify the value of PGDATA. It should not be necessary to repeat all
this for every file access within the database directory.
regards, tom lane
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-13 15:12 Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 12+ messages in thread
From: Hiroshi Saito @ 2009-04-13 15:12 UTC (permalink / raw)
To: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>; Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-hackers
Hi.
Anyhow, I appreciate discussion.
----- Original Message -----
From: "Tom Lane" <tgl@sss.pgh.pa.us>
> Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:
>> Here is a patch to implement GetPlatformEncoding() and convert absolute
>> file paths from database encoding to platform encoding.
>
> This seems like a fairly significant overhead added to solve a really
> minor problem (if it's not minor why has it never come up before?).
>
> I'm also not convinced by any of the details --- why are GetACP and
> pg_get_encoding_from_locale the things to look at, and why is fd.c an
> appropriate place to hook in? Surely if we need it here, we need it in
> places like initdb as well. But really this is much too low a level to
> be solving the problem at. If we have to convert path encodings in the
> backend, we should be doing it once somewhere around the place where we
> identify the value of PGDATA. It should not be necessary to repeat all
> this for every file access within the database directory.
Ahh, I think this is a sensitive problem and requires careful handling too.
However, following tests are shown in order to help your understanding.
This is the case which can't be operated if no apply the patch of Itagaki-san.
C:\work>set PGDATA=C:\tmp\$BF|K\8l(B data
C:\work>set PGPORT=5444
C:\work>set PGHOME=C:\MinGW\local\pgsql
C:\work>cmd.exe
Microsoft Windows XP [Version 5.1.2600]
(C) Copyright 1985-2001 Microsoft Corp.
C:\work>initdb -E UTF-8 --no-locale
$B%G!<%?%Y!<%9%7%9%F%`Fb$N%U%!%$%k$N=jM-<T$O(B"HIROSHI"$B%f!<%6$G$7$?!#(B
$B$3$N%f!<%6$,%5!<%P%W%m%;%9$r=jM-$7$J$1$l$P$J$j$^$;$s!#(B
$B%G!<%?%Y!<%9%/%i%9%?$O%m%1!<%k(BC$B$G=i4|2=$5$l$^$9!#(B
$B%G%U%)%k%H$N%F%-%9%H8!:w@_Dj$O(Benglish$B$K@_Dj$5$l$^$7$?!#(B
$B%G%#%l%/%H%j(BC:/tmp/$BF|K\8l(B data$B$N8"8B$r@_Dj$7$F$$$^$9(B ... ok
$B%5%V%G%#%l%/%H%j$r:n@.$7$F$$$^$9(B ... ok
$B%G%U%)%k%H$N(Bmax_connections$B$rA*Br$7$F$$$^$9(B ... 100
$B%G%U%)%k%H$N(B shared_buffers $B$rA*Br$7$F$$$^$9(B ... 32MB
$B@_Dj%U%!%$%k$r:n@.$7$F$$$^$9(B ... ok
C:/tmp/$BF|K\8l(B data/base/1$B$K(Btemplate1$B%G!<%?%Y!<%9$r:n@.$7$F$$$^$9(B ... ok
pg_authid$B$r=i4|2=$7$F$$$^$9(B ... ok
$B0MB84X78$r=i4|2=$7$F$$$^$9(B ... ok
$B%7%9%F%`%S%e!<$r:n@.$7$F$$$^$9(B ... ok
$B%7%9%F%`%*%V%8%'%/%H$NDj5A$r%m!<%I$7$F$$$^$9(B ... ok
$BJQ49$r:n@.$7$F$$$^$9(B ... ok
$B%G%#%l%/%H%j$r:n@.$7$F$$$^$9(B ... ok
$BAH$_9~$_%*%V%8%'%/%H$K8"8B$r@_Dj$7$F$$$^$9(B ... ok
$B>pJs%9%-!<%^$r:n@.$7$F$$$^$9(B ... ok
template1$B%G!<%?%Y!<%9$r%P%-%e!<%`$7$F$$$^$9(B ... ok
template1$B$+$i(Btemplate0$B$X%3%T!<$7$F$$$^$9(B ... ok
template1$B$+$i(Bpostgres$B$X%3%T!<$7$F$$$^$9(B ... ok
$B7Y9p(B: $B%m!<%+%k@\B38~$1$K(B"trust"$BG'>Z$,M-8z$G$9!#(B
pg_hba.conf$B$rJT=8$9$k!"$b$7$/$O!"<!2s(Binitdb$B$r<B9T$9$k;~$K(B-A$B%*%W%7%g(B
$B%s$r;HMQ$9$k$3$H$GJQ99$9$k$3$H$,$G$-$^$9!#(B
$B@.8y$7$^$7$?!#0J2<$r;HMQ$7$F%G!<%?%Y!<%9%5!<%P$r5/F0$9$k$3$H$,$G$-$^$9!#(B
"postmaster" -D "C:/tmp/$BF|K\8l(B data"
$B$^$?$O(B
"pg_ctl" -D "C:/tmp/$BF|K\8l(B data" -l logfile start
C:\work>set PGCLIENTENCODING=SJIS
C:\work>psql postgres
psql (8.4beta1)
"help" $B$G%X%k%W$rI=<($7$^$9(B.
postgres=# create table $BF|K\8l(B($B$-!<(B text);
CREATE TABLE
postgres=# insert into $BF|K\8l(B values('$B$$$l$?(B');
INSERT 0 1
postgres=# copy $BF|K\8l(B to 'C:/tmp/$BF|K\8l(B data/$BF|K\8l(Butf8.txt';
COPY 1
postgres=# delete from $BF|K\8l(B;
DELETE 1
postgres=# copy $BF|K\8l(B from 'C:/tmp/$BF|K\8l(B data/$BF|K\8l(Butf8.txt';
COPY 1
postgres=# select * from $BF|K\8l(B;
$B$-!<(B
--------
$B$$$l$?(B
(1 $B9T(B)
C:\work>dir "C:\tmp\$BF|K\8l(B data"
$B%I%i%$%V(B C $B$N%\%j%e!<%`(B $B%i%Y%k$O(B SYS $B$G$9(B
$B%\%j%e!<%`(B $B%7%j%"%kHV9f$O(B 1433-2C7C $B$G$9(B
C:\tmp\$BF|K\8l(B data $B$N%G%#%l%/%H%j(B
2009/04/13 23:22 <DIR> .
2009/04/13 23:22 <DIR> ..
2009/04/13 23:18 <DIR> base
2009/04/13 23:19 <DIR> global
2009/04/13 23:17 <DIR> pg_clog
2009/04/13 23:17 3,616 pg_hba.conf
2009/04/13 23:17 1,611 pg_ident.conf
2009/04/13 23:17 <DIR> pg_multixact
2009/04/13 23:23 <DIR> pg_stat_tmp
2009/04/13 23:17 <DIR> pg_subtrans
2009/04/13 23:17 <DIR> pg_tblspc
2009/04/13 23:17 <DIR> pg_twophase
2009/04/13 23:17 4 PG_VERSION
2009/04/13 23:17 <DIR> pg_xlog
2009/04/13 23:17 17,112 postgresql.conf
2009/04/13 23:19 38 postmaster.opts
2009/04/13 23:19 24 postmaster.pid
2009/04/13 23:22 8 $BF|K\8l(Butf8.txt
7 $B8D$N%U%!%$%k(B 22,413 $B%P%$%H(B
11 $B8D$N%G%#%l%/%H%j(B 42,780,246,016 $B%P%$%H$N6u$-NN0h(B
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-14 00:51 Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
parent: Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 12+ messages in thread
From: Itagaki Takahiro @ 2009-04-14 00:51 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; pgsql-hackers
Tom Lane <tgl@sss.pgh.pa.us> wrote:
> Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:
> > Here is a patch to implement GetPlatformEncoding() and convert absolute
> > file paths from database encoding to platform encoding.
>
> This seems like a fairly significant overhead added to solve a really
> minor problem (if it's not minor why has it never come up before?).
It's not always a minor problem in Japan. It has been discussed in
users group in Japan several times. However, surely I should pay attention
to the performance. One of the solutions might be to cache the encoding
in GetPlatformEncoding(). There will be no overheads when database
encoding and platform encoding are same, that would be a typical use.
> It should not be necessary to repeat all
> this for every file access within the database directory.
That's why I added checking with is_absolute_path() there. We can
avoid conversion in normal file access under PGDATA because relative
paths are used for it. But I should have checked all of file access
not only in backends but also in client programs. I'll research them...
Regards,
---
ITAGAKI Takahiro
NTT Open Source Software Center
^ permalink raw reply [nested|flat] 12+ messages in thread
* Re: Solution of the file name problem of copy on windows.
@ 2009-04-14 20:41 Sergey Burladyan <eshkinkot@gmail.com>
parent: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
1 sibling, 0 replies; 12+ messages in thread
From: Sergey Burladyan @ 2009-04-14 20:41 UTC (permalink / raw)
To: Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>; +Cc: Hiroshi Saito <z-saito@guitar.ocn.ne.jp>; Tom Lane <tgl@sss.pgh.pa.us>; pgsql-hackers
Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp> writes:
> There are some issues:
> * Is it possible to determine the platform encoding?
There is no platform encoding in linux. File name encoding depend on user
locale, so different users can have different encoding of file name.
--
Sergey Burladyan
^ permalink raw reply [nested|flat] 12+ messages in thread
end of thread, other threads:[~2009-04-14 20:41 UTC | newest]
Thread overview: 12+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2009-04-07 16:40 Solution of the file name problem of copy on windows. Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
2009-04-07 17:46 ` Tom Lane <tgl@sss.pgh.pa.us>
2009-04-08 10:31 ` Hiroshi Inoue <inoue@tpf.co.jp>
2009-04-08 14:00 ` Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
2009-04-08 02:19 ` Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
2009-04-08 16:01 ` Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
2009-04-09 00:22 ` Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
2009-04-13 10:13 ` Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
2009-04-13 14:14 ` Tom Lane <tgl@sss.pgh.pa.us>
2009-04-13 15:12 ` Hiroshi Saito <z-saito@guitar.ocn.ne.jp>
2009-04-14 00:51 ` Itagaki Takahiro <itagaki.takahiro@oss.ntt.co.jp>
2009-04-14 20:41 ` Sergey Burladyan <eshkinkot@gmail.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox