pg.ddx.io pgsql-sql@postgresql.org mailing list archive
help / color / mirror / Atom feedScadinavian characters in regular expressions
7+ messages / 4 participants
[nested] [flat]
* Scadinavian characters in regular expressions
@ 2002-04-09 08:51 Søren Vainio <sva@Netpointers.com>
2002-04-09 09:53 ` Re: Scadinavian characters in regular expressions Andreas Joseph Krogh <andreak@officenet.no>
0 siblings, 1 reply; 7+ messages in thread
From: Søren Vainio @ 2002-04-09 08:51 UTC (permalink / raw)
To: pgsql-sql
Can someone please explain the following?
I am using a regular expression to find strings containing two words (begin
with one or more characters not being spaces followed by a space followed by
one or more characters not being spaces).
But when scandinavian characters are included it returns different results
depending on where the character is positioned.
The first two-word example returns TRUE as expected.
The second three-word example returns FALSE as expected.
But when I let an å (å å a-ring) traverse through the string it
unexpectedly returns TRUE when the character is positioned as the
second-last or last character in the two first words.
SELECT 'one two' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'åone two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'oåne two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'onåe two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one åtwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one tåwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one twåo three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one twoå three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one two åthree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two tåhree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two thåree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two thråee' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two threåe' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two threeå' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
Thank you for any response.
Søren Vainio, Denmark
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Scadinavian characters in regular expressions
2002-04-09 08:51 Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
@ 2002-04-09 09:53 ` Andreas Joseph Krogh <andreak@officenet.no>
0 siblings, 0 replies; 7+ messages in thread
From: Andreas Joseph Krogh @ 2002-04-09 09:53 UTC (permalink / raw)
To: pgsql-sql
On Tuesday 09 April 2002 10:51, Søren Vainio wrote:
> Can someone please explain the following?
> I am using a regular expression to find strings containing two words (begin
> with one or more characters not being spaces followed by a space followed
> by one or more characters not being spaces).
> But when scandinavian characters are included it returns different results
> depending on where the character is positioned.
> The first two-word example returns TRUE as expected.
> The second three-word example returns FALSE as expected.
> But when I let an å (å å a-ring) traverse through the string it
> unexpectedly returns TRUE when the character is positioned as the
> second-last or last character in the two first words.
>
> SELECT 'one two' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'åone two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'oåne two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'onåe two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one åtwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one tåwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one twåo three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one twoå three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one two åthree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two tåhree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two thåree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two thråee' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two threåe' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two threeå' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
>
> Thank you for any response.
>
> Søren Vainio, Denmark
I just tried the following which returned false as expected:
andreak=# SELECT 'oneå two three' ~ '^[^\s]+[\s][^\s]+$';
?column?
----------
f
(1 row)
andreak=# select version();
version
-----------------------------------------------------------
PostgreSQL 7.2 on i686-pc-linux-gnu, compiled by GCC 2.96
(1 row)
NOTE: I replaced your [^ ] with the properly formated pattarn for whitespace:
[^\s]
--
Andreas Joseph Krogh (Senior Software Developer) <andreak@officenet.no>
A hen is an egg's way of making another egg.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Scadinavian characters in regular expressions
@ 2002-04-09 11:28 Søren Vainio <sva@Netpointers.com>
2002-04-09 11:49 ` Re: Scadinavian characters in regular expressions Andreas Joseph Krogh <andreak@officenet.no>
2002-04-09 13:33 ` Re: Scadinavian characters in regular expressions Tom Lane <tgl@sss.pgh.pa.us>
0 siblings, 2 replies; 7+ messages in thread
From: Søren Vainio @ 2002-04-09 11:28 UTC (permalink / raw)
To: Andreas Joseph Krogh <andreak@officenet.no>; pgsql-sql
Using \s does produce FALSE for SELECT 'oneå two three' ~
'^[^\s]+[\s][^\s]+$';
But it also produces FALSE for any two-word string ex:
SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would expect TRUE???
(I am using PostgreSQL 7.1.3)
> -----Oprindelig meddelelse-----
> Fra: pgsql-sql-owner@postgresql.org
> [mailto:pgsql-sql-owner@postgresql.org]På vegne af Andreas
> Joseph Krogh
> Sendt: 9. april 2002 11:53
> Til: 'pgsql-sql@postgresql.org'
> Emne: Re: [SQL] Scadinavian characters in regular expressions
>
>
> On Tuesday 09 April 2002 10:51, Søren Vainio wrote:
> > Can someone please explain the following?
> > I am using a regular expression to find strings containing
> two words (begin
> > with one or more characters not being spaces followed by a
> space followed
> > by one or more characters not being spaces).
> > But when scandinavian characters are included it returns
> different results
> > depending on where the character is positioned.
> > The first two-word example returns TRUE as expected.
> > The second three-word example returns FALSE as expected.
> > But when I let an å (å å a-ring) traverse through
> the string it
> > unexpectedly returns TRUE when the character is positioned as the
> > second-last or last character in the two first words.
> >
> > SELECT 'one two' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'åone two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'oåne two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'onåe two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one åtwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one tåwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one twåo three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one twoå three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one two åthree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two tåhree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two thåree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two thråee' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two threåe' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two threeå' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> >
> > Thank you for any response.
> >
> > Søren Vainio, Denmark
>
> I just tried the following which returned false as expected:
> andreak=# SELECT 'oneå two three' ~ '^[^\s]+[\s][^\s]+$';
> ?column?
> ----------
> f
> (1 row)
>
> andreak=# select version();
> version
> -----------------------------------------------------------
> PostgreSQL 7.2 on i686-pc-linux-gnu, compiled by GCC 2.96
> (1 row)
>
> NOTE: I replaced your [^ ] with the properly formated pattarn
> for whitespace:
> [^\s]
>
> --
> Andreas Joseph Krogh (Senior Software Developer)
> <andreak@officenet.no>
> A hen is an egg's way of making another egg.
>
> ---------------------------(end of
> broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
>
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Scadinavian characters in regular expressions
2002-04-09 11:28 Re: Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
@ 2002-04-09 11:49 ` Andreas Joseph Krogh <andreak@officenet.no>
1 sibling, 0 replies; 7+ messages in thread
From: Andreas Joseph Krogh @ 2002-04-09 11:49 UTC (permalink / raw)
To: Søren Vainio <sva@Netpointers.com>; pgsql-sql
On Tuesday 09 April 2002 13:28, Søren Vainio wrote:
> Using \s does produce FALSE for SELECT 'oneå two three' ~
> '^[^\s]+[\s][^\s]+$';
> But it also produces FALSE for any two-word string ex:
> SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would expect TRUE???
> (I am using PostgreSQL 7.1.3)
Funny, in Perl it works:
perl -e '$p = "hei hopp"; print eval ($p =~ /^[^\s]+\s[^\s]+$/) ? "true\n" :
"false\n";'
prints:
true
--
Andreas Joseph Krogh (Senior Software Developer) <andreak@officenet.no>
A hen is an egg's way of making another egg.
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Scadinavian characters in regular expressions
2002-04-09 11:28 Re: Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
@ 2002-04-09 13:33 ` Tom Lane <tgl@sss.pgh.pa.us>
1 sibling, 0 replies; 7+ messages in thread
From: Tom Lane @ 2002-04-09 13:33 UTC (permalink / raw)
To: Søren Vainio <sva@Netpointers.com>; +Cc: 'Andreas Joseph Krogh' <andreak@officenet.no>; pgsql-sql
=?iso-8859-1?Q?S=F8ren_Vainio?= <sva@Netpointers.com> writes:
> Using \s does produce FALSE for SELECT 'oneå two three' ~
> '^[^\s]+[\s][^\s]+$';
> But it also produces FALSE for any two-word string ex:
> SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would expect TRUE???
> (I am using PostgreSQL 7.1.3)
I do not believe that Postgres' regular expression engine recognizes \s
as meaning anything except "s". See
http://www.ca.postgresql.org/users-lounge/docs/7.2/postgres/functions-matching.html
In the above, it's even worse: the backslashes were eaten by the
string-literal parser, so what arrived at the RE engine was just
^[^s]+[s][^s]+$ ... not likely to produce what you wanted.
As for the original issue, I wonder whether you are storing the string
as UTF-8 or Latin1 encoding. I have a suspicion that the å (å
å a-ring) is actually a multibyte sequence inside the database
and for some reason Postgres isn't configured to recognize it as a
single logical character.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Scadinavian characters in regular expressions
@ 2002-04-09 14:21 Søren Vainio <sva@Netpointers.com>
2002-04-10 01:06 ` Re: Scadinavian characters in regular expressions Tatsuo Ishii <t-ishii@sra.co.jp>
0 siblings, 1 reply; 7+ messages in thread
From: Søren Vainio @ 2002-04-09 14:21 UTC (permalink / raw)
To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-sql
There is obviously a problem with the scecial characters.
The query SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$';
produced FALSE on a database with ENCODING = 'LATIN1' and TRUE on a database
with ENCODING = 'UNICODE'.
Do you have a suggestion to how I can find the count of two-word strings
with ENCODING = 'UNICODE'?
Thank you
Søren Vainio
> -----Oprindelig meddelelse-----
> Fra: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sendt: 9. april 2002 15:34
> Til: Søren Vainio
> Cc: 'Andreas Joseph Krogh'; 'pgsql-sql@postgresql.org'
> Emne: Re: [SQL] Scadinavian characters in regular expressions
>
>
> =?iso-8859-1?Q?S=F8ren_Vainio?= <sva@Netpointers.com> writes:
> > Using \s does produce FALSE for SELECT 'oneå two three' ~
> > '^[^\s]+[\s][^\s]+$';
> > But it also produces FALSE for any two-word string ex:
> > SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would
> expect TRUE???
> > (I am using PostgreSQL 7.1.3)
>
> I do not believe that Postgres' regular expression engine
> recognizes \s
> as meaning anything except "s". See
> http://www.ca.postgresql.org/users-lounge/docs/7.2/postgres/fu
nctions-matching.html
In the above, it's even worse: the backslashes were eaten by the
string-literal parser, so what arrived at the RE engine was just
^[^s]+[s][^s]+$ ... not likely to produce what you wanted.
As for the original issue, I wonder whether you are storing the string
as UTF-8 or Latin1 encoding. I have a suspicion that the å (å
å a-ring) is actually a multibyte sequence inside the database
and for some reason Postgres isn't configured to recognize it as a
single logical character.
regards, tom lane
^ permalink raw reply [nested|flat] 7+ messages in thread
* Re: Scadinavian characters in regular expressions
2002-04-09 14:21 Re: Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
@ 2002-04-10 01:06 ` Tatsuo Ishii <t-ishii@sra.co.jp>
0 siblings, 0 replies; 7+ messages in thread
From: Tatsuo Ishii @ 2002-04-10 01:06 UTC (permalink / raw)
To: sva@netpointers.com; +Cc: tgl@sss.pgh.pa.us; pgsql-sql
> There is obviously a problem with the scecial characters.
> The query SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$';
> produced FALSE on a database with ENCODING = 'LATIN1' and TRUE on a database
> with ENCODING = 'UNICODE'.
>
> Do you have a suggestion to how I can find the count of two-word strings
> with ENCODING = 'UNICODE'?
I see no problem here.
SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$';
?column?
----------
f
(1 row)
SELECT 'oneå three' ~ '^[^ ]+[ ][^ ]+$';
?column?
----------
t
(1 row)
(The database encoding is UNICODE)
This is PostgreSQL 7.1.3/7.2.1 running on a Linux box. I guess you
have some problems with your installation or platform.
--
Tatsuo Ishii
^ permalink raw reply [nested|flat] 7+ messages in thread
end of thread, other threads:[~2002-04-10 01:06 UTC | newest]
Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2002-04-09 08:51 Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
2002-04-09 09:53 ` Andreas Joseph Krogh <andreak@officenet.no>
2002-04-09 11:28 Re: Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
2002-04-09 11:49 ` Andreas Joseph Krogh <andreak@officenet.no>
2002-04-09 13:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2002-04-09 14:21 Re: Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
2002-04-10 01:06 ` Tatsuo Ishii <t-ishii@sra.co.jp>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox