pg.ddx.io  pgsql-sql@postgresql.org mailing list archive  
help / color / mirror / Atom feed
Scadinavian characters in regular expressions
7+ messages / 4 participants
[nested] [flat]

* Scadinavian characters in regular expressions
@ 2002-04-09 08:51  Søren Vainio <sva@Netpointers.com>
  0 siblings, 1 reply; 7+ messages in thread

From: Søren Vainio @ 2002-04-09 08:51 UTC (permalink / raw)
  To: pgsql-sql

Can someone please explain the following?
I am using a regular expression to find strings containing two words (begin
with one or more characters not being spaces followed by a space followed by
one or more characters not being spaces).
But when scandinavian characters are included it returns different results
depending on where the character is positioned.
The first two-word example returns TRUE as expected.
The second three-word example returns FALSE as expected.
But when I let an å (&#229 &aring a-ring) traverse through the string it
unexpectedly returns TRUE when the character is positioned as the
second-last or last character in the two first words.
 
SELECT 'one two' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'åone two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'oåne two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'onåe two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one åtwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one tåwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one twåo three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one twoå three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
SELECT 'one two åthree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two tåhree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two thåree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two thråee' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two threåe' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
SELECT 'one two threeå' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
 
Thank you for any response.
 
Søren Vainio, Denmark

^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Scadinavian characters in regular expressions
@ 2002-04-09 09:53  Andreas Joseph Krogh <andreak@officenet.no>
  parent: Søren Vainio <sva@Netpointers.com>
  0 siblings, 0 replies; 7+ messages in thread

From: Andreas Joseph Krogh @ 2002-04-09 09:53 UTC (permalink / raw)
  To: pgsql-sql

On Tuesday 09 April 2002 10:51, Søren Vainio wrote:
> Can someone please explain the following?
> I am using a regular expression to find strings containing two words (begin
> with one or more characters not being spaces followed by a space followed
> by one or more characters not being spaces).
> But when scandinavian characters are included it returns different results
> depending on where the character is positioned.
> The first two-word example returns TRUE as expected.
> The second three-word example returns FALSE as expected.
> But when I let an å (&#229 &aring a-ring) traverse through the string it
> unexpectedly returns TRUE when the character is positioned as the
> second-last or last character in the two first words.
>
> SELECT 'one two' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'åone two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'oåne two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'onåe two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one åtwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one tåwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one twåo three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one twoå three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> SELECT 'one two åthree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two tåhree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two thåree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two thråee' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two threåe' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> SELECT 'one two threeå' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
>
> Thank you for any response.
>
> Søren Vainio, Denmark

I just tried the following which returned false as expected:
andreak=# SELECT 'oneå two three' ~ '^[^\s]+[\s][^\s]+$';
 ?column?
----------
 f
(1 row)

andreak=# select version();
                          version
-----------------------------------------------------------
 PostgreSQL 7.2 on i686-pc-linux-gnu, compiled by GCC 2.96
(1 row)

NOTE: I replaced your [^ ] with the properly formated pattarn for whitespace: 
[^\s]

-- 
Andreas Joseph Krogh (Senior Software Developer) <andreak@officenet.no>
A hen is an egg's way of making another egg.



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Scadinavian characters in regular expressions
@ 2002-04-09 11:28  Søren Vainio <sva@Netpointers.com>
  0 siblings, 2 replies; 7+ messages in thread

From: Søren Vainio @ 2002-04-09 11:28 UTC (permalink / raw)
  To: Andreas Joseph Krogh <andreak@officenet.no>; pgsql-sql

Using \s does produce FALSE for SELECT 'oneå two three' ~
'^[^\s]+[\s][^\s]+$';
But it also produces FALSE for any two-word string ex:
SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would expect TRUE???
(I am using PostgreSQL 7.1.3)

> -----Oprindelig meddelelse-----
> Fra: pgsql-sql-owner@postgresql.org
> [mailto:pgsql-sql-owner@postgresql.org]På vegne af Andreas 
> Joseph Krogh
> Sendt: 9. april 2002 11:53
> Til: 'pgsql-sql@postgresql.org'
> Emne: Re: [SQL] Scadinavian characters in regular expressions
> 
> 
> On Tuesday 09 April 2002 10:51, Søren Vainio wrote:
> > Can someone please explain the following?
> > I am using a regular expression to find strings containing 
> two words (begin
> > with one or more characters not being spaces followed by a 
> space followed
> > by one or more characters not being spaces).
> > But when scandinavian characters are included it returns 
> different results
> > depending on where the character is positioned.
> > The first two-word example returns TRUE as expected.
> > The second three-word example returns FALSE as expected.
> > But when I let an å (&#229 &aring a-ring) traverse through 
> the string it
> > unexpectedly returns TRUE when the character is positioned as the
> > second-last or last character in the two first words.
> >
> > SELECT 'one two' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'åone two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'oåne two three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'onåe two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one åtwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one tåwo three' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one twåo three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one twoå three' ~ '^[^ ]+[ ][^ ]+$'; returns TRUE
> > SELECT 'one two åthree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two tåhree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two thåree' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two thråee' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two threåe' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> > SELECT 'one two threeå' ~ '^[^ ]+[ ][^ ]+$'; returns FALSE
> >
> > Thank you for any response.
> >
> > Søren Vainio, Denmark
> 
> I just tried the following which returned false as expected:
> andreak=# SELECT 'oneå two three' ~ '^[^\s]+[\s][^\s]+$';
>  ?column?
> ----------
>  f
> (1 row)
> 
> andreak=# select version();
>                           version
> -----------------------------------------------------------
>  PostgreSQL 7.2 on i686-pc-linux-gnu, compiled by GCC 2.96
> (1 row)
> 
> NOTE: I replaced your [^ ] with the properly formated pattarn 
> for whitespace: 
> [^\s]
> 
> -- 
> Andreas Joseph Krogh (Senior Software Developer) 
> <andreak@officenet.no>
> A hen is an egg's way of making another egg.
> 
> ---------------------------(end of 
> broadcast)---------------------------
> TIP 3: if posting/reading through Usenet, please send an appropriate
> subscribe-nomail command to majordomo@postgresql.org so that your
> message can get through to the mailing list cleanly
> 



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Scadinavian characters in regular expressions
@ 2002-04-09 11:49  Andreas Joseph Krogh <andreak@officenet.no>
  parent: Søren Vainio <sva@Netpointers.com>
  1 sibling, 0 replies; 7+ messages in thread

From: Andreas Joseph Krogh @ 2002-04-09 11:49 UTC (permalink / raw)
  To: Søren Vainio <sva@Netpointers.com>; pgsql-sql

On Tuesday 09 April 2002 13:28, Søren Vainio wrote:
> Using \s does produce FALSE for SELECT 'oneå two three' ~
> '^[^\s]+[\s][^\s]+$';
> But it also produces FALSE for any two-word string ex:
> SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would expect TRUE???
> (I am using PostgreSQL 7.1.3)

Funny, in Perl it works:

perl -e '$p = "hei hopp"; print eval ($p =~ /^[^\s]+\s[^\s]+$/) ? "true\n" : 
"false\n";'

prints:
true


-- 
Andreas Joseph Krogh (Senior Software Developer) <andreak@officenet.no>
A hen is an egg's way of making another egg.



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Scadinavian characters in regular expressions
@ 2002-04-09 13:33  Tom Lane <tgl@sss.pgh.pa.us>
  parent: Søren Vainio <sva@Netpointers.com>
  1 sibling, 0 replies; 7+ messages in thread

From: Tom Lane @ 2002-04-09 13:33 UTC (permalink / raw)
  To: Søren Vainio <sva@Netpointers.com>; +Cc: 'Andreas Joseph Krogh' <andreak@officenet.no>; pgsql-sql

=?iso-8859-1?Q?S=F8ren_Vainio?= <sva@Netpointers.com> writes:
> Using \s does produce FALSE for SELECT 'oneå two three' ~
> '^[^\s]+[\s][^\s]+$';
> But it also produces FALSE for any two-word string ex:
> SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would expect TRUE???
> (I am using PostgreSQL 7.1.3)

I do not believe that Postgres' regular expression engine recognizes \s
as meaning anything except "s".  See
http://www.ca.postgresql.org/users-lounge/docs/7.2/postgres/functions-matching.html

In the above, it's even worse: the backslashes were eaten by the
string-literal parser, so what arrived at the RE engine was just
^[^s]+[s][^s]+$ ... not likely to produce what you wanted.

As for the original issue, I wonder whether you are storing the string
as UTF-8 or Latin1 encoding.  I have a suspicion that the å (&#229
&aring a-ring) is actually a multibyte sequence inside the database
and for some reason Postgres isn't configured to recognize it as a
single logical character.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Scadinavian characters in regular expressions
@ 2002-04-09 14:21  Søren Vainio <sva@Netpointers.com>
  0 siblings, 1 reply; 7+ messages in thread

From: Søren Vainio @ 2002-04-09 14:21 UTC (permalink / raw)
  To: Tom Lane <tgl@sss.pgh.pa.us>; +Cc: pgsql-sql

There is obviously a problem with the scecial characters.
The query SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$';
produced FALSE on a database with ENCODING = 'LATIN1' and TRUE on a database
with ENCODING = 'UNICODE'.

Do you have a suggestion to how I can find the count of two-word strings
with ENCODING = 'UNICODE'?

Thank you
Søren Vainio

> -----Oprindelig meddelelse-----
> Fra: Tom Lane [mailto:tgl@sss.pgh.pa.us]
> Sendt: 9. april 2002 15:34
> Til: Søren Vainio
> Cc: 'Andreas Joseph Krogh'; 'pgsql-sql@postgresql.org'
> Emne: Re: [SQL] Scadinavian characters in regular expressions 
> 
> 
> =?iso-8859-1?Q?S=F8ren_Vainio?= <sva@Netpointers.com> writes:
> > Using \s does produce FALSE for SELECT 'oneå two three' ~
> > '^[^\s]+[\s][^\s]+$';
> > But it also produces FALSE for any two-word string ex:
> > SELECT 'one two' ~ '^[^\s]+[\s][^\s]+$'; where I would 
> expect TRUE???
> > (I am using PostgreSQL 7.1.3)
> 
> I do not believe that Postgres' regular expression engine 
> recognizes \s
> as meaning anything except "s".  See
> http://www.ca.postgresql.org/users-lounge/docs/7.2/postgres/fu
nctions-matching.html

In the above, it's even worse: the backslashes were eaten by the
string-literal parser, so what arrived at the RE engine was just
^[^s]+[s][^s]+$ ... not likely to produce what you wanted.

As for the original issue, I wonder whether you are storing the string
as UTF-8 or Latin1 encoding.  I have a suspicion that the å (&#229
&aring a-ring) is actually a multibyte sequence inside the database
and for some reason Postgres isn't configured to recognize it as a
single logical character.

			regards, tom lane



^ permalink  raw  reply  [nested|flat] 7+ messages in thread

* Re: Scadinavian characters in regular expressions
@ 2002-04-10 01:06  Tatsuo Ishii <t-ishii@sra.co.jp>
  parent: Søren Vainio <sva@Netpointers.com>
  0 siblings, 0 replies; 7+ messages in thread

From: Tatsuo Ishii @ 2002-04-10 01:06 UTC (permalink / raw)
  To: sva@netpointers.com; +Cc: tgl@sss.pgh.pa.us; pgsql-sql

> There is obviously a problem with the scecial characters.
> The query SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$';
> produced FALSE on a database with ENCODING = 'LATIN1' and TRUE on a database
> with ENCODING = 'UNICODE'.
> 
> Do you have a suggestion to how I can find the count of two-word strings
> with ENCODING = 'UNICODE'?

I see no problem here.

SELECT 'oneå two three' ~ '^[^ ]+[ ][^ ]+$';
 ?column? 
----------
 f
(1 row)

SELECT 'oneå three' ~ '^[^ ]+[ ][^ ]+$';
 ?column? 
----------
 t
(1 row)

(The database encoding is UNICODE)

This is PostgreSQL 7.1.3/7.2.1 running on a Linux box. I guess you
have some problems with your installation or platform.
--
Tatsuo Ishii



^ permalink  raw  reply  [nested|flat] 7+ messages in thread


end of thread, other threads:[~2002-04-10 01:06 UTC | newest]

Thread overview: 7+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2002-04-09 08:51 Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
2002-04-09 09:53 ` Andreas Joseph Krogh <andreak@officenet.no>
2002-04-09 11:28 Re: Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
2002-04-09 11:49 ` Andreas Joseph Krogh <andreak@officenet.no>
2002-04-09 13:33 ` Tom Lane <tgl@sss.pgh.pa.us>
2002-04-09 14:21 Re: Scadinavian characters in regular expressions Søren Vainio <sva@Netpointers.com>
2002-04-10 01:06 ` Tatsuo Ishii <t-ishii@sra.co.jp>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox