public inbox for [email protected]  

help / color / mirror / Atom feed

comments tablecmds.c
29+ messages / 5 participants
[nested] [flat]

* comments tablecmds.c
@ 2016-12-25 12:38  Erik Rijkers <[email protected]>
  0 siblings, 1 reply; 29+ messages in thread

From: Erik Rijkers @ 2016-12-25 12:38 UTC (permalink / raw)
  To: pgsql-hackers

'the the' -> 'the'

and

'ie' -> 'i.e.'

Although (concening the latter change) the present counts are 'ie' 428, 
and 'i.e.' 428.
so it might be debatable (but let's not)

thanks,

Erik Rijkers



-- 
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Attachments:

  [text/x-diff] tablecmds.c.diff (566B, ../../[email protected]/2-tablecmds.c.diff)
  download | inline diff:
--- src/backend/commands/tablecmds.c.orig	2016-12-25 13:29:28.715237491 +0100
+++ src/backend/commands/tablecmds.c	2016-12-25 13:31:03.244444936 +0100
@@ -13244,7 +13244,7 @@
 			skip_validate = true;
 
 		/*
-		 * We choose to err on the safer side, ie, give up on skipping the the
+		 * We choose to err on the safer side, i.e., give up on skipping the
 		 * validation scan, if the partition key column doesn't have the NOT
 		 * NULL constraint and the table is to become a list partition that
 		 * does not accept nulls.  In this case, the partition predicate


^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* Re: comments tablecmds.c
@ 2016-12-25 13:35  Erik Rijkers <[email protected]>
  parent: Erik Rijkers <[email protected]>
  0 siblings, 1 reply; 29+ messages in thread

From: Erik Rijkers @ 2016-12-25 13:35 UTC (permalink / raw)
  To: pgsql-hackers; +Cc: [email protected]

On 2016-12-25 13:38, Erik Rijkers wrote:
> 'the the' -> 'the'
> 
> and
> 
> 'ie' -> 'i.e.'
> 
> Although (concening the latter change) the present counts are 'ie'
> 428, and 'i.e.' 428.
> so it might be debatable (but let's not)
> 

Sorry; I meant:  'ie' 428, and 'i.e.' 305.




-- 
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* Re: comments tablecmds.c
@ 2016-12-27 09:25  Magnus Hagander <[email protected]>
  parent: Erik Rijkers <[email protected]>
  0 siblings, 1 reply; 29+ messages in thread

From: Magnus Hagander @ 2016-12-27 09:25 UTC (permalink / raw)
  To: Erik Rijkers <[email protected]>; +Cc: pgsql-hackers; [email protected]

On Sun, Dec 25, 2016 at 2:35 PM, Erik Rijkers <[email protected]> wrote:

> On 2016-12-25 13:38, Erik Rijkers wrote:
>
>> 'the the' -> 'the'
>>
>> and
>>
>> 'ie' -> 'i.e.'
>>
>> Although (concening the latter change) the present counts are 'ie'
>> 428, and 'i.e.' 428.
>> so it might be debatable (but let's not)
>>
>>
> Sorry; I meant:  'ie' 428, and 'i.e.' 305.


Applied, thanks.

Seems to me that in comments it's not worth chasing them down individually,
but if you happen to fix one alongside another one like here, then it's a
different story :)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* Re: comments tablecmds.c
@ 2016-12-27 09:36  Erik Rijkers <[email protected]>
  parent: Magnus Hagander <[email protected]>
  0 siblings, 1 reply; 29+ messages in thread

From: Erik Rijkers @ 2016-12-27 09:36 UTC (permalink / raw)
  To: Magnus Hagander <[email protected]>; +Cc: pgsql-hackers; [email protected]

On 2016-12-27 10:25, Magnus Hagander wrote:
> On Sun, Dec 25, 2016 at 2:35 PM, Erik Rijkers <[email protected]> wrote:
> 
> Applied, thanks.
> 
> Seems to me that in comments it's not worth chasing them down 
> individually,
> but if you happen to fix one alongside another one like here, then it's 
> a
> different story :)
> 

Well, it had been triggering my (very simple) spell-checker at each 
compile, for a month a so.  I just wanted to silence it :)

Thanks for doing that :)

Erik Rijkers



-- 
Sent via pgsql-hackers mailing list ([email protected])
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* Re: comments tablecmds.c
@ 2016-12-27 09:41  Magnus Hagander <[email protected]>
  parent: Erik Rijkers <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Magnus Hagander @ 2016-12-27 09:41 UTC (permalink / raw)
  To: Erik Rijkers <[email protected]>; +Cc: pgsql-hackers; [email protected]

On Tue, Dec 27, 2016 at 10:36 AM, Erik Rijkers <[email protected]> wrote:

> On 2016-12-27 10:25, Magnus Hagander wrote:
>
>> On Sun, Dec 25, 2016 at 2:35 PM, Erik Rijkers <[email protected]> wrote:
>>
>> Applied, thanks.
>>
>> Seems to me that in comments it's not worth chasing them down
>> individually,
>> but if you happen to fix one alongside another one like here, then it's a
>> different story :)
>>
>>
> Well, it had been triggering my (very simple) spell-checker at each
> compile, for a month a so.  I just wanted to silence it :)
>
> Thanks for doing that :)


FWIW, that comment was in reference to the i.e. vs ie. The "the the"
problem was definitely something to fix :)


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature.
@ 2021-02-01 15:30  Heikki Linnakangas <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Heikki Linnakangas @ 2021-02-01 15:30 UTC (permalink / raw)

Per John Naylor's report, and some bugs I found while testing.

TODO: to be squashed.
---
 src/backend/utils/mb/conv.c                      | 16 ++++++----------
 .../utf8_and_iso8859_1/utf8_and_iso8859_1.c      |  8 ++++++++
 src/backend/utils/mb/mbutils.c                   |  5 +++--
 3 files changed, 17 insertions(+), 12 deletions(-)

diff --git a/src/backend/utils/mb/conv.c b/src/backend/utils/mb/conv.c
index b83358bc7a5..f3bda5753d0 100644
--- a/src/backend/utils/mb/conv.c
+++ b/src/backend/utils/mb/conv.c
@@ -497,7 +497,6 @@ UtfToLocal(const unsigned char *utf, int len,
 	int			l;
 	const pg_utf_to_local_combined *cp;
 	const unsigned char *start = utf;
-	const unsigned char *cur = utf;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -511,8 +510,6 @@ UtfToLocal(const unsigned char *utf, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*utf == '\0')
 			break;
@@ -648,10 +645,11 @@ UtfToLocal(const unsigned char *utf, int len,
 		}
 
 		/* failed to translate this character */
+		utf -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(PG_UTF8, encoding,
-								   (const char *) (utf - l), len);
+								   (const char *) utf, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -660,7 +658,7 @@ UtfToLocal(const unsigned char *utf, int len,
 
 	*iso = '\0';
 
-	return cur - start;
+	return utf - start;
 }
 
 /*
@@ -701,7 +699,6 @@ LocalToUtf(const unsigned char *iso, int len,
 	int			l;
 	const pg_local_to_utf_combined *cp;
 	const unsigned char *start = iso;
-	const unsigned char *cur = iso;
 
 	if (!PG_VALID_ENCODING(encoding))
 		ereport(ERROR,
@@ -715,8 +712,6 @@ LocalToUtf(const unsigned char *iso, int len,
 		unsigned char b3 = 0;
 		unsigned char b4 = 0;
 
-		cur = iso;
-
 		/* "break" cases all represent errors */
 		if (*iso == '\0')
 			break;
@@ -799,10 +794,11 @@ LocalToUtf(const unsigned char *iso, int len,
 		}
 
 		/* failed to translate this character */
+		iso -= l;
 		if (noError)
 			break;
 		report_untranslatable_char(encoding, PG_UTF8,
-								   (const char *) (iso - l), len);
+								   (const char *) iso, len);
 	}
 
 	/* if we broke out of loop early, must be invalid input */
@@ -811,5 +807,5 @@ LocalToUtf(const unsigned char *iso, int len,
 
 	*utf = '\0';
 
-	return cur - start;
+	return iso - start;
 }
diff --git a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
index 8ac93604a1b..d0dc4cca378 100644
--- a/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
+++ b/src/backend/utils/mb/conversion_procs/utf8_and_iso8859_1/utf8_and_iso8859_1.c
@@ -104,10 +104,18 @@ utf8_to_iso8859_1(PG_FUNCTION_ARGS)
 			int			l = pg_utf_mblen(src);
 
 			if (l > len || !pg_utf8_islegal(src, l))
+			{
+				if (noError)
+					break;
 				report_invalid_encoding(PG_UTF8, (const char *) src, len);
+			}
 			if (l != 2)
+			{
+				if (noError)
+					break;
 				report_untranslatable_char(PG_UTF8, PG_LATIN1,
 										   (const char *) src, len);
+			}
 			c1 = src[1] & 0x3f;
 			c = ((c & 0x1f) << 6) | c1;
 			if (c >= 0x80 && c <= 0xff)
diff --git a/src/backend/utils/mb/mbutils.c b/src/backend/utils/mb/mbutils.c
index 3e106027d75..af1cafe523a 100644
--- a/src/backend/utils/mb/mbutils.c
+++ b/src/backend/utils/mb/mbutils.c
@@ -818,12 +818,13 @@ perform_default_encoding_conversion(const char *src, int len,
 		MemoryContextAllocHuge(CurrentMemoryContext,
 							   (Size) len * MAX_CONVERSION_GROWTH + 1);
 
-	FunctionCall5(flinfo,
+	FunctionCall6(flinfo,
 				  Int32GetDatum(src_encoding),
 				  Int32GetDatum(dest_encoding),
 				  CStringGetDatum(src),
 				  CStringGetDatum(result),
-				  Int32GetDatum(len));
+				  Int32GetDatum(len),
+				  BoolGetDatum(false));
 
 	/*
 	 * Release extra space if there might be a lot --- see comments in
-- 
2.29.2


--------------39E46EF15EBFAA37D53332E5
Content-Type: text/x-patch; charset=UTF-8;
 name="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
 filename*0="v3-0005-Do-COPY-FROM-encoding-conversion-verification-in-.pa";
 filename*1="tch"



^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* Re: Remove extra Sort node above a btree-compatible Index Scan
@ 2025-02-28 05:30  Peter Geoghegan <[email protected]>
  0 siblings, 1 reply; 29+ messages in thread

From: Peter Geoghegan @ 2025-02-28 05:30 UTC (permalink / raw)
  To: Alexandra Wang <[email protected]>; +Cc: pgsql-hackers

On Fri, Feb 28, 2025 at 12:23 AM Alexandra Wang
<[email protected]> wrote:
> While reviewing Mark Dilger’s patch series "Index AM API cleanup" [1],
> I noticed some unexpected plan diffs when running the xtree and treeb
> test modules. Specifically, an additional Sort node appears on top of
> an Index Only Scan, even when the index key and sort key are the same.
> For example:
>
> --- src/test/modules/xtree/expected/interval.out
> +++ src/test/modules/xtree/results/interval.out

What's the xtree module? There's no such test module?

-- 
Peter Geoghegan






^ permalink  raw  reply  [nested|flat] 29+ messages in thread

* Re: Remove extra Sort node above a btree-compatible Index Scan
@ 2025-02-28 18:01  Alexandra Wang <[email protected]>
  parent: Peter Geoghegan <[email protected]>
  0 siblings, 0 replies; 29+ messages in thread

From: Alexandra Wang @ 2025-02-28 18:01 UTC (permalink / raw)
  To: Peter Geoghegan <[email protected]>; +Cc: pgsql-hackers

Hi Peter,

On Thu, Feb 27, 2025 at 11:31 PM Peter Geoghegan <[email protected]> wrote:

> On Fri, Feb 28, 2025 at 12:23 AM Alexandra Wang
> <[email protected]> wrote:
> > While reviewing Mark Dilger’s patch series "Index AM API cleanup" [1],
> > I noticed some unexpected plan diffs when running the xtree and treeb
> > test modules. Specifically, an additional Sort node appears on top of
> > an Index Only Scan, even when the index key and sort key are the same.
> > For example:
> >
> > --- src/test/modules/xtree/expected/interval.out
> > +++ src/test/modules/xtree/results/interval.out
>
> What's the xtree module? There's no such test module?
>

Thank you so much for asking!

I borrowed the following three patches from the Index AM API cleanup
thread [1] and attached them here for reference:

v20-0001: TEST - Add loadable modules as tests of the AM API
v20-0003: TEST - Stop requiring BTREE_AM_OID outside btree
v20-0012: Use CompareType more and StrategyNumber less

The v20-0001 patch introduces the xtree loadable test module. Xtree as
an extension is a shallow copy of the btree index, it should behave
exactly like btree. Specifically, it uses the same operators and
strategy numbers as btree, and all of its IndexAMRoutine methods point
to the corresponding btree methods. The only difference is that xtree
belongs to a different opfamily.

v20-0001 adds two loadable modules:
src/test/module/xtree (a shallow copy of btree)
src/test/module/xhash (a shallow copy of hash, not relevant to this
discussion)

These test modules are not meant for commit; they are used solely for
testing the Index AM API.

By running "make check" in src/test/module/xtree, a script included in
the patch copies all tests from src/test/regress and replaces all
btree indexes with xtree indexes, allowing verification of how the
Index AM API works with non-core indexes.

The other two patches serve the following purposes:

v20-0003 removes BTREE_AM_OID-specific assertions outside btree code,
allowing other Index AMs.

v20-0012 is nice to have since it also touches the PathKey struct and
adds a CompareType field in PathKey. CompareType is more generic for
sort ordering compared to AM-specific strategy numbers. I’d like to
include this patch because even if two AMs have different strategy
numbers, they could still map to the same CompareType. When it comes
to deciding whether two PathKeys represent the same ordering, what
really matters is the underlying sort operator, and CompareType is
very relevant in that regard.

So, with the above patches applied (v20-0012 is optional for this
demonstration), I can run the following queries:

-- setup
create extension xtree;
create table t (b int, x int);
create index on t using btree(b);
create index on t using xtree(x);
insert into t select i, i from generate_series(1, 1000)i;
analyze t;

-- queries
explain (costs off) select b from t where b < 10 order by (b);
explain (costs off) select x from t where x < 10 order by (x);

-- output:
test=# explain (costs off) select b from t where b < 10 order by (b);
             QUERY PLAN
------------------------------------
 Index Only Scan using t_b_idx on t
   Index Cond: (b < 10)
(2 rows)

test=# explain (costs off) select x from t where x < 10 order by (x);
                QUERY PLAN
------------------------------------------
 Sort
   Sort Key: x
   ->  Index Only Scan using t_x_idx on t
         Index Cond: (x < 10)
(4 rows)

Notice that the xtree index scan introduces an unnecessary Sort node,
even though the Index Only Scan already returns rows of x sorted by
the same operator (int4lt (”<”)) used in the ORDER BY (x) clause.

The problem I'm trying to solve is to eliminate this redundant sort
for btree-like index AMs.

[1]
https://www.postgresql.org/message-id/a5dfb7cd-7a89-48ab-a913-e5304eee0854%40eisentraut.org

Best,
Alex


Attachments:

  [application/octet-stream] v20-0003-TEST-Stop-requiring-BTREE_AM_OID-from-within-btr.patch.no-cfbot (2.7K, ../../CAK98qZ247C_xBww4J7wDs45x9z9Ljr8qvVePGzSHwORBMZg3sQ@mail.gmail.com/3-v20-0003-TEST-Stop-requiring-BTREE_AM_OID-from-within-btr.patch.no-cfbot)
  download

  [application/octet-stream] v20-0012-Use-CompareType-more-and-StrategyNumber-less.patch.no-cfbot (74.3K, ../../CAK98qZ247C_xBww4J7wDs45x9z9Ljr8qvVePGzSHwORBMZg3sQ@mail.gmail.com/4-v20-0012-Use-CompareType-more-and-StrategyNumber-less.patch.no-cfbot)
  download

  [application/octet-stream] v20-0001-TEST-Add-loadable-modules-as-tests-of-the-amapi.patch.no-cfbot (65.1K, ../../CAK98qZ247C_xBww4J7wDs45x9z9Ljr8qvVePGzSHwORBMZg3sQ@mail.gmail.com/5-v20-0001-TEST-Add-loadable-modules-as-tests-of-the-amapi.patch.no-cfbot)
  download

  [application/octet-stream] v1-0001-Remove-unnecessary-Sort-node-above-Index-Scan-of-.patch.no-cfbot (5.6K, ../../CAK98qZ247C_xBww4J7wDs45x9z9Ljr8qvVePGzSHwORBMZg3sQ@mail.gmail.com/6-v1-0001-Remove-unnecessary-Sort-node-above-Index-Scan-of-.patch.no-cfbot)
  download

^ permalink  raw  reply  [nested|flat] 29+ messages in thread

end of thread, other threads:[~2025-02-28 18:01 UTC | newest]

Thread overview: 29+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2016-12-25 12:38 comments tablecmds.c Erik Rijkers <[email protected]>
2016-12-25 13:35 ` Erik Rijkers <[email protected]>
2016-12-27 09:25   ` Magnus Hagander <[email protected]>
2016-12-27 09:36     ` Erik Rijkers <[email protected]>
2016-12-27 09:41       ` Magnus Hagander <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2021-02-01 15:30 [PATCH v3 4/5] Fix bugs in the commit to change conversion function signature. Heikki Linnakangas <[email protected]>
2025-02-28 05:30 Re: Remove extra Sort node above a btree-compatible Index Scan Peter Geoghegan <[email protected]>
2025-02-28 18:01 ` Re: Remove extra Sort node above a btree-compatible Index Scan Alexandra Wang <[email protected]>

This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox