public inbox for [email protected]
help / color / mirror / Atom feed[PATCH 07/10] cirrus/ccache: use G rather than GB suffix..
69+ messages / 15 participants
[nested] [flat]
* [PATCH 07/10] cirrus/ccache: use G rather than GB suffix..
@ 2022-11-13 20:03 Justin Pryzby <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Justin Pryzby @ 2022-11-13 20:03 UTC (permalink / raw)
the former being the documented spelling.
---
.cirrus.yml | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.cirrus.yml b/.cirrus.yml
index f2129787529..10ec7ac406a 100644
--- a/.cirrus.yml
+++ b/.cirrus.yml
@@ -665,7 +665,7 @@ task:
# Use larger ccache cache, as this task compiles with multiple compilers /
# flag combinations
- CCACHE_MAXSIZE: "1GB"
+ CCACHE_MAXSIZE: "1G"
CCACHE_DIR: "/tmp/ccache_dir"
LINUX_CONFIGURE_FEATURES: *LINUX_CONFIGURE_FEATURES
--
2.25.1
--kjpMrWxdCilgNbo1
Content-Type: text/x-diff; charset=us-ascii
Content-Disposition: attachment;
filename="0008-comments-grammar-extended-and-other-stats.patch"
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-05 10:57 Sergey Prokhorenko <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-05 10:57 UTC (permalink / raw)
To: Jelte Fennema-Nio <[email protected]>; Przemysław Sztoch <[email protected]>; Andrey M. Borodin <[email protected]>; +Cc: Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; [email protected] <[email protected]>
Hello Przemysław and Andrey,
When naming functions, I would advise using the shorter abbreviation uuidv7 from the new version of the RFC instead of uuid_v7. When people search Google for new versions of UUIDs, they enter the abbreviation uuidv7 into the search bar. The name generate_uuidv7() looks good, as well as uuidv1_to_uuidv6() and timestamp_to_uuidv7().
Best regards,
Sergey [email protected]
On Friday, 5 January 2024 at 11:53:04 am GMT+3, Przemysław Sztoch <[email protected]> wrote:
Andrey M. Borodin wrote on 1/4/2024 7:20 PM:
Hello Przemysław,
thanks for your interest in this patch!
On 3 Jan 2024, at 04:37, Przemysław Sztoch <[email protected]> wrote:
1. Is it possible to add a function that returns the version of the generated uuid?
It will be very useful.
I don't know if it's possible, but I think there are bits in the UUID that inform about the version.
What do you think if we have functions get_uuid_v7_ver(uuid) and get_uuid_v7_var(uuid) to extract bit fields according to [0] ? Or, perhaps, this should be one function with two return parameters?
It's not in a patch yet, I'm just considering how this functionality should look like.
uuid_ver(uuid) -> smallint/integer 1/3/4/5/6/7/8
Of course there is RFC 4122 Variant "bits: 10x". If it is other variant then uuid_ver should return -1 OR NULL.
For UUIDs generated by your patch this function should always return 7.
2. If there is any doubt about adding the function to the main sources (standard development in progress), in my opinion you can definitely add this function to the uuid-ossp extension.
From my POV we can just have this function in the core. OSSP support for UUID seems more or less dead [1]: "Newsflash: 04-Jul-2008: Released OSSP uuid 1.6.2". Or do I look into wrong place?
After two days of thinking about UUID v7, I consider it a very important functionality that should be included in PG17.
3. Wouldn't it be worth including UUID version 6 as well?
The standard in [0] says "Systems that do not involve legacy UUIDv1 SHOULD use UUIDv7 Section 5.7 instead." If there's a point in developing v6 - I'm OK to do so.
IETF standard should provide information about possibility of conversion from v1 to v6.
Then the usefulness of v6 is much greater and it would be worth implementing this version as well.
4. Sometimes you will need to generate a uuid for historical time. There should be an additional function gen_uuid_v7(timestamp).
Done, please see patch attached. But I changed signature to gen_uuid_v7(int8), to avoid messing with bytes from user who knows what they want. Or do you think gen_uuid_v7(timestamp) would be more convenient?
I talked to my colleagues and everyone chooses the timestamp version.
If timestamp is outside the allowed range, the function must return an error.
We also talked about uuid-ossp. Still, v5 is a great solution in some applications.
It is worth moving this function from extension to PG17. Many people don't use it because they don't know it and this uuid schema.
We think it would be quite reasonable to add:
uuid_generate_v5 (namespace uuid, name text) -> uuid
uuid_generate_v6 () -> uuid
uuid_generate_v6 (timestamptz) -> uuid
uuid_generate_v7() -> uuid
uuid_generate_v7(timestamptz) -> uuid
uuid_ver(uuid) -> smallint -1/1/2/3/4/5/6/7/8
uuid_ts(uuid) -> timestamptz (for 1/6/7 version, for other should return NULL, error is too heavy in our opinion)
uuid_v1_to_v6 (uuid) -> uuid
The naming of this family of functions needs to be rethought.
Do we adopt the naming standard from Postgres and the uuid-ossp extension?
Or should we continue with a slightly less accurate name for PG: get_random_uuid (get_random_uuid, get_uuid_v7)?
5. Please add in docs reference to RFC4122 (https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-14#uuid)
People should read standards. :-)
Thanks!
Best regards, Andrey Borodin.
[0] https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-14#uuidv7
[1] http://www.ossp.org/
--
Przemysław Sztoch | Mobile +48 509 99 00 66
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-16 12:15 Andrey Borodin <[email protected]>
parent: Sergey Prokhorenko <[email protected]>
0 siblings, 2 replies; 69+ messages in thread
From: Andrey Borodin @ 2024-01-16 12:15 UTC (permalink / raw)
To: Sergey Prokhorenko <[email protected]>; +Cc: Jelte Fennema-Nio <[email protected]>; Przemysław Sztoch <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 5 Jan 2024, at 15:57, Sergey Prokhorenko <[email protected]> wrote:
Sergey, Przemysław, Jelte, thanks for your feedback.
Here's v9. Changes:
1. Swapped type of the argument to timestamptz in gen_uuid_v7()
2. Renamed get_uuid_v7_time() to uuid_v7_time()
3. Added uuid_ver() and uuid_var().
What do you think?
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v9-0001-Implement-UUID-v7-as-per-IETF-draft.patch (12.6K, ../../[email protected]/2-v9-0001-Implement-UUID-v7-as-per-IETF-draft.patch)
download | inline diff:
From 25460815ccbfcd6f86d39beddc6c0aa7005fc0a4 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v9] Implement UUID v7 as per IETF draft
This commit addes function to generate UUID v7.
This function optionally accepts datetime used to generate
next UUID.
Also we add a function to extract timestamp from UUID v7.
Authors: Andrey Borodin, Sergey Prokhorenko
---
doc/src/sgml/func.sgml | 18 ++-
src/backend/utils/adt/pseudotypes.c | 12 +-
src/backend/utils/adt/uuid.c | 157 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 14 ++
src/test/regress/expected/opr_sanity.out | 4 +
src/test/regress/expected/uuid.out | 43 +++++++
src/test/regress/sql/uuid.sql | 22 ++++
7 files changed, 267 insertions(+), 3 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 210c7c0b02..d6c83cb13f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14130,13 +14130,29 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>gen_uuid_v7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_v7_time</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes two functions to generate a UUID:
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
</synopsis>
This function returns a version 4 (random) UUID. This is the most commonly
used type of UUID and is appropriate for most applications.
+<synopsis>
+<function>gen_uuid_v7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 (time-ordered + random) UUID.
+<synopsis>
+<function>uuid_v7_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID version 7.
</para>
<para>
diff --git a/src/backend/utils/adt/pseudotypes.c b/src/backend/utils/adt/pseudotypes.c
index a3a991f634..82c2d1309d 100644
--- a/src/backend/utils/adt/pseudotypes.c
+++ b/src/backend/utils/adt/pseudotypes.c
@@ -23,6 +23,7 @@
#include "postgres.h"
#include "libpq/pqformat.h"
+#include "miscadmin.h"
#include "utils/array.h"
#include "utils/builtins.h"
#include "utils/rangetypes.h"
@@ -332,11 +333,18 @@ shell_out(PG_FUNCTION_ARGS)
*
* We must disallow input of pg_node_tree values because the SQL functions
* that operate on the type are not secure against malformed input.
- * We do want to allow output, though.
+ * We do want to allow output, though. Also we need input during bootstrap.
*/
-PSEUDOTYPE_DUMMY_INPUT_FUNC(pg_node_tree);
PSEUDOTYPE_DUMMY_RECEIVE_FUNC(pg_node_tree);
+Datum
+pg_node_tree_in(PG_FUNCTION_ARGS)
+{
+ if (!IsBootstrapProcessingMode())
+ elog(ERROR, "cannot accept a value of type pg_node_tree_in");
+ return textin(fcinfo);
+}
+
Datum
pg_node_tree_out(PG_FUNCTION_ARGS)
{
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c7..66d9672dd1 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,6 +13,9 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
@@ -20,6 +23,7 @@
#include "utils/builtins.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +425,156 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+
+
+Datum
+gen_uuid_v7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ if (PG_NARGS() == 0 || PG_ARGISNULL(0))
+ {
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = tms <= previous_timestamp;
+ }
+ else
+ {
+ ts = PG_GETARG_TIMESTAMPTZ(0);
+ tms = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC) / 1000;
+ /*
+ * The time can leap backwards when provided by the user, so we use
+ * counter only when called with exactly same unix_ts_ms argument.
+ */
+ increment_counter = (tms == previous_timestamp);
+ }
+
+ if (increment_counter)
+ {
+ /* Time did not increment from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * http://tools.ietf.org/html/rfc ???
+ * https://datatracker.ietf.org/doc/html/draft-peabody-dispatch-new-uuid-format#name-creating-a-uuidv7-value
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_v7_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if (((uuid->data[6] & 0xf0) != 0x70)
+ || ((uuid->data[8] & 0xc0) != 0x80))
+ elog(ERROR,"uuid_v7_time() can only extract timestamp from UUID v7");
+
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+}
+
+Datum
+uuid_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ elog(ERROR,"uuid_ver() is only defined for RFC 4122 variants");
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58811a6530..c00cd1320f 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,20 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate UUID version 7', proisstrict => 'f',
+ proname => 'gen_uuid_v7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => 'timestamptz', prosrc => 'gen_uuid_v7',
+ proargnames => '{unix_ts_ms}', pronargdefaults => 1, proargmodes => '{i}',
+ proargdefaults => '({CONST :consttype 1184 :consttypmod -1 :constcollid 0 :constlen 8 :constbyval true :constisnull true :location 46 :constvalue <>})' },
+{ oid => '9896', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_v7_time', proleakproof => 't', provolatile => 'i',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_v7_time' },
+{ oid => '9897', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_ver', proleakproof => 't', provolatile => 'i',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_ver' },
+{ oid => '9898', descr => 'extract variant from UUID',
+ proname => 'uuid_var', proleakproof => 't', provolatile => 'i',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d6..163658f002 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -872,6 +872,10 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+gen_uuid_v7(timestamp with time zone)
+uuid_v7_time(uuid)
+uuid_ver(uuid)
+uuid_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d..b8426b1f8e 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,48 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7());
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7(now()));
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- check that timestamp is extracted correctly
+SELECT uuid_v7_time(gen_uuid_v7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+ ?column?
+----------
+ @ 0
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_ver(gen_uuid_v7());
+ uuid_ver
+----------
+ 7
+(1 row)
+
+SELECT uuid_var(gen_uuid_v7());
+ uuid_var
+----------
+ 2
+(1 row)
+
+-- uuid_v7_time() must refuse to accept non-UUIDv7
+select uuid_v7_time(gen_random_uuid());
+ERROR: uuid_v7_time() can only extract timestamp from UUID v7
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d..fb28766ece 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,27 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7());
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7(now()));
+INSERT INTO guid1 (guid_field) VALUES (gen_uuid_v7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- check that timestamp is extracted correctly
+SELECT uuid_v7_time(gen_uuid_v7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+
+-- support functions for UUID versions and variants
+SELECT uuid_ver(gen_uuid_v7());
+SELECT uuid_var(gen_uuid_v7());
+
+-- uuid_v7_time() must refuse to accept non-UUIDv7
+select uuid_v7_time(gen_random_uuid());
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.37.1 (Apple Git-137.1)
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-16 13:00 Aleksander Alekseev <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 2 replies; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-16 13:00 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey Borodin <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Przemysław Sztoch <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi Andrey,
> Sergey, Przemysław, Jelte, thanks for your feedback.
> Here's v9. Changes:
> 1. Swapped type of the argument to timestamptz in gen_uuid_v7()
> 2. Renamed get_uuid_v7_time() to uuid_v7_time()
> 3. Added uuid_ver() and uuid_var().
>
> What do you think?
Many thanks for the updated patch. It's an important work and I very
much hope we will see this in the upcoming PG release.
```
+Datum
+pg_node_tree_in(PG_FUNCTION_ARGS)
+{
+ if (!IsBootstrapProcessingMode())
+ elog(ERROR, "cannot accept a value of type pg_node_tree_in");
+ return textin(fcinfo);
+}
```
Not 100% sure what this is for. Any chance this could be part of another patch?
One thing I don't particularly like about the tests is the fact that
they don't check if a correct UUID was actually generated. I realize
that's not quite trivial due to the random nature of the function, but
maybe we could use some substring/regex magic here? Something like:
```
select gen_uuid_v7() :: text ~ '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$';
?column?
----------
t
select regexp_replace(gen_uuid_v7('2024-01-16 15:45:33 MSK') :: text,
'[0-9a-f]{4}-[0-9a-f]{12}$', 'XXXX-' || repeat('X', 12));
regexp_replace
--------------------------------------
018d124e-39c8-74c7-XXXX-XXXXXXXXXXXX
```
```
+ proname => 'uuid_v7_time', proleakproof => 't', provolatile => 'i',
```
I don't think we conventionally specify IMMUTABLE volatility, it's the
default. Other values also are worth checking.
Another question: how did you choose between using TimestampTz and
Timestamp types? I realize that internally it's all the same. Maybe
Timestamp will be slightly better since the way it is displayed
doesn't depend on the session settings. Many people I talked to find
this part of TimestampTz confusing.
Also I would like to point out that part of the documentation is
missing, but I guess at this stage of the game it's OK.
Last but not least: maybe we should support casting Timestamp[Tz] to
UUIDv7 and vice versa? Shouldn't be difficult to implement and I
suspect somebody will request this eventually. During the cast to UUID
we will always get the same value for the given Timestamp[Tz], which
probably can be useful in certain applications. It can't be done with
gen_uuid_v7() and its volatility doesn't permit it.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-16 14:44 Andrey M. Borodin <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 2 replies; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-16 14:44 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Przemysław Sztoch <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Thanks for your review, Aleksander!
> On 16 Jan 2024, at 18:00, Aleksander Alekseev <[email protected]> wrote:
>
>
> ```
> +Datum
> +pg_node_tree_in(PG_FUNCTION_ARGS)
> +{
> + if (!IsBootstrapProcessingMode())
> + elog(ERROR, "cannot accept a value of type pg_node_tree_in");
> + return textin(fcinfo);
> +}
> ```
>
> Not 100% sure what this is for. Any chance this could be part of another patch?
Nope, it’s necessary there. Without these changes catalog functions cannot have defaults for arguments. These defaults have type pg_node_tree which has no-op in function.
> One thing I don't particularly like about the tests is the fact that
> they don't check if a correct UUID was actually generated. I realize
> that's not quite trivial due to the random nature of the function, but
> maybe we could use some substring/regex magic here? Something like:
>
> ```
> select gen_uuid_v7() :: text ~ '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$';
> ?column?
> ----------
> t
>
> select regexp_replace(gen_uuid_v7('2024-01-16 15:45:33 MSK') :: text,
> '[0-9a-f]{4}-[0-9a-f]{12}$', 'XXXX-' || repeat('X', 12));
> regexp_replace
> --------------------------------------
> 018d124e-39c8-74c7-XXXX-XXXXXXXXXXXX
> ```
Any 8 bytes which have ver and var bits (6 bits total) are correct UUID.
This is checked by tests when uuid_var() and uuid_ver() functions are exercised.
> ```
> + proname => 'uuid_v7_time', proleakproof => 't', provolatile => 'i',
> ```
>
> I don't think we conventionally specify IMMUTABLE volatility, it's the
> default. Other values also are worth checking.
Makes sense, I’ll drop this values in next version.
BTW I’m in doubt if provided functions are leakproof. They ERROR-out with messages that can give a clue about several bits of UUID. Does this break leakproofness? I think yest, but I’m not sure.
gen_uuid_v7() seems leakproof to me.
> Another question: how did you choose between using TimestampTz and
> Timestamp types? I realize that internally it's all the same. Maybe
> Timestamp will be slightly better since the way it is displayed
> doesn't depend on the session settings. Many people I talked to find
> this part of TimestampTz confusing.
I mean, this argument is expected to be used to implement K-way sorted identifiers. In this context, it seems to me, it’s good to remember to developer that time shift also depend on timezones.
But this is too vague.
Do you have any reasons that apply to UUID generation?
> Also I would like to point out that part of the documentation is
> missing, but I guess at this stage of the game it's OK.
>
> Last but not least: maybe we should support casting Timestamp[Tz] to
> UUIDv7 and vice versa? Shouldn't be difficult to implement and I
> suspect somebody will request this eventually. During the cast to UUID
> we will always get the same value for the given Timestamp[Tz], which
> probably can be useful in certain applications. It can't be done with
> gen_uuid_v7() and its volatility doesn't permit it.
I’m strongly opposed to doing this cast. I was not adding this function to extract timestamp from UUID, because standard does not recommend it. But a lot of people asked for this.
But supporting easy way to do unrecommended thing seem bad.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-16 16:49 Sergey Prokhorenko <[email protected]>
parent: Andrey M. Borodin <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-16 16:49 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; Andrey M. Borodin <[email protected]>; +Cc: pgsql-hackers; Jelte Fennema-Nio <[email protected]>; Przemysław Sztoch <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Andrey,
It is not clear how to interpret uuid_v7_time():
- uuid_v7 to time() (extracting the timestamp)
- time() to uuid_v7 (generation of the uuid_v7)
It is worth improving the naming, for example, adding prepositions.
Sergey [email protected]
On Tuesday, 16 January 2024 at 05:44:51 pm GMT+3, Andrey M. Borodin <[email protected]> wrote:
Thanks for your review, Aleksander!
> On 16 Jan 2024, at 18:00, Aleksander Alekseev <[email protected]> wrote:
>
>
> ```
> +Datum
> +pg_node_tree_in(PG_FUNCTION_ARGS)
> +{
> + if (!IsBootstrapProcessingMode())
> + elog(ERROR, "cannot accept a value of type pg_node_tree_in");
> + return textin(fcinfo);
> +}
> ```
>
> Not 100% sure what this is for. Any chance this could be part of another patch?
Nope, it’s necessary there. Without these changes catalog functions cannot have defaults for arguments. These defaults have type pg_node_tree which has no-op in function.
> One thing I don't particularly like about the tests is the fact that
> they don't check if a correct UUID was actually generated. I realize
> that's not quite trivial due to the random nature of the function, but
> maybe we could use some substring/regex magic here? Something like:
>
> ```
> select gen_uuid_v7() :: text ~ '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$';
> ?column?
> ----------
> t
>
> select regexp_replace(gen_uuid_v7('2024-01-16 15:45:33 MSK') :: text,
> '[0-9a-f]{4}-[0-9a-f]{12}$', 'XXXX-' || repeat('X', 12));
> regexp_replace
> --------------------------------------
> 018d124e-39c8-74c7-XXXX-XXXXXXXXXXXX
> ```
Any 8 bytes which have ver and var bits (6 bits total) are correct UUID.
This is checked by tests when uuid_var() and uuid_ver() functions are exercised.
> ```
> + proname => 'uuid_v7_time', proleakproof => 't', provolatile => 'i',
> ```
>
> I don't think we conventionally specify IMMUTABLE volatility, it's the
> default. Other values also are worth checking.
Makes sense, I’ll drop this values in next version.
BTW I’m in doubt if provided functions are leakproof. They ERROR-out with messages that can give a clue about several bits of UUID. Does this break leakproofness? I think yest, but I’m not sure.
gen_uuid_v7() seems leakproof to me.
> Another question: how did you choose between using TimestampTz and
> Timestamp types? I realize that internally it's all the same. Maybe
> Timestamp will be slightly better since the way it is displayed
> doesn't depend on the session settings. Many people I talked to find
> this part of TimestampTz confusing.
I mean, this argument is expected to be used to implement K-way sorted identifiers. In this context, it seems to me, it’s good to remember to developer that time shift also depend on timezones.
But this is too vague.
Do you have any reasons that apply to UUID generation?
> Also I would like to point out that part of the documentation is
> missing, but I guess at this stage of the game it's OK.
>
> Last but not least: maybe we should support casting Timestamp[Tz] to
> UUIDv7 and vice versa? Shouldn't be difficult to implement and I
> suspect somebody will request this eventually. During the cast to UUID
> we will always get the same value for the given Timestamp[Tz], which
> probably can be useful in certain applications. It can't be done with
> gen_uuid_v7() and its volatility doesn't permit it.
I’m strongly opposed to doing this cast. I was not adding this function to extract timestamp from UUID, because standard does not recommend it. But a lot of people asked for this.
But supporting easy way to do unrecommended thing seem bad.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-16 20:10 Jelte Fennema-Nio <[email protected]>
parent: Andrey M. Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Jelte Fennema-Nio @ 2024-01-16 20:10 UTC (permalink / raw)
To: Andrey M. Borodin <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Sergey Prokhorenko <[email protected]>; Przemysław Sztoch <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Tue, 16 Jan 2024 at 15:44, Andrey M. Borodin <[email protected]> wrote:
> > On 16 Jan 2024, at 18:00, Aleksander Alekseev <[email protected]> wrote:
> > Not 100% sure what this is for. Any chance this could be part of another patch?
> Nope, it’s necessary there. Without these changes catalog functions cannot have defaults for arguments. These defaults have type pg_node_tree which has no-op in function.
That seems like the wrong way to make that work then. How about
instead we define the same function name twice, once with and once
without a timestamp argument. That's how this is done for other
functions that are overloaded in pg_catalog.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-16 20:20 Przemysław Sztoch <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Przemysław Sztoch @ 2024-01-16 20:20 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; pgsql-hackers; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Andrey Borodin wrote on 1/16/2024 1:15 PM:
> Sergey, Przemysław, Jelte, thanks for your feedback.
> Here's v9. Changes:
> 1. Swapped type of the argument to timestamptz in gen_uuid_v7()
Please update docs part about optional timestamp argument.
> 2. Renamed get_uuid_v7_time() to uuid_v7_time()
Pleaserename uuid_v7_time to uuid_time() and add support for v1 and v6.
If version is incompatible then return NULL.
> 3. Added uuid_ver() and uuid_var().
Looks good.
But for me, throwing an error is problematic. Wouldn't it be better to
return -1.
> What do you think?
> Best regards, Andrey Borodin.
--
Przemysław Sztoch | Mobile +48 509 99 00 66
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-16 21:09 Przemysław Sztoch <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Przemysław Sztoch @ 2024-01-16 21:09 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Andrey Borodin <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> Another question: how did you choose between using TimestampTz and
> Timestamp types? I realize that internally it's all the same. Maybe
> Timestamp will be slightly better since the way it is displayed
> doesn't depend on the session settings. Many people I talked to find
> this part of TimestampTz confusing.
timstamptz internally always store UTC.
I believe that in SQL, when operating with time in UTC, you should
always use timestamptz.
timestamp is theoretically the same thing. But internally it does not
convert time to UTC and will lead to incorrect use.
--
Przemysław Sztoch | Mobile +48 509 99 00 66
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 13:17 Andrey Borodin <[email protected]>
parent: Sergey Prokhorenko <[email protected]>
0 siblings, 2 replies; 69+ messages in thread
From: Andrey Borodin @ 2024-01-18 13:17 UTC (permalink / raw)
To: Jelte Fennema-Nio <[email protected]>; +Cc: Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 17 Jan 2024, at 02:19, Jelte Fennema-Nio <[email protected]> wrote:
I want to ask Kyzer or Brad, I hope they will see this message. I'm working on the patch for time extraction for v1, v6 and v7.
Do I understand correctly, that UUIDs contain local time, not UTC time? For examples in [0] I see that "A.6. Example of a UUIDv7 Value" I see that February 22, 2022 2:22:22.00 PM GMT-05:00 results in unix_ts_ms = 0x017F22E279B0, which is not UTC, but local time.
Is it intentional? Section "5.1. UUID Version 1" states otherwise.
If so, I should swap signatures of functions from TimestampTz to Timestamp.
I'm hard-coding examples from this standard to tests, so I want to be precise...
If I follow the standard I see this in tests:
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') at time zone 'GMT-05';
+ timezone
+--------------------------
+ Wed Feb 23 00:22:22 2022
+(1 row)
Current patch version attached. I've addressed all other requests: function renames, aliases, multiple functions instead of optional params, cleaner catalog definitions, not throwing error when [var,ver,time] value is unknown.
What is left: deal with timezones, improve documentation.
Best regards, Andrey Borodin.
[0] https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis#name-example-of-a-uuidv1-value
Attachments:
[application/octet-stream] v10-0001-Implement-UUID-v7-as-per-IETF-draft.patch (15.1K, ../../[email protected]/2-v10-0001-Implement-UUID-v7-as-per-IETF-draft.patch)
download | inline diff:
From f5e527d49a792b9b140562c7d42e0b1b15f5b315 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v10] Implement UUID v7 as per IETF draft
This commit addes function to generate UUID v7.
This function optionally accepts datetime used to generate
next UUID.
Also we add a function to extract timestamp from UUID v7.
Authors: Andrey Borodin, Sergey Prokhorenko
---
doc/src/sgml/func.sgml | 18 ++-
src/backend/utils/adt/uuid.c | 192 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 19 +++
src/test/regress/expected/opr_sanity.out | 13 +-
src/test/regress/expected/uuid.out | 88 +++++++++++
src/test/regress/sql/uuid.sql | 35 +++++
6 files changed, 361 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 210c7c0b02..417e0c7f19 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14130,13 +14130,29 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes two functions to generate a UUID:
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
</synopsis>
This function returns a version 4 (random) UUID. This is the most commonly
used type of UUID and is appropriate for most applications.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 (time-ordered + random) UUID.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID version 7.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c7..3538fcba6d 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,190 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ if (PG_NARGS() == 0 || PG_ARGISNULL(0))
+ {
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = tms <= previous_timestamp;
+ }
+ else
+ {
+ ts = PG_GETARG_TIMESTAMPTZ(0);
+ tms = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC) / 1000;
+ /*
+ * The time can leap backwards when provided by the user, so we use
+ * counter only when called with exactly same unix_ts_ms argument.
+ */
+ increment_counter = (tms == previous_timestamp);
+ }
+
+ if (increment_counter)
+ {
+ /* Time did not increment from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * http://tools.ietf.org/html/rfc ???
+ * https://datatracker.ietf.org/doc/html/draft-peabody-dispatch-new-uuid-format#name-creating-a-uuidv7-value
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - date2j(1582,10,15)) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - date2j(1582,10,15)) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58811a6530..956fb08ce9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,25 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'generate UUID version 7', proisstrict => 'f',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => 'timestamptz', prosrc => 'uuidv7',
+ proargnames => '{unix_ts_ms}', proargmodes => '{i}' },
+{ oid => '9898', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9899', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9900', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d6..1c37533975 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -126,9 +126,10 @@ WHERE p1.oid < p2.oid AND
p1.proretset != p2.proretset OR
p1.provolatile != p2.provolatile OR
p1.pronargs != p2.pronargs);
- oid | proname | oid | proname
------+---------+-----+---------
-(0 rows)
+ oid | proname | oid | proname
+------+---------+------+---------
+ 9896 | uuidv7 | 9897 | uuidv7
+(1 row)
-- Look for uses of different type OIDs in the argument/result type fields
-- for different aliases of the same built-in function.
@@ -872,6 +873,12 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuidv7(timestamp with time zone)
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d..b61f7a64ff 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,93 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+ ?column?
+----------
+ @ 0
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') at time zone 'GMT-05';
+ timezone
+--------------------------
+ Wed Feb 23 00:22:22 2022
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') at time zone 'GMT-05';
+ timezone
+--------------------------
+ Wed Feb 23 00:22:22 2022
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') at time zone 'GMT-05';
+ timezone
+--------------------------
+ Wed Feb 23 00:22:22 2022
+(1 row)
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d..d7185759b1 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,40 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') at time zone 'GMT-05';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') at time zone 'GMT-05';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') at time zone 'GMT-05';
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.37.1 (Apple Git-137.1)
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 14:20 Aleksander Alekseev <[email protected]>
parent: Przemysław Sztoch <[email protected]>
0 siblings, 2 replies; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-18 14:20 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Przemysław Sztoch <[email protected]>; Andrey Borodin <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> Another question: how did you choose between using TimestampTz and
> Timestamp types? I realize that internally it's all the same. Maybe
> Timestamp will be slightly better since the way it is displayed
> doesn't depend on the session settings. Many people I talked to find
> this part of TimestampTz confusing.
>
> timstamptz internally always store UTC.
> I believe that in SQL, when operating with time in UTC, you should always use timestamptz.
> timestamp is theoretically the same thing. But internally it does not convert time to UTC and will lead to incorrect use.
No.
Timestamp and TimestampTz are absolutely the same thing. The only
difference is how they are shown to the user. TimestampTz uses session
context in order to be displayed in the TZ chosen by the user. Thus
typically it is somewhat more confusing to the users and thus I asked
whether there was a good reason to choose TimestampTz over Timestamp.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 15:39 Andrey Borodin <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 3 replies; 69+ messages in thread
From: Andrey Borodin @ 2024-01-18 15:39 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 18 Jan 2024, at 19:20, Aleksander Alekseev <[email protected]> wrote:
>
> Timestamp and TimestampTz are absolutely the same thing.
My question is not about Postgres data types. I'm asking about examples in the standard.
There's an example 017F22E2-79B0-7CC3-98C4-DC0C0C07398F. It is expected to be generated on "Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00".
It's exaplained to be 164555774200000ns after 1582-10-15 00:00:00 UTC.
But 164555774200000ns after 1582-10-15 00:00:00 UTC was 2022-02-22 19:22:22 UTC. And that was 2022-02-23 00:22:22 in UTC-05.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 16:21 Aleksander Alekseev <[email protected]>
parent: Andrey Borodin <[email protected]>
2 siblings, 1 reply; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-18 16:21 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi Andrey,
> > Timestamp and TimestampTz are absolutely the same thing.
> My question is not about Postgres data types. I'm asking about examples in the standard.
>
> There's an example 017F22E2-79B0-7CC3-98C4-DC0C0C07398F. It is expected to be generated on "Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00".
> It's exaplained to be 164555774200000ns after 1582-10-15 00:00:00 UTC.
>
> But 164555774200000ns after 1582-10-15 00:00:00 UTC was 2022-02-22 19:22:22 UTC. And that was 2022-02-23 00:22:22 in UTC-05.
Not 100% sure which text you are referring to exactly, but I'm
guessing it's section B.2 of [1]
"""
This example UUIDv7 test vector utilizes a well-known 32 bit Unix
epoch with additional millisecond precision to fill the first 48 bits
[...]
The timestamp is Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00
represented as 0x17F22E279B0 or 1645557742000
"""
If this is the case, I think the example is indeed wrong:
```
=# select extract(epoch from 'Tuesday, February 22, 2022 2:22:22.00 PM
GMT-05:00' :: timestamptz)*1000;
?column?
----------------------
1645521742000.000000
(1 row)
```
And the difference between the value in the text and the actual value
is 10 hours as you pointed out.
Also you named the date 1582-10-15 00:00:00 UTC. Maybe you actually
meant 1970-01-01 00:00:00 UTC?
[1]: https://www.ietf.org/archive/id/draft-peabody-dispatch-new-uuid-format-04.html
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 18:28 Sergey Prokhorenko <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-18 18:28 UTC (permalink / raw)
To: pgsql-hackers; Aleksander Alekseev <[email protected]>; +Cc: Andrey Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi Andrey,
Aleksander Alekseev wrote: "If this is the case, I think the example is indeed wrong".
This is one of the reasons why I was categorically against any examples of implementation in the new RFC. The examples have been very poorly studied and discussed, and therefore it is better not to use them at all. But the text of the RFC itself clearly refers to UTC, and not at all about local time: "UUID version 7 features a time-ordered value field derived from the widely implemented and well known Unix Epoch timestamp source, the number of milliseconds since midnight 1 Jan 1970 UTC, leap seconds excluded". The main reason for using UTC is so that UUIDv7's, generated approximately simultaneously in different time zones, are correctly ordered in time when they get into one database.
Sergey Prokhorenko
[email protected]
On Thursday, 18 January 2024 at 07:22:05 pm GMT+3, Aleksander Alekseev <[email protected]> wrote:
Hi Andrey,
> > Timestamp and TimestampTz are absolutely the same thing.
> My question is not about Postgres data types. I'm asking about examples in the standard.
>
> There's an example 017F22E2-79B0-7CC3-98C4-DC0C0C07398F. It is expected to be generated on "Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00".
> It's exaplained to be 164555774200000ns after 1582-10-15 00:00:00 UTC.
>
> But 164555774200000ns after 1582-10-15 00:00:00 UTC was 2022-02-22 19:22:22 UTC. And that was 2022-02-23 00:22:22 in UTC-05.
Not 100% sure which text you are referring to exactly, but I'm
guessing it's section B.2 of [1]
"""
This example UUIDv7 test vector utilizes a well-known 32 bit Unix
epoch with additional millisecond precision to fill the first 48 bits
[...]
The timestamp is Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00
represented as 0x17F22E279B0 or 1645557742000
"""
If this is the case, I think the example is indeed wrong:
```
=# select extract(epoch from 'Tuesday, February 22, 2022 2:22:22.00 PM
GMT-05:00' :: timestamptz)*1000;
?column?
----------------------
1645521742000.000000
(1 row)
```
And the difference between the value in the text and the actual value
is 10 hours as you pointed out.
Also you named the date 1582-10-15 00:00:00 UTC. Maybe you actually
meant 1970-01-01 00:00:00 UTC?
[1]: https://www.ietf.org/archive/id/draft-peabody-dispatch-new-uuid-format-04.html
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 18:31 Andrey Borodin <[email protected]>
parent: Andrey Borodin <[email protected]>
2 siblings, 2 replies; 69+ messages in thread
From: Andrey Borodin @ 2024-01-18 18:31 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 18 Jan 2024, at 20:39, Andrey Borodin <[email protected]> wrote:
>
> But 164555774200000ns after 1582-10-15 00:00:00 UTC was 2022-02-22 19:22:22 UTC. And that was 2022-02-23 00:22:22 in UTC-05.
'2022-02-22 19:22:22 UTC' is exactly that moment which was encoded into example UUIDs. It's not '2022-02-23 00:22:22 in UTC-05' as I thought.
I got confused by "at timezone" changes which in fact removes timezone information. And that's per SQL standard...
Now I'm completely lost in time... I've set local time to NY (UTC-5).
postgres=# select TIMESTAMP WITH TIME ZONE '2022-02-22 14:22:22-05' - TIMESTAMP WITH TIME ZONE 'Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00';
?column?
----------
10:00:00
(1 row)
postgres=# select TIMESTAMP WITH TIME ZONE 'Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00';
timestamptz
------------------------
2022-02-22 04:22:22-05
(1 row)
I cannot wrap my mind around it... Any pointers would be appreciated.
I'm certain that code extracted UTC time correctly, I just want a reliable test that verifies timestamp constant (+ I understand what is going on).
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 18:59 Sergey Prokhorenko <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-18 18:59 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; Andrey Borodin <[email protected]>; +Cc: pgsql-hackers; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi Andrey,
You'd better generate a test UUIDv7 for midnight 1 Jan 1970 UTC. In this case, the timestamp in UUIDv7 according to the new RFC must be filled with zeros. By extracting the timestamp from this test UUIDv7, you should get exactly midnight 1 Jan 1970 UTC.
I also recommend this article: https://habr.com/ru/articles/772954/
Sergey Prokhorenko
[email protected]
On Thursday, 18 January 2024 at 09:31:16 pm GMT+3, Andrey Borodin <[email protected]> wrote:
> On 18 Jan 2024, at 20:39, Andrey Borodin <[email protected]> wrote:
>
> But 164555774200000ns after 1582-10-15 00:00:00 UTC was 2022-02-22 19:22:22 UTC. And that was 2022-02-23 00:22:22 in UTC-05.
'2022-02-22 19:22:22 UTC' is exactly that moment which was encoded into example UUIDs. It's not '2022-02-23 00:22:22 in UTC-05' as I thought.
I got confused by "at timezone" changes which in fact removes timezone information. And that's per SQL standard...
Now I'm completely lost in time... I've set local time to NY (UTC-5).
postgres=# select TIMESTAMP WITH TIME ZONE '2022-02-22 14:22:22-05' - TIMESTAMP WITH TIME ZONE 'Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00';
?column?
----------
10:00:00
(1 row)
postgres=# select TIMESTAMP WITH TIME ZONE 'Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00';
timestamptz
------------------------
2022-02-22 04:22:22-05
(1 row)
I cannot wrap my mind around it... Any pointers would be appreciated.
I'm certain that code extracted UTC time correctly, I just want a reliable test that verifies timestamp constant (+ I understand what is going on).
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 20:26 Przemysław Sztoch <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Przemysław Sztoch @ 2024-01-18 20:26 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Jelte Fennema-Nio <[email protected]>; Sergey Prokhorenko <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Using localtime would be absurd. Especially since time goes back during
summer time change.
I believe our implementation should use UTC. No one forbids us from
assuming that our local time for generating uuid is UTC.
Andrey Borodin wrote on 1/18/2024 2:17 PM:
>
>> On 17 Jan 2024, at 02:19, Jelte Fennema-Nio <[email protected]> wrote:
> I want to ask Kyzer or Brad, I hope they will see this message. I'm working on the patch for time extraction for v1, v6 and v7.
>
> Do I understand correctly, that UUIDs contain local time, not UTC time? For examples in [0] I see that "A.6. Example of a UUIDv7 Value" I see that February 22, 2022 2:22:22.00 PM GMT-05:00 results in unix_ts_ms = 0x017F22E279B0, which is not UTC, but local time.
> Is it intentional? Section "5.1. UUID Version 1" states otherwise.
>
> If so, I should swap signatures of functions from TimestampTz to Timestamp.
> I'm hard-coding examples from this standard to tests, so I want to be precise...
>
> If I follow the standard I see this in tests:
> +-- extract UUID v1, v6 and v7 timestamp
> +SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') at time zone 'GMT-05';
> + timezone
> +--------------------------
> + Wed Feb 23 00:22:22 2022
> +(1 row)
>
> Current patch version attached. I've addressed all other requests: function renames, aliases, multiple functions instead of optional params, cleaner catalog definitions, not throwing error when [var,ver,time] value is unknown.
> What is left: deal with timezones, improve documentation.
>
>
> Best regards, Andrey Borodin.
>
> [0] https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis#name-example-of-a-uuidv1-value
--
Przemysław Sztoch | Mobile +48 509 99 00 66
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 20:39 Przemysław Sztoch <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Przemysław Sztoch @ 2024-01-18 20:39 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Andrey Borodin <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Aleksander Alekseev wrote on 1/18/2024 3:20 PM:
> Hi,
>
>> Another question: how did you choose between using TimestampTz and
>> Timestamp types? I realize that internally it's all the same. Maybe
>> Timestamp will be slightly better since the way it is displayed
>> doesn't depend on the session settings. Many people I talked to find
>> this part of TimestampTz confusing.
>>
>> timstamptz internally always store UTC.
>> I believe that in SQL, when operating with time in UTC, you should always use timestamptz.
>> timestamp is theoretically the same thing. But internally it does not convert time to UTC and will lead to incorrect use.
> No.
>
> Timestamp and TimestampTz are absolutely the same thing. The only
> difference is how they are shown to the user. TimestampTz uses session
> context in order to be displayed in the TZ chosen by the user. Thus
> typically it is somewhat more confusing to the users and thus I asked
> whether there was a good reason to choose TimestampTz over Timestamp.
>
Theoretically, you're right. But look at this example:
SET timezone TO 'Europe/Warsaw';
SELECT extract(epoch from '2024-01-18 9:27:30'::timestamp),
extract(epoch from '2024-01-18 9:27:30'::timestamptz);
date_part | date_part
------------+------------
1705570050 | 1705566450
(1 row)
In my opinion, timestamptz gives greater guarantees that the time
internally is in UTC and the user gets the time in his/her time zone.
In the case of timestamp, it is never certain whether it keeps time in
UTC or in the local zone.
In the case of argument's type, there would be no problem because we
could create two functions.
Of course timestamp would be treated the same as timestamptz.
But here we have a problem with the function return type, which can only
be one. And since the time returned is in UTC, it should be timestamptz.
--
Przemysław Sztoch | Mobile +48 509 99 00 66
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-18 20:49 Przemysław Sztoch <[email protected]>
parent: Andrey Borodin <[email protected]>
2 siblings, 0 replies; 69+ messages in thread
From: Przemysław Sztoch @ 2024-01-18 20:49 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
We are not allowed to consider any time other than UTC.
You need to write to the authors of the standard. I suppose this is a
mistake.
I know from experience that errors in such standards most often appear
in examples.
Nobody detects them at first.
Everyone reads and checks ideas, not calculations.
Then developers during implementation tears out their hair.
Andrey Borodin wrote on 1/18/2024 4:39 PM:
>
>> On 18 Jan 2024, at 19:20, Aleksander Alekseev <[email protected]> wrote:
>>
>> Timestamp and TimestampTz are absolutely the same thing.
> My question is not about Postgres data types. I'm asking about examples in the standard.
>
> There's an example 017F22E2-79B0-7CC3-98C4-DC0C0C07398F. It is expected to be generated on "Tuesday, February 22, 2022 2:22:22.00 PM GMT-05:00".
> It's exaplained to be 164555774200000ns after 1582-10-15 00:00:00 UTC.
>
> But 164555774200000ns after 1582-10-15 00:00:00 UTC was 2022-02-22 19:22:22 UTC. And that was 2022-02-23 00:22:22 in UTC-05.
>
>
> Best regards, Andrey Borodin.
--
Przemysław Sztoch | Mobile +48 509 99 00 66
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-19 02:58 Lukas Fittl <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Lukas Fittl @ 2024-01-19 02:58 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Jelte Fennema-Nio <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Thu, Jan 18, 2024 at 5:18 AM Andrey Borodin <[email protected]> wrote:
> Current patch version attached. I've addressed all other requests:
> function renames, aliases, multiple functions instead of optional params,
> cleaner catalog definitions, not throwing error when [var,ver,time] value
> is unknown.
> What is left: deal with timezones, improve documentation.
>
I've done a test of the v10 patch, and ran into an interesting behavior
when passing in a timestamp to the function (which, as a side note, is
actually very useful to have as a feature, to support creating time-based
range partitions on UUIDv7 fields):
postgres=# SELECT uuid_extract_time(uuidv7());
uuid_extract_time
---------------------------
2024-01-18 18:49:00.01-08
(1 row)
postgres=# SELECT uuid_extract_time(uuidv7('2024-04-01'));
uuid_extract_time
------------------------
2024-04-01 00:00:00-07
(1 row)
postgres=# SELECT uuid_extract_time(uuidv7());
uuid_extract_time
------------------------
2024-04-01 00:00:00-07
(1 row)
Note how calling the uuidv7 function again after having called it with a
fixed future timestamp, returns the future timestamp, even though it should
return the current time.
I believe this is caused by incorrectly re-using the cached
previous_timestamp. In the second call here (with a fixed future
timestamp), we end up setting ts and tms to 2024-04-01, with
increment_counter = false, which leads us to set previous_timestamp to the
passed in timestamp (else branch of the second if in uuidv7). When we then
call the function again without an argument, we end up getting a new
timestamp from gettimeofday, but because we try to detect backwards leaps,
we set increment_counter to true, and thus end up reusing the previous
(future) timestamp here:
/* protection from leap backward */
tms = previous_timestamp;
Not sure how to fix this, but clearly something is amiss here.
Thanks,
Lukas
--
Lukas Fittl
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-19 03:24 David G. Johnston <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: David G. Johnston @ 2024-01-19 03:24 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Thu, Jan 18, 2024 at 11:31 AM Andrey Borodin <[email protected]>
wrote:
>
> Now I'm completely lost in time... I've set local time to NY (UTC-5).
>
> postgres=# select TIMESTAMP WITH TIME ZONE '2022-02-22 14:22:22-05' -
> TIMESTAMP WITH TIME ZONE 'Tuesday, February 22, 2022 2:22:22.00 PM
> GMT-05:00';
> ?column?
> ----------
> 10:00:00
> (1 row)
>
>
You are mixing POSIX and ISO-8601 conventions and, as noted in our
appendix, they disagree on the direction that is positive.
https://www.postgresql.org/docs/current/datetime-posix-timezone-specs.html
The offset fields specify the hours, and optionally minutes and seconds,
difference from UTC. They have the format hh[:mm[:ss]] optionally with a
leading sign (+ or -). The positive sign is used for zones west of
Greenwich. (Note that this is the opposite of the ISO-8601 sign convention
used elsewhere in PostgreSQL.)
David J.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-19 08:25 Andrey Borodin <[email protected]>
parent: David G. Johnston <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey Borodin @ 2024-01-19 08:25 UTC (permalink / raw)
To: David G. Johnston <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 19 Jan 2024, at 08:24, David G. Johnston <[email protected]> wrote:
>
>
> You are mixing POSIX and ISO-8601 conventions and, as noted in our appendix, they disagree on the direction that is positive.
Thanks! Now everything seems on its place.
I want to include in the patch following tests:
-- extract UUID v1, v6 and v7 timestamp
SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
How do you think, will it be stable all across buildfarm? Or should we change anything to avoid false positives inferred from different timestamp parsing?
> On 19 Jan 2024, at 07:58, Lukas Fittl <[email protected]> wrote:
>
> Note how calling the uuidv7 function again after having called it with a fixed future timestamp, returns the future timestamp, even though it should return the current time.
Thanks for the review.
Well, that was intentional. But now I see it's kind of confusing behaviour. I've changed it to more expected version.
Also, I've added some documentation on all functions.
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v11-0001-Implement-UUID-v7-as-per-IETF-draft.patch (16.2K, ../../[email protected]/2-v11-0001-Implement-UUID-v7-as-per-IETF-draft.patch)
download | inline diff:
From 98726fc9b75d9abaec8bb4b305531e79d16bf6b0 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v11] Implement UUID v7 as per IETF draft
This commit adds function to generate UUID v7.
This function optionally accepts datetime used to generate
next UUID.
Also we add a function to extract timestamp from UUID v7.
Authors: Andrey Borodin, Sergey Prokhorenko
---
doc/src/sgml/func.sgml | 36 ++++-
src/backend/utils/adt/uuid.c | 195 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 19 +++
src/test/regress/expected/opr_sanity.out | 13 +-
src/test/regress/expected/uuid.out | 88 ++++++++++
src/test/regress/sql/uuid.sql | 35 ++++
6 files changed, 380 insertions(+), 6 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 210c7c0b02..1d4d48d7cb 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14130,13 +14130,43 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes several functions to generate a UUID:
+ <function>gen_random_uuid</function>, <function>uuidv4</function>, and <function>uuidv7</function>.
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ Both functions return a version 4 (random) UUID. This is the most commonly
+ used type of UUID and is appropriate when random distribution of keys does
+ not affect performance of an application.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 (time-ordered + random) UUID. This UUID
+ version should be used when application prefers locality of identifiers.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID versions 1,6 and 7. For other
+ versions and variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_ver</function> (uuid) <returnvalue>int2</returnvalue>
+</synopsis>
+ This function extracts a version bits from UUID of variants described by
+ IETF standard (b10xx variant). For other variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_var</function> (uuid) <returnvalue>int2</returnvalue>
</synopsis>
- This function returns a version 4 (random) UUID. This is the most commonly
- used type of UUID and is appropriate for most applications.
+ This function extracts a vartiant bits from UUID.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c7..6125061b35 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,193 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+static bool external_times_used = false;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ if (PG_NARGS() == 0 || PG_ARGISNULL(0))
+ {
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = (tms <= previous_timestamp) && !external_times_used;
+ external_times_used = false;
+ }
+ else
+ {
+ ts = PG_GETARG_TIMESTAMPTZ(0);
+ tms = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC) / 1000;
+ /*
+ * The time can leap backwards when provided by the user, so we use
+ * counter only when called with exactly same unix_ts_ms argument.
+ */
+ increment_counter = (tms == previous_timestamp);
+ external_times_used = true;
+ }
+
+ if (increment_counter)
+ {
+ /* Time did not increment from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * http://tools.ietf.org/html/rfc ???
+ * https://datatracker.ietf.org/doc/html/draft-peabody-dispatch-new-uuid-format#name-creating-a-uuidv7-value
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - date2j(1582,10,15)) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - date2j(1582,10,15)) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58811a6530..956fb08ce9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,25 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'generate UUID version 7', proisstrict => 'f',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => 'timestamptz', prosrc => 'uuidv7',
+ proargnames => '{unix_ts_ms}', proargmodes => '{i}' },
+{ oid => '9898', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9899', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9900', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d6..1c37533975 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -126,9 +126,10 @@ WHERE p1.oid < p2.oid AND
p1.proretset != p2.proretset OR
p1.provolatile != p2.provolatile OR
p1.pronargs != p2.pronargs);
- oid | proname | oid | proname
------+---------+-----+---------
-(0 rows)
+ oid | proname | oid | proname
+------+---------+------+---------
+ 9896 | uuidv7 | 9897 | uuidv7
+(1 row)
-- Look for uses of different type OIDs in the argument/result type fields
-- for different aliases of the same built-in function.
@@ -872,6 +873,12 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuidv7(timestamp with time zone)
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d..df78fd0385 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,93 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+ ?column?
+----------
+ @ 0
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d..c7a09dd21d 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,40 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.37.1 (Apple Git-137.1)
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-19 11:07 Aleksander Alekseev <[email protected]>
parent: Przemysław Sztoch <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-19 11:07 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Przemysław Sztoch <[email protected]>; Andrey Borodin <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> No.
>
> Timestamp and TimestampTz are absolutely the same thing. The only
> difference is how they are shown to the user. TimestampTz uses session
> context in order to be displayed in the TZ chosen by the user. Thus
> typically it is somewhat more confusing to the users and thus I asked
> whether there was a good reason to choose TimestampTz over Timestamp.
>
>
> Theoretically, you're right. But look at this example:
>
> SET timezone TO 'Europe/Warsaw';
> SELECT extract(epoch from '2024-01-18 9:27:30'::timestamp), extract(epoch from '2024-01-18 9:27:30'::timestamptz);
>
> date_part | date_part
> ------------+------------
> 1705570050 | 1705566450
> (1 row)
>
> In my opinion, timestamptz gives greater guarantees that the time internally is in UTC and the user gets the time in his/her time zone.
I believe you didn't notice, but this example just proves my point.
In this case you have two timestamps that are different _internally_,
but the way they are _shown_ is the same because the first one is in
UTC and the second one in your local session timezone, Europe/Warsaw.
extract(epoch ...) extract UNIX epoch, i.e. relies on the _internal_
representation. This is why you got different results.
This demonstrates that TimestampTz is a permanent source of confusion
for the users and the reason why personally I would prefer if UUIDv7
always used Timestamp (no Tz). TimestampTz can be converted to
TimestampTz by users who need them and have experience using them.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-19 18:07 Andrey Borodin <[email protected]>
parent: Andrey Borodin <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey Borodin @ 2024-01-19 18:07 UTC (permalink / raw)
To: David G. Johnston <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 19 Jan 2024, at 13:25, Andrey Borodin <[email protected]> wrote:
>
> Also, I've added some documentation on all functions.
Here's v12. Changes:
1. Documentation improvements
2. Code comments
3. Better commit message and reviews list
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v12-0001-Implement-UUID-v7.patch (17.8K, ../../[email protected]/2-v12-0001-Implement-UUID-v7.patch)
download | inline diff:
From 10401a50691e7d8c8416895b9862beb1444d2bfc Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v12] Implement UUID v7
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This commit adds function for UUID generation.
Most important function here is uuidv7() which generates
new UUID according to new standard. This function can optionally
accept a timestamp used instead of current time. This allows
implementation of k-way sotable identifiers.
For code readability this commit adds alias uuidv4() to function
gen_random_uuid().
Also we add a function to extract timestamp from UUID v1, v6 and v7.
To allow user to distinguish various UUID versions and variants
we add functions uuid_extract_ver() and uuid_extract_var().
Author: Andrey Borodin
Reviewers: Sergey Prokhorenko, Kirk Wolak, Przemysław Sztoch
Reviewers: Nikolay Samokhvalov, Jelte Fennema-Nio, Aleksander Alekseev
Reviewers: Peter Eisentraut, Chris Travers, Lukas Fittl
---
doc/src/sgml/func.sgml | 49 +++++-
src/backend/utils/adt/uuid.c | 195 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 19 +++
src/include/datatype/timestamp.h | 3 +-
src/test/regress/expected/opr_sanity.out | 13 +-
src/test/regress/expected/uuid.out | 88 ++++++++++
src/test/regress/sql/uuid.sql | 35 ++++
7 files changed, 395 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 210c7c0b02..ce6715721f 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14130,13 +14130,56 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv4</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_ver</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_var</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes several functions to generate a UUID:
+ <function>gen_random_uuid</function>, <function>uuidv4</function>, and <function>uuidv7</function>.
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ Both functions return a version 4 (random) UUID. This is the most commonly
+ used type of UUID and is appropriate when random distribution of keys does
+ not affect performance of an application.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 (time-ordered + random) UUID. This UUID
+ version should be used when application prefers locality of identifiers.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
+ versions and variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_ver</function> (uuid) <returnvalue>int2</returnvalue>
+</synopsis>
+ This function extracts a version bits from UUID of variant described by
+ <ulink url="https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis">IETF standard</ulink>
+ (b10xx variant). For other variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_var</function> (uuid) <returnvalue>int2</returnvalue>
</synopsis>
- This function returns a version 4 (random) UUID. This is the most commonly
- used type of UUID and is appropriate for most applications.
+ This function extracts a vartiant bits from UUID.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c7..386cdb7a73 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,193 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+static bool external_times_used = false;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ if (PG_NARGS() == 0 || PG_ARGISNULL(0))
+ {
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = (tms <= previous_timestamp) && !external_times_used;
+ external_times_used = false;
+ }
+ else
+ {
+ ts = PG_GETARG_TIMESTAMPTZ(0);
+ tms = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC) / 1000;
+ /*
+ * The time can leap backwards when provided by the user, so we use
+ * counter only when called with exactly same unix_ts_ms argument.
+ */
+ increment_counter = (tms == previous_timestamp);
+ external_times_used = true;
+ }
+
+ if (increment_counter)
+ {
+ /* Time did not advance from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ /* read randomly initialized bits of counter */
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 58811a6530..956fb08ce9 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,25 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'generate UUID version 7', proisstrict => 'f',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => 'timestamptz', prosrc => 'uuidv7',
+ proargnames => '{unix_ts_ms}', proargmodes => '{i}' },
+{ oid => '9898', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9899', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9900', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 3a37cb661e..652aeb428e 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -230,9 +230,10 @@ struct pg_itm_in
((y) < JULIAN_MAXYEAR || \
((y) == JULIAN_MAXYEAR && ((m) < JULIAN_MAXMONTH))))
-/* Julian-date equivalents of Day 0 in Unix and Postgres reckoning */
+/* Julian-date equivalents of Day 0 in Unix, Postgres and Gregorian epochs */
#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define GREGORIAN_EPOCH_JDATE 2299161 /* == date2j(1582,10,15) */
/*
* Range limits for dates and timestamps.
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d6..1c37533975 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -126,9 +126,10 @@ WHERE p1.oid < p2.oid AND
p1.proretset != p2.proretset OR
p1.provolatile != p2.provolatile OR
p1.pronargs != p2.pronargs);
- oid | proname | oid | proname
------+---------+-----+---------
-(0 rows)
+ oid | proname | oid | proname
+------+---------+------+---------
+ 9896 | uuidv7 | 9897 | uuidv7
+(1 row)
-- Look for uses of different type OIDs in the argument/result type fields
-- for different aliases of the same built-in function.
@@ -872,6 +873,12 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuidv7(timestamp with time zone)
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d..df78fd0385 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,93 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+ ?column?
+----------
+ @ 0
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d..c7a09dd21d 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,40 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.37.1 (Apple Git-137.1)
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-22 04:22 Nikolay Samokhvalov <[email protected]>
parent: Andrey Borodin <[email protected]>
0 siblings, 3 replies; 69+ messages in thread
From: Nikolay Samokhvalov @ 2024-01-22 04:22 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Fri, Jan 19, 2024 at 10:07 AM Andrey Borodin <[email protected]>
wrote:
>
>
> > On 19 Jan 2024, at 13:25, Andrey Borodin <[email protected]> wrote:
> >
> > Also, I've added some documentation on all functions.
>
> Here's v12. Changes:
> 1. Documentation improvements
> 2. Code comments
> 3. Better commit message and reviews list
>
Thank you, Andrey! I have just checked v12 – cleanly applied to HEAD, and
functions work well. I especially like that fact that we keep
uuid_extract_time(..) here – this is a great thing to have for time-based
partitioning, and in many cases we will be able to decide not to have a
creation column timestamp (e.g., "created_at") at all, saving 8 bytes.
The docs and comments look great too.
Overall, the patch looks mature enough. It would be great to have it in
pg17. Yes, the RFC is not fully finalized yet, but it's very close. And
many libraries are already including implementation of UUIDv7 – here are
some examples:
- https://www.npmjs.com/package/uuidv7
- https://crates.io/crates/uuidv7
- https://github.com/google/uuid/pull/139
Nik
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-22 04:24 Nikolay Samokhvalov <[email protected]>
parent: Nikolay Samokhvalov <[email protected]>
2 siblings, 0 replies; 69+ messages in thread
From: Nikolay Samokhvalov @ 2024-01-22 04:24 UTC (permalink / raw)
To: [email protected]; +Cc: Andrey Borodin <[email protected]>; Kirk Wolak <[email protected]>
The following review has been posted through the commitfest application:
make installcheck-world: not tested
Implements feature: tested, passed
Spec compliant: not tested
Documentation: tested, passed
Manually tested uuidv7(), uuid_extract_time() – they work as expected. The basic docs provided look clear.
I haven't checked the tests though and possible edge cases, so leaving it as "needs review" waiting for more reviewers
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-22 15:02 Aleksander Alekseev <[email protected]>
parent: Nikolay Samokhvalov <[email protected]>
2 siblings, 1 reply; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-22 15:02 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> But now (after big timeseries project with multiple time zones and DST problems) I think differently.
> Even though timestamp and timestamptz are practically the same, timestamptz should be used to store the time in UTC.
> Choosing timestamp is more likely to lead to problems and misunderstandings than timestamptz.
As somebody who contributed TZ support to TimescaleDB I'm more or less
aware about the pros and cons of Timestamp and TimestampTz :)
Engineering is all about compromises. I can imagine a project where it
makes sense to use only TimestampTz for the entire database, and the
opposite - when it's better to use only UTC and Timestamp. In this
particular case I was merely concerned that the particular choice
could be confusing for the users but I think I changed my mind by now,
see below.
>> Here's v12. Changes:
>> 1. Documentation improvements
>> 2. Code comments
>> 3. Better commit message and reviews list
>
>
> Thank you, Andrey! I have just checked v12 – cleanly applied to HEAD, and functions work well. I especially like that fact that we keep uuid_extract_time(..) here – this is a great thing to have for time-based partitioning, and in many cases we will be able to decide not to have a creation column timestamp (e.g., "created_at") at all, saving 8 bytes.
>
> The docs and comments look great too.
>
> Overall, the patch looks mature enough. It would be great to have it in pg17. Yes, the RFC is not fully finalized yet, but it's very close. And many libraries are already including implementation of UUIDv7 – here are some examples:
>
> - https://www.npmjs.com/package/uuidv7
> - https://crates.io/crates/uuidv7
> - https://github.com/google/uuid/pull/139
Thanks!
After playing with v12 I'm inclined to agree that it's RfC.
I only have a couple of silly nitpicks:
- It could make sense to decompose the C implementation of uuidv7() in
two functions, for readability.
- It could make sense to get rid of curly braces in SQL tests when
calling uuid_extract_ver() and uuid_extract_ver(), for consistency.
I'm not going to insist on these changes though and prefer leaving it
to the author and the committer to decide.
Also I take back what I said above about using Timestamp instead of
TimestampTz. I forgot that Timestamps are implicitly casted to
TimestampTz's, so users preferring Timestamps can do this:
```
=# select uuidv7('2024-01-22 12:34:56' :: timestamp);
uuidv7
--------------------------------------
018d3085-de00-77c1-9e7b-7b04ddb9ebb9
```
Cfbot also seems to be happy with the patch so I'm changing the CF
entry status to RfC.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 12:31 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-24 12:31 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> Cfbot also seems to be happy with the patch so I'm changing the CF
> entry status to RfC.
I've found a bug:
```
=# select now() - interval '5000 years';
?column?
----------------------------------------
2977-01-24 15:29:01.779462+02:30:17 BC
Time: 0.957 ms
=# select uuidv7(now() - interval '5000 years');
uuidv7
--------------------------------------
720c1868-0764-7677-99cd-265b84ea08b9
=# select uuid_extract_time('720c1868-0764-7677-99cd-265b84ea08b9');
uuid_extract_time
----------------------------
5943-08-26 21:30:44.836+03
```
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 12:40 Andrey M. Borodin <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-24 12:40 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 24 Jan 2024, at 17:31, Aleksander Alekseev <[email protected]> wrote:
>
> Hi,
>
>> Cfbot also seems to be happy with the patch so I'm changing the CF
>> entry status to RfC.
>
> I've found a bug:
>
> ```
> =# select now() - interval '5000 years';
> ?column?
> ----------------------------------------
> 2977-01-24 15:29:01.779462+02:30:17 BC
>
> Time: 0.957 ms
>
> =# select uuidv7(now() - interval '5000 years');
> uuidv7
> --------------------------------------
> 720c1868-0764-7677-99cd-265b84ea08b9
>
> =# select uuid_extract_time('720c1868-0764-7677-99cd-265b84ea08b9');
> uuid_extract_time
> ----------------------------
> 5943-08-26 21:30:44.836+03
> ```
UUIDv7 range does not correspond to timestamp range. But it’s purpose is not in storing timestamp, but in being unique identifier. So I don’t think it worth throwing an error when overflowing value is given. BTW if you will subtract some nanoseconds - you will not get back timestamp you put into UUID too.
UUID does not store timpestamp, it only uses it to generate an identifier. Some value can be extracted back, but with limited precision, limited range and only if UUID was generated precisely by the specification in standard (and standard allows deviation! Most of implementation try to tradeoff something).
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 13:02 Aleksander Alekseev <[email protected]>
parent: Andrey M. Borodin <[email protected]>
0 siblings, 2 replies; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-24 13:02 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey M. Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> UUIDv7 range does not correspond to timestamp range. But it’s purpose is not in storing timestamp, but in being unique identifier. So I don’t think it worth throwing an error when overflowing value is given. BTW if you will subtract some nanoseconds - you will not get back timestamp you put into UUID too.
> UUID does not store timpestamp, it only uses it to generate an identifier. Some value can be extracted back, but with limited precision, limited range and only if UUID was generated precisely by the specification in standard (and standard allows deviation! Most of implementation try to tradeoff something).
I don't claim that UUIDv7 purpose is storing timestamps, but I think
the invariant:
```
uuid_extract_time(uidv7(X)) == X
```
and (!) even more importantly:
```
if X > Y then uuidv7(X) > uuidv7(Y)
```
... should hold. Otherwise you can calculate crc64(X) or sha256(X)
internally in order to generate an unique ID and claim that it's fine.
Values that violate named invariants should be rejected with an error.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 13:16 Andrey M. Borodin <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-24 13:16 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 24 Jan 2024, at 18:02, Aleksander Alekseev <[email protected]> wrote:
>
> Hi,
>
>> UUIDv7 range does not correspond to timestamp range. But it’s purpose is not in storing timestamp, but in being unique identifier. So I don’t think it worth throwing an error when overflowing value is given. BTW if you will subtract some nanoseconds - you will not get back timestamp you put into UUID too.
>> UUID does not store timpestamp, it only uses it to generate an identifier. Some value can be extracted back, but with limited precision, limited range and only if UUID was generated precisely by the specification in standard (and standard allows deviation! Most of implementation try to tradeoff something).
>
> I don't claim that UUIDv7 purpose is storing timestamps, but I think
> the invariant:
>
> ```
> uuid_extract_time(uidv7(X)) == X
> ```
>
> and (!) even more importantly:
>
> ```
> if X > Y then uuidv7(X) > uuidv7(Y)
> ```
>
> ... should hold.
Function to extract timestamp does not provide any guarantees at all. Standard states this, see Kyzer answers upthread.
Moreover, standard urges against relying on that if uuidX was generated before uuidY, then uuidX<uuid. The standard is doing a lot to make this happen, but does not guaranty that.
All what is guaranteed is the uniqueness at certain conditions.
> Otherwise you can calculate crc64(X) or sha256(X)
> internally in order to generate an unique ID and claim that it's fine.
>
> Values that violate named invariants should be rejected with an error.
Think about the value that you pass to uuid generation function as an entropy. It’s there to ensure uniqueness and promote ordering (but not guarantee).
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 13:16 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-24 13:16 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey M. Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> Values that violate named invariants should be rejected with an error.
To clarify, I don't think we should bother about the precision part.
"Equals" in the example above means "equal within UUIDv7 precision",
same for "more" and "less". However, years 2977 BC and 5943 AC are
clearly not equal, thus 2977 BC should be rejected as an invalid value
for UUIDv7.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 13:29 Aleksander Alekseev <[email protected]>
parent: Andrey M. Borodin <[email protected]>
0 siblings, 2 replies; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-24 13:29 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey M. Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> Function to extract timestamp does not provide any guarantees at all. Standard states this, see Kyzer answers upthread.
> Moreover, standard urges against relying on that if uuidX was generated before uuidY, then uuidX<uuid. The standard is doing a lot to make this happen, but does not guaranty that.
> All what is guaranteed is the uniqueness at certain conditions.
>
> > Otherwise you can calculate crc64(X) or sha256(X)
> > internally in order to generate an unique ID and claim that it's fine.
> >
> > Values that violate named invariants should be rejected with an error.
>
> Think about the value that you pass to uuid generation function as an entropy. It’s there to ensure uniqueness and promote ordering (but not guarantee).
If the standard doesn't guarantee something it doesn't mean it forbids
us to give stronger guarantees. I'm convinced that these guarantees
will be useful in real-world applications, at least the ones acting
exclusively within Postgres.
This being said, I understand your point of view too. Let's see what
other people think.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 15:29 Andrey M. Borodin <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-24 15:29 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 24 Jan 2024, at 18:29, Aleksander Alekseev <[email protected]> wrote:
>
> Hi,
>
>> Function to extract timestamp does not provide any guarantees at all. Standard states this, see Kyzer answers upthread.
>> Moreover, standard urges against relying on that if uuidX was generated before uuidY, then uuidX<uuid. The standard is doing a lot to make this happen, but does not guaranty that.
>> All what is guaranteed is the uniqueness at certain conditions.
>>
>>> Otherwise you can calculate crc64(X) or sha256(X)
>>> internally in order to generate an unique ID and claim that it's fine.
>>>
>>> Values that violate named invariants should be rejected with an error.
>>
>> Think about the value that you pass to uuid generation function as an entropy. It’s there to ensure uniqueness and promote ordering (but not guarantee).
>
> If the standard doesn't guarantee something it doesn't mean it forbids
> us to give stronger guarantees.
No, the standard makes these guarantees impossible.
If we insist that uuid_extract_time(uuidv7(time))==time, we won't be able to generate uuidv7 most of the time. uuidv7(now()) will always ERROR-out.
Standard implies more coarse-grained timestamp that we have.
Also, please not that uuidv7(time+1us) and uuidv7(time) will have the same internal timestamp, so despite time+1us > time, still second uuid will be greater.
Both invariants you proposed cannot be reasonably guaranteed. Upholding any of them greatly reduces usability of UUID v7.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 15:46 Aleksander Alekseev <[email protected]>
parent: Andrey M. Borodin <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-24 15:46 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey M. Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> Also, please not that uuidv7(time+1us) and uuidv7(time) will have the same internal timestamp, so despite time+1us > time, still second uuid will be greater.
>
> Both invariants you proposed cannot be reasonably guaranteed. Upholding any of them greatly reduces usability of UUID v7.
Again, personally I don't insist on the 1us precision [1]. Only the
fact that timestamp from the far past generates UUID from the future
bothers me.
[1]: https://postgr.es/m/CAJ7c6TPCSprWwVNdOB%3D%3DpgKZPqO5q%3DHRgmU7zmYqz9Dz5ffVYw%40mail.gmail.com
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 16:54 Andrey M. Borodin <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-24 16:54 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 24 Jan 2024, at 20:46, Aleksander Alekseev <[email protected]> wrote:
>
> Only the
> fact that timestamp from the far past generates UUID from the future
> bothers me.
PFA implementation of guard checks, but I'm afraid that this can cause failures in ID generation unexpected to the user...
See tests
+-- errors in edge cases of UUID v7
+SELECT 1 FROM uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '0ms');
+SELECT uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '1ms'); -- ERROR expected
+SELECT 1 FROM uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000'));
+SELECT uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000')+'1ms'); -- ERROR expected
Range is from 1970-01-01 00:00:00 to 10889-08-02 05:31:50.655. I'm not sure we should give this information in error message...
Thanks!
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v13-0001-Implement-UUID-v7.patch (18.9K, ../../[email protected]/2-v13-0001-Implement-UUID-v7.patch)
download | inline diff:
From 5ae983c668ed5e17c0b34faefbc62b8200704bb7 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v13] Implement UUID v7
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This commit adds function for UUID generation.
Most important function here is uuidv7() which generates
new UUID according to new standard. This function can optionally
accept a timestamp used instead of current time. This allows
implementation of k-way sotable identifiers.
For code readability this commit adds alias uuidv4() to function
gen_random_uuid().
Also we add a function to extract timestamp from UUID v1, v6 and v7.
To allow user to distinguish various UUID versions and variants
we add functions uuid_extract_ver() and uuid_extract_var().
Author: Andrey Borodin
Reviewers: Sergey Prokhorenko, Kirk Wolak, Przemysław Sztoch
Reviewers: Nikolay Samokhvalov, Jelte Fennema-Nio, Aleksander Alekseev
Reviewers: Peter Eisentraut, Chris Travers, Lukas Fittl
---
doc/src/sgml/func.sgml | 49 +++++-
src/backend/utils/adt/uuid.c | 200 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 19 +++
src/include/datatype/timestamp.h | 3 +-
src/test/regress/expected/opr_sanity.out | 13 +-
src/test/regress/expected/uuid.out | 105 ++++++++++++
src/test/regress/sql/uuid.sql | 41 +++++
7 files changed, 423 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 5030a1045f..ec09d06bd8 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14130,13 +14130,56 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv4</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_ver</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_var</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes several functions to generate a UUID:
+ <function>gen_random_uuid</function>, <function>uuidv4</function>, and <function>uuidv7</function>.
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ Both functions return a version 4 (random) UUID. This is the most commonly
+ used type of UUID and is appropriate when random distribution of keys does
+ not affect performance of an application.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 (time-ordered + random) UUID. This UUID
+ version should be used when application prefers locality of identifiers.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
+ versions and variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_ver</function> (uuid) <returnvalue>int2</returnvalue>
+</synopsis>
+ This function extracts a version bits from UUID of variant described by
+ <ulink url="https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis">IETF standard</ulink>
+ (b10xx variant). For other variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_var</function> (uuid) <returnvalue>int2</returnvalue>
</synopsis>
- This function returns a version 4 (random) UUID. This is the most commonly
- used type of UUID and is appropriate for most applications.
+ This function extracts a vartiant bits from UUID.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c7..f88ae95710 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,198 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+static bool external_times_used = false;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ if (PG_NARGS() == 0 || PG_ARGISNULL(0))
+ {
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = (tms <= previous_timestamp) && !external_times_used;
+ external_times_used = false;
+ }
+ else
+ {
+ ts = PG_GETARG_TIMESTAMPTZ(0);
+ tms = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC) / 1000;
+ /*
+ * The time can leap backwards when provided by the user, so we use
+ * counter only when called with exactly same unix_ts_ms argument.
+ */
+ increment_counter = (tms == previous_timestamp);
+ external_times_used = true;
+ if (tms & ~0xFFFFFFFFFFFF)
+ {
+ /* The standard allows only 6bytes of tms */
+ elog(ERROR, "Time argument of UUID v7 cannot exceed 6 bytes");
+ }
+ }
+
+ if (increment_counter)
+ {
+ /* Time did not advance from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ /* read randomly initialized bits of counter */
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index e4115cd084..16eabfbcbd 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,25 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'generate UUID version 7', proisstrict => 'f',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => 'timestamptz', prosrc => 'uuidv7',
+ proargnames => '{unix_ts_ms}', proargmodes => '{i}' },
+{ oid => '9898', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9899', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9900', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 3a37cb661e..652aeb428e 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -230,9 +230,10 @@ struct pg_itm_in
((y) < JULIAN_MAXYEAR || \
((y) == JULIAN_MAXYEAR && ((m) < JULIAN_MAXMONTH))))
-/* Julian-date equivalents of Day 0 in Unix and Postgres reckoning */
+/* Julian-date equivalents of Day 0 in Unix, Postgres and Gregorian epochs */
#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define GREGORIAN_EPOCH_JDATE 2299161 /* == date2j(1582,10,15) */
/*
* Range limits for dates and timestamps.
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d6..1c37533975 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -126,9 +126,10 @@ WHERE p1.oid < p2.oid AND
p1.proretset != p2.proretset OR
p1.provolatile != p2.provolatile OR
p1.pronargs != p2.pronargs);
- oid | proname | oid | proname
------+---------+-----+---------
-(0 rows)
+ oid | proname | oid | proname
+------+---------+------+---------
+ 9896 | uuidv7 | 9897 | uuidv7
+(1 row)
-- Look for uses of different type OIDs in the argument/result type fields
-- for different aliases of the same built-in function.
@@ -872,6 +873,12 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuidv7(timestamp with time zone)
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d..22488d5990 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,110 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+ ?column?
+----------
+ @ 0
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+-- errors in edge cases of UUID v7
+SELECT 1 FROM uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '0ms');
+ ?column?
+----------
+ 1
+(1 row)
+
+SELECT uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '1ms'); -- ERROR expected
+ERROR: Time argument of UUID v7 cannot exceed 6 bytes
+SELECT 1 FROM uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000'));
+ ?column?
+----------
+ 1
+(1 row)
+
+SELECT uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000')+'1ms'); -- ERROR expected
+ERROR: Time argument of UUID v7 cannot exceed 6 bytes
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d..40c3152697 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,46 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+
+-- errors in edge cases of UUID v7
+SELECT 1 FROM uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '0ms');
+SELECT uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '1ms'); -- ERROR expected
+SELECT 1 FROM uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000'));
+SELECT uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000')+'1ms'); -- ERROR expected
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.37.1 (Apple Git-137.1)
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 17:00 Marcos Pegoraro <[email protected]>
parent: Andrey M. Borodin <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Marcos Pegoraro @ 2024-01-24 17:00 UTC (permalink / raw)
To: Andrey M. Borodin <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Is enough from 1970 ?
How about if user wants to have an UUID of his birth date ?
regards
Marcos
Em qua., 24 de jan. de 2024 às 13:54, Andrey M. Borodin <
[email protected]> escreveu:
>
>
> > On 24 Jan 2024, at 20:46, Aleksander Alekseev <[email protected]>
> wrote:
> >
> > Only the
> > fact that timestamp from the far past generates UUID from the future
> > bothers me.
>
> PFA implementation of guard checks, but I'm afraid that this can cause
> failures in ID generation unexpected to the user...
> See tests
>
> +-- errors in edge cases of UUID v7
> +SELECT 1 FROM uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval
> '0ms');
> +SELECT uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '1ms'); --
> ERROR expected
> +SELECT 1 FROM
> uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000'));
> +SELECT
> uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000')+'1ms'); --
> ERROR expected
>
> Range is from 1970-01-01 00:00:00 to 10889-08-02 05:31:50.655. I'm not
> sure we should give this information in error message...
> Thanks!
>
>
> Best regards, Andrey Borodin.
>
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 17:51 Andrey Borodin <[email protected]>
parent: Marcos Pegoraro <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey Borodin @ 2024-01-24 17:51 UTC (permalink / raw)
To: Marcos Pegoraro <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 24 Jan 2024, at 22:00, Marcos Pegoraro <[email protected]> wrote:
>
> Is enough from 1970 ?
Per standard unix_ts_ms field is a number of milliseconds from UNIX start date 1970-01-01.
> How about if user wants to have an UUID of his birth date ?
I've claimed my
0078c135-bd00-70b1-865a-63c3741922a5
But again, UUIDs are not designed to store timestamp. They are unique and v7 promote data locality via time-ordering.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 20:47 Marcos Pegoraro <[email protected]>
parent: Andrey Borodin <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Marcos Pegoraro @ 2024-01-24 20:47 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Aleksander Alekseev <[email protected]>; pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
I understand your point, but
'2000-01-01' :: timestamp and '1900-01-01' :: timestamp are both valid
timestamps.
So looks strange if user can do
select uuidv7(TIMESTAMP '2000-01-01')
but cannot do
select uuidv7(TIMESTAMP '1900-01-01')
Regards
Marcos
Em qua., 24 de jan. de 2024 às 14:51, Andrey Borodin <[email protected]>
escreveu:
>
>
> > On 24 Jan 2024, at 22:00, Marcos Pegoraro <[email protected]> wrote:
> >
> > Is enough from 1970 ?
> Per standard unix_ts_ms field is a number of milliseconds from UNIX start
> date 1970-01-01.
>
> > How about if user wants to have an UUID of his birth date ?
>
> I've claimed my
> 0078c135-bd00-70b1-865a-63c3741922a5
>
> But again, UUIDs are not designed to store timestamp. They are unique and
> v7 promote data locality via time-ordering.
>
>
> Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 21:15 Jelte Fennema-Nio <[email protected]>
parent: Marcos Pegoraro <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Jelte Fennema-Nio @ 2024-01-24 21:15 UTC (permalink / raw)
To: Marcos Pegoraro <[email protected]>; +Cc: Andrey Borodin <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Sergey Prokhorenko <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Wed, 24 Jan 2024 at 21:47, Marcos Pegoraro <[email protected]> wrote:
>
> I understand your point, but
> '2000-01-01' :: timestamp and '1900-01-01' :: timestamp are both valid timestamps.
>
> So looks strange if user can do
> select uuidv7(TIMESTAMP '2000-01-01')
> but cannot do
> select uuidv7(TIMESTAMP '1900-01-01')
I think that would be okay honestly. I don't think there's any
reasonable value for the uuid when a timestamp is given outside of the
date range that the uuid7 "algorithm" supports.
So +1 for erroring when you provide a timestamp outside of that range
(either too far in the past or too far in the future).
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 21:30 Sergey Prokhorenko <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-24 21:30 UTC (permalink / raw)
To: pgsql-hackers; Aleksander Alekseev <[email protected]>; +Cc: Andrey M. Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
"Other people" think that extracting the timestamp from UUIDv7 in violation of the new RFC, and generating UUIDv7 from the timestamp were both terrible and poorly thought out ideas. The authors of the new RFC had very good reasons to prohibit this. And the problems you face are the best confirmation of the correctness of the new RFC. It’s better to throw all this gag out of the official patch. Don't tempt developers to break the new RFC with these error-producing functions.
Sergey [email protected]
On Wednesday, 24 January 2024 at 04:30:02 pm GMT+3, Aleksander Alekseev <[email protected]> wrote:
Hi,
> Function to extract timestamp does not provide any guarantees at all. Standard states this, see Kyzer answers upthread.
> Moreover, standard urges against relying on that if uuidX was generated before uuidY, then uuidX<uuid. The standard is doing a lot to make this happen, but does not guaranty that.
> All what is guaranteed is the uniqueness at certain conditions.
>
> > Otherwise you can calculate crc64(X) or sha256(X)
> > internally in order to generate an unique ID and claim that it's fine.
> >
> > Values that violate named invariants should be rejected with an error.
>
> Think about the value that you pass to uuid generation function as an entropy. It’s there to ensure uniqueness and promote ordering (but not guarantee).
If the standard doesn't guarantee something it doesn't mean it forbids
us to give stronger guarantees. I'm convinced that these guarantees
will be useful in real-world applications, at least the ones acting
exclusively within Postgres.
This being said, I understand your point of view too. Let's see what
other people think.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-24 21:49 Sergey Prokhorenko <[email protected]>
parent: Nikolay Samokhvalov <[email protected]>
2 siblings, 2 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-24 21:49 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; +Cc: David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
That's right! There is no point in waiting for the official approval of the new RFC, which obviously will not change anything. I have been a contributor to this RFC for several years, and I can testify that every aspect imaginable has been thoroughly researched and agreed upon. Nothing new will definitely appear in the new RFC.
Sergey [email protected]
On Monday, 22 January 2024 at 07:22:32 am GMT+3, Nikolay Samokhvalov <[email protected]> wrote:
On Fri, Jan 19, 2024 at 10:07 AM Andrey Borodin <[email protected]> wrote:
> On 19 Jan 2024, at 13:25, Andrey Borodin <[email protected]> wrote:
>
> Also, I've added some documentation on all functions.
Here's v12. Changes:
1. Documentation improvements
2. Code comments
3. Better commit message and reviews list
Thank you, Andrey! I have just checked v12 – cleanly applied to HEAD, and functions work well. I especially like that fact that we keep uuid_extract_time(..) here – this is a great thing to have for time-based partitioning, and in many cases we will be able to decide not to have a creation column timestamp (e.g., "created_at") at all, saving 8 bytes.
The docs and comments look great too.
Overall, the patch looks mature enough. It would be great to have it in pg17. Yes, the RFC is not fully finalized yet, but it's very close. And many libraries are already including implementation of UUIDv7 – here are some examples:
- https://www.npmjs.com/package/uuidv7
- https://crates.io/crates/uuidv7
- https://github.com/google/uuid/pull/139
Nik
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 04:40 Nikolay Samokhvalov <[email protected]>
parent: Sergey Prokhorenko <[email protected]>
1 sibling, 2 replies; 69+ messages in thread
From: Nikolay Samokhvalov @ 2024-01-25 04:40 UTC (permalink / raw)
To: Sergey Prokhorenko <[email protected]>; +Cc: Andrey Borodin <[email protected]>; David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
--0000000000002792f3060fbdc8f7
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
On Wed, Jan 24, 2024 at 1:52=E2=80=AFPM Sergey Prokhorenko <
[email protected]> wrote:
> That's right! There is no point in waiting for the official approval of
> the new RFC, which obviously will not change anything. I have been a
> contributor to this RFC
> <https://www.ietf.org/archive/id/draft-ietf-uuidrev-rfc4122bis-14.html#na=
me-acknowledgements>
> for several years, and I can testify that every aspect imaginable has bee=
n
> thoroughly researched and agreed upon. Nothing new will definitely appear
> in the new RFC.
>
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 04:41 Nikolay Samokhvalov <[email protected]>
parent: Nikolay Samokhvalov <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Nikolay Samokhvalov @ 2024-01-25 04:41 UTC (permalink / raw)
To: Sergey Prokhorenko <[email protected]>; +Cc: Andrey Borodin <[email protected]>; David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Wed, Jan 24, 2024 at 8:40 PM Nikolay Samokhvalov <[email protected]> wrote:
> On Wed, Jan 24, 2024 at 1:52 PM Sergey Prokhorenko <
> [email protected]> wrote:
>
>> That's right! There is no point in waiting for the official approval of
>> the new RFC, which obviously will not change anything. I have been a
>> contributor to this RFC
>> <https://www.ietf.org/archive/id/draft-ietf-uuidrev-rfc4122bis-14.html#name-acknowledgements;
>> for several years, and I can testify that every aspect imaginable has been
>> thoroughly researched and agreed upon. Nothing new will definitely
>> appear in the new RFC.
>>
>
> From a practical point of view, these two things are extremely important
> to have to support partitioning. It is better to implement limitations than
> throw them away.
>
> Without them, this functionality will be of a very limited use in
> databases. We need to think about large tables – which means partitioning.
>
apologies -- this was a response to another email from you:
> "Other people" think that extracting the timestamp from UUIDv7 in
violation of the new RFC, and generating UUIDv7 from the timestamp were
both terrible and poorly thought out ideas. The authors of the new RFC had
very good reasons to prohibit this. And the problems you face are the best
confirmation of the correctness of the new RFC. It’s better to throw all
this gag out of the official patch. Don't tempt developers to break the new
RFC with these error-producing functions.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 06:51 Andrey M. Borodin <[email protected]>
parent: Nikolay Samokhvalov <[email protected]>
1 sibling, 2 replies; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-25 06:51 UTC (permalink / raw)
To: Nikolay Samokhvalov <[email protected]>; +Cc: Sergey Prokhorenko <[email protected]>; David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 25 Jan 2024, at 09:40, Nikolay Samokhvalov <[email protected]> wrote:
>
> From a practical point of view, these two things are extremely important to have to support partitioning. It is better to implement limitations than throw them away.
Postgres always was a bit hackerish, allowing slightly more then is safe. I.e. you can define immutable function that is not really immutable, turn off autovacuum or fsync. Why bother with safety guards here?
My opinion is that we should have this function to extract timestamp. Even if it can return strange values for imprecise RFC implementation.
> On 25 Jan 2024, at 02:15, Jelte Fennema-Nio <[email protected]> wrote:
>
> So +1 for erroring when you provide a timestamp outside of that range
> (either too far in the past or too far in the future).
OK, it seems like we have some consensus on ERRORing..
Do we have any other open items? Does v13 address all open items? Maybe let’s compose better error message?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 08:09 Sergey Prokhorenko <[email protected]>
parent: Andrey M. Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-25 08:09 UTC (permalink / raw)
To: Nikolay Samokhvalov <[email protected]>; Andrey M. Borodin <[email protected]>; +Cc: David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
I am against turning the DBMS into another C++, in which they do not so much design something new as fix bugs in production after a crash.
As for partitioning, I already wrote to Andrey Borodin that we need a special function to generate a partition id using the UUIDv7 timestamp or even simultaneously with the generation of the timestamp. For example, every month (or so, since precision is not needed here) a new partition is created. Here's a good example: https://elixirforum.com/t/partitioning-postgres-tables-by-timestamp-based-uuids/60916
But without a separate function for extracting the entire timestamp from the UUID! Let's solve this specific problem, and not give the developers a grenade with the safety removed. Many developers have already decided to store the timestamp in UUIDv7, so as not to create a separate created_at field. Then they will delete table records with the old timestamp, etc. Horrible mistakes are simply guaranteed.
Sergey Prokhorenko [email protected]
On Thursday, 25 January 2024 at 09:51:58 am GMT+3, Andrey M. Borodin <[email protected]> wrote:
> On 25 Jan 2024, at 09:40, Nikolay Samokhvalov <[email protected]> wrote:
>
> From a practical point of view, these two things are extremely important to have to support partitioning. It is better to implement limitations than throw them away.
Postgres always was a bit hackerish, allowing slightly more then is safe. I.e. you can define immutable function that is not really immutable, turn off autovacuum or fsync. Why bother with safety guards here?
My opinion is that we should have this function to extract timestamp. Even if it can return strange values for imprecise RFC implementation.
> On 25 Jan 2024, at 02:15, Jelte Fennema-Nio <[email protected]> wrote:
>
> So +1 for erroring when you provide a timestamp outside of that range
> (either too far in the past or too far in the future).
OK, it seems like we have some consensus on ERRORing..
Do we have any other open items? Does v13 address all open items? Maybe let’s compose better error message?
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 11:14 Przemysław Sztoch <[email protected]>
parent: Andrey M. Borodin <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Przemysław Sztoch @ 2024-01-25 11:14 UTC (permalink / raw)
To: Andrey M. Borodin <[email protected]>; +Cc: Nikolay Samokhvalov <[email protected]>; Sergey Prokhorenko <[email protected]>; David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Andrey M. Borodin wrote on 25.01.2024 07:51:
>
>> On 25 Jan 2024, at 09:40, Nikolay Samokhvalov <[email protected]> wrote:
>>
>> From a practical point of view, these two things are extremely important to have to support partitioning. It is better to implement limitations than throw them away.
> Postgres always was a bit hackerish, allowing slightly more then is safe. I.e. you can define immutable function that is not really immutable, turn off autovacuum or fsync. Why bother with safety guards here?
> My opinion is that we should have this function to extract timestamp. Even if it can return strange values for imprecise RFC implementation.
>
>
>> On 25 Jan 2024, at 02:15, Jelte Fennema-Nio <[email protected]> wrote:
>>
>> So +1 for erroring when you provide a timestamp outside of that range
>> (either too far in the past or too far in the future).
>
> OK, it seems like we have some consensus on ERRORing..
>
> Do we have any other open items? Does v13 address all open items? Maybe let’s compose better error message?
+1 for erroring when ts is outside range.
v13 looks good for me. I think we have reached a optimal compromise.
--
Przemysław Sztoch | Mobile +48 509 99 00 66
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 12:06 Aleksander Alekseev <[email protected]>
parent: Przemysław Sztoch <[email protected]>
0 siblings, 2 replies; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-25 12:06 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey M. Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; Sergey Prokhorenko <[email protected]>; David G. Johnston <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi,
> Postgres always was a bit hackerish, allowing slightly more then is safe. I.e. you can define immutable function that is not really immutable, turn off autovacuum or fsync. Why bother with safety guards here?
> My opinion is that we should have this function to extract timestamp. Even if it can return strange values for imprecise RFC implementation.
Completely agree.
Users that don't like or don't need it can pretend there are no
uuid_extract_time() and uuidv7(T) in Postgres. If we don't provide
them however, users that need them will end up writing their own
probably buggy and not compatible implementations. That would be much
worse.
> So +1 for erroring when you provide a timestamp outside of that range
> (either too far in the past or too far in the future).
>
> OK, it seems like we have some consensus on ERRORing..
>
> Do we have any other open items? Does v13 address all open items? Maybe let’s compose better error message?
>
> +1 for erroring when ts is outside range.
>
> v13 looks good for me. I think we have reached a optimal compromise.
Andrey, many thanks for the updated patch.
LGTM, cfbot is happy and I don't think we have any open items left. So
changing CF entry status back to RfC.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 12:31 Aleksander Alekseev <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Aleksander Alekseev @ 2024-01-25 12:31 UTC (permalink / raw)
To: pgsql-hackers; +Cc: Andrey M. Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; Sergey Prokhorenko <[email protected]>; David G. Johnston <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>; Alexander Lakhin <[email protected]>
Hi,
> Andrey, many thanks for the updated patch.
>
> LGTM, cfbot is happy and I don't think we have any open items left. So
> changing CF entry status back to RfC.
PFA v14. I changed:
```
elog(ERROR, "Time argument of UUID v7 cannot exceed 6 bytes");
```
... to:
```
ereport(ERROR,
(errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("Time argument of UUID v7 is outside of the valid range")));
```
Which IMO tells a bit more to the average user and is translatable.
> At a quick glance, the patch needs improving English, IMO.
Agree. We could use some help from a native English speaker for this.
--
Best regards,
Aleksander Alekseev
Attachments:
[application/octet-stream] v14-0001-Implement-UUID-v7.patch (19.1K, ../../CAJ7c6TPi8o7+Kn=WtGArDnEfnm=cnrZRfbM-ZDBhyAomuBhr3A@mail.gmail.com/2-v14-0001-Implement-UUID-v7.patch)
download | inline diff:
From 874d7653f0345a93db3b6b8d954061d073d37915 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v14] Implement UUID v7
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This commit adds function for UUID generation. Most important function here
is uuidv7() which generates new UUID according to the new standard.
This function can optionally accept a timestamp used instead of current time.
This allows implementation of k-way sotable identifiers. For code readability
this commit adds alias uuidv4() to function gen_random_uuid().
Also we add a function to extract timestamp from UUID v1, v6 and v7.
To allow user to distinguish various UUID versions and variants
we add functions uuid_extract_ver() and uuid_extract_var().
Author: Andrey Borodin
Reviewers: Sergey Prokhorenko, Kirk Wolak, Przemysław Sztoch
Reviewers: Nikolay Samokhvalov, Jelte Fennema-Nio, Aleksander Alekseev
Reviewers: Peter Eisentraut, Chris Travers, Lukas Fittl
Discussion: https://postgr.es/m/CAAhFRxitJv%3DyoGnXUgeLB_O%2BM7J2BJAmb5jqAT9gZ3bij3uLDA%40mail.gmail.com
---
doc/src/sgml/func.sgml | 49 +++++-
src/backend/utils/adt/uuid.c | 202 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 19 +++
src/include/datatype/timestamp.h | 3 +-
src/test/regress/expected/opr_sanity.out | 13 +-
src/test/regress/expected/uuid.out | 105 ++++++++++++
src/test/regress/sql/uuid.sql | 41 +++++
7 files changed, 425 insertions(+), 7 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 968e8d59fb..74fbb982ab 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14130,13 +14130,56 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv4</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_ver</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_var</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes several functions to generate a UUID:
+ <function>gen_random_uuid</function>, <function>uuidv4</function>, and <function>uuidv7</function>.
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ Both functions return a version 4 (random) UUID. This is the most commonly
+ used type of UUID and is appropriate when random distribution of keys does
+ not affect performance of an application.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 (time-ordered + random) UUID. This UUID
+ version should be used when application prefers locality of identifiers.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
+ versions and variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_ver</function> (uuid) <returnvalue>int2</returnvalue>
+</synopsis>
+ This function extracts a version bits from UUID of variant described by
+ <ulink url="https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis">IETF standard</ulink>
+ (b10xx variant). For other variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_var</function> (uuid) <returnvalue>int2</returnvalue>
</synopsis>
- This function returns a version 4 (random) UUID. This is the most commonly
- used type of UUID and is appropriate for most applications.
+ This function extracts a vartiant bits from UUID.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c7..665e27f498 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,200 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+static bool external_times_used = false;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ if (PG_NARGS() == 0 || PG_ARGISNULL(0))
+ {
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = (tms <= previous_timestamp) && !external_times_used;
+ external_times_used = false;
+ }
+ else
+ {
+ ts = PG_GETARG_TIMESTAMPTZ(0);
+ tms = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC) / 1000;
+ /*
+ * The time can leap backwards when provided by the user, so we use
+ * counter only when called with exactly same unix_ts_ms argument.
+ */
+ increment_counter = (tms == previous_timestamp);
+ external_times_used = true;
+ if (tms & ~0xFFFFFFFFFFFF)
+ {
+ /* The standard allows only 6bytes of tms */
+ ereport(ERROR,
+ (errcode(ERRCODE_INVALID_PARAMETER_VALUE),
+ errmsg("Time argument of UUID v7 is outside of the valid range")));
+ }
+ }
+
+ if (increment_counter)
+ {
+ /* Time did not advance from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ /* read randomly initialized bits of counter */
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 29af4ce65d..414a9a417f 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,25 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'generate UUID version 7', proisstrict => 'f',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => 'timestamptz', prosrc => 'uuidv7',
+ proargnames => '{unix_ts_ms}', proargmodes => '{i}' },
+{ oid => '9898', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9899', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9900', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 3a37cb661e..652aeb428e 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -230,9 +230,10 @@ struct pg_itm_in
((y) < JULIAN_MAXYEAR || \
((y) == JULIAN_MAXYEAR && ((m) < JULIAN_MAXMONTH))))
-/* Julian-date equivalents of Day 0 in Unix and Postgres reckoning */
+/* Julian-date equivalents of Day 0 in Unix, Postgres and Gregorian epochs */
#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define GREGORIAN_EPOCH_JDATE 2299161 /* == date2j(1582,10,15) */
/*
* Range limits for dates and timestamps.
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d6..1c37533975 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -126,9 +126,10 @@ WHERE p1.oid < p2.oid AND
p1.proretset != p2.proretset OR
p1.provolatile != p2.provolatile OR
p1.pronargs != p2.pronargs);
- oid | proname | oid | proname
------+---------+-----+---------
-(0 rows)
+ oid | proname | oid | proname
+------+---------+------+---------
+ 9896 | uuidv7 | 9897 | uuidv7
+(1 row)
-- Look for uses of different type OIDs in the argument/result type fields
-- for different aliases of the same built-in function.
@@ -872,6 +873,12 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuidv7(timestamp with time zone)
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d..e85174ae82 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,110 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+ ?column?
+----------
+ @ 0
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+-- errors in edge cases of UUID v7
+SELECT 1 FROM uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '0ms');
+ ?column?
+----------
+ 1
+(1 row)
+
+SELECT uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '1ms'); -- ERROR expected
+ERROR: Time argument of UUID v7 is outside of the valid range
+SELECT 1 FROM uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000'));
+ ?column?
+----------
+ 1
+(1 row)
+
+SELECT uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000')+'1ms'); -- ERROR expected
+ERROR: Time argument of UUID v7 is outside of the valid range
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d..40c3152697 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,46 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7 with same unix_ts_ms
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+INSERT INTO guid1 (guid_field) VALUES (uuidv7(now()));
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- check that timestamp is extracted correctly
+SELECT uuid_extract_time(uuidv7(TIMESTAMP '2024-01-16 13:37:00')) - TIMESTAMP '2024-01-16 13:37:00';
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+
+-- errors in edge cases of UUID v7
+SELECT 1 FROM uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '0ms');
+SELECT uuidv7('1970-01-01 00:00:00+00'::timestamptz - interval '1ms'); -- ERROR expected
+SELECT 1 FROM uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000'));
+SELECT uuidv7(uuid_extract_time('FFFFFFFF-FFFF-7FFF-B000-000000000000')+'1ms'); -- ERROR expected
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.43.0
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-25 17:04 Sergey Prokhorenko <[email protected]>
parent: Aleksander Alekseev <[email protected]>
1 sibling, 2 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-25 17:04 UTC (permalink / raw)
To: pgsql-hackers; Aleksander Alekseev <[email protected]>; +Cc: Andrey M. Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Aleksander,
In this case the documentation must state that the functions uuid_extract_time() and uuidv7(T) are against the RFC requirements, and that developers may use these functions with caution at their own risk, and these functions are not recommended for production environment.
The function uuidv7(T) is not better than uuid_extract_time(). Careless developers may well pass any business date into this function: document date, registration date, payment date, reporting date, start date of the current month, data download date, and even a constant. This would be a profanation of UUIDv7 with very negative consequences.
Sergey [email protected]
On Thursday, 25 January 2024 at 03:06:50 pm GMT+3, Aleksander Alekseev <[email protected]> wrote:
Hi,
> Postgres always was a bit hackerish, allowing slightly more then is safe. I.e. you can define immutable function that is not really immutable, turn off autovacuum or fsync. Why bother with safety guards here?
> My opinion is that we should have this function to extract timestamp. Even if it can return strange values for imprecise RFC implementation.
Completely agree.
Users that don't like or don't need it can pretend there are no
uuid_extract_time() and uuidv7(T) in Postgres. If we don't provide
them however, users that need them will end up writing their own
probably buggy and not compatible implementations. That would be much
worse.
> So +1 for erroring when you provide a timestamp outside of that range
> (either too far in the past or too far in the future).
>
> OK, it seems like we have some consensus on ERRORing..
>
> Do we have any other open items? Does v13 address all open items? Maybe let’s compose better error message?
>
> +1 for erroring when ts is outside range.
>
> v13 looks good for me. I think we have reached a optimal compromise.
Andrey, many thanks for the updated patch.
LGTM, cfbot is happy and I don't think we have any open items left. So
changing CF entry status back to RfC.
--
Best regards,
Aleksander Alekseev
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-28 12:42 Sergey Prokhorenko <[email protected]>
parent: Sergey Prokhorenko <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-28 12:42 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; Nikolay Samokhvalov <[email protected]>; +Cc: David G. Johnston <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; Przemysław Sztoch <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
By the way, the Go language has also already implemented a function for UUIDv7: https://pkg.go.dev/github.com/gofrs/uuid#NewV7
Sergey Prokhorenko [email protected]
On Thursday, 25 January 2024 at 12:49:46 am GMT+3, Sergey Prokhorenko <[email protected]> wrote:
That's right! There is no point in waiting for the official approval of the new RFC, which obviously will not change anything. I have been a contributor to this RFC for several years, and I can testify that every aspect imaginable has been thoroughly researched and agreed upon. Nothing new will definitely appear in the new RFC.
Sergey [email protected]
On Monday, 22 January 2024 at 07:22:32 am GMT+3, Nikolay Samokhvalov <[email protected]> wrote:
On Fri, Jan 19, 2024 at 10:07 AM Andrey Borodin <[email protected]> wrote:
> On 19 Jan 2024, at 13:25, Andrey Borodin <[email protected]> wrote:
>
> Also, I've added some documentation on all functions.
Here's v12. Changes:
1. Documentation improvements
2. Code comments
3. Better commit message and reviews list
Thank you, Andrey! I have just checked v12 – cleanly applied to HEAD, and functions work well. I especially like that fact that we keep uuid_extract_time(..) here – this is a great thing to have for time-based partitioning, and in many cases we will be able to decide not to have a creation column timestamp (e.g., "created_at") at all, saving 8 bytes.
The docs and comments look great too.
Overall, the patch looks mature enough. It would be great to have it in pg17. Yes, the RFC is not fully finalized yet, but it's very close. And many libraries are already including implementation of UUIDv7 – here are some examples:
- https://www.npmjs.com/package/uuidv7
- https://crates.io/crates/uuidv7
- https://github.com/google/uuid/pull/139
Nik
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-29 11:38 Jelte Fennema-Nio <[email protected]>
parent: Sergey Prokhorenko <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Jelte Fennema-Nio @ 2024-01-29 11:38 UTC (permalink / raw)
To: Sergey Prokhorenko <[email protected]>; +Cc: pgsql-hackers; Aleksander Alekseev <[email protected]>; Andrey M. Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
tl;dr I believe we should remove the uuidv7(timestamp) function from
this patchset.
On Thu, 25 Jan 2024 at 18:04, Sergey Prokhorenko
<[email protected]> wrote:
> In this case the documentation must state that the functions uuid_extract_time() and uuidv7(T) are against the RFC requirements, and that developers may use these functions with caution at their own risk, and these functions are not recommended for production environment.
>
> The function uuidv7(T) is not better than uuid_extract_time(). Careless developers may well pass any business date into this function: document date, registration date, payment date, reporting date, start date of the current month, data download date, and even a constant. This would be a profanation of UUIDv7 with very negative consequences.
After re-reading the RFC more diligently, I'm inclined to agree with
Sergey that uuidv7(timestamp) is quite problematic. And I would even
say that we should not provide uuidv7(timestamp) at all, and instead
should only provide uuidv7(). Providing an explicit timestamp for
UUIDv7 is explicitly against the spec (in my reading):
> Implementations acquire the current timestamp from a reliable
> source to provide values that are time-ordered and continually
> increasing. Care must be taken to ensure that timestamp changes
> from the environment or operating system are handled in a way that
> is consistent with implementation requirements. For example, if
> it is possible for the system clock to move backward due to either
> manual adjustment or corrections from a time synchronization
> protocol, implementations need to determine how to handle such
> cases. (See Altering, Fuzzing, or Smearing below.)
>
> ...
>
> UUID version 1 and 6 both utilize a Gregorian epoch timestamp
> while UUIDv7 utilizes a Unix Epoch timestamp. If other timestamp
> sources or a custom timestamp epoch are required, UUIDv8 MUST be
> used.
>
> ...
>
> Monotonicity (each subsequent value being greater than the last) is
> the backbone of time-based sortable UUIDs.
By allowing users to provide a timestamp we're not using a continually
increasing timestamp for our UUIDv7 generation, and thus it would not
be a valid UUIDv7 implementation.
I do agree with others however, that being able to pass in an
arbitrary timestamp for UUID generation would be very useful. For
example to be able to partition by the timestamp in the UUID and then
being able to later load data for an older timestamp and have it be
added to to the older partition. But it's possible to do that while
still following the spec, by using a UUIDv8 instead of UUIDv7. So for
this usecase we could make a helper function that generates a UUIDv8
using the same format as a UUIDv7, but allows storing arbitrary
timestamps. You might say, why not sligthly change UUIDv7 then? Well
mainly because of this critical sentence in the RFC:
> UUIDv8's uniqueness will be implementation-specific and MUST NOT be assumed.
That would allow us to say that using this UUIDv8 helper requires
careful usage and checks if uniqueness is required.
So I believe we should remove the uuidv7(timestamp) function from this patchset.
I don't see a problem with including uuid_extract_time though. Afaict
the only thing the RFC says about extracting timestamps is that the
RFC does not give a requirement or guarantee about how close the
stored timestamp is to the actual time:
> Implementations MAY alter the actual timestamp. Some examples
> include security considerations around providing a real clock
> value within a UUID, to correct inaccurate clocks, to handle leap
> seconds, or instead of dividing a number of microseconds by 1000
> to obtain a millisecond value; dividing by 1024 (or some other
> value) for performance reasons. This specification makes no
> requirement or guarantee about how close the clock value needs to
> be to the actual time.
I see no reason why we cannot make stronger guarantees about the
timestamps that we use to generate UUIDs with our uuidv7() function.
And then we can update the documentation for
uuid_extract_time to something like this:
> This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
> versions and variants this function returns NULL. The extracted timestamp
> does not necessarily equate to the time of UUID generation. How close it is
> to the actual time depends on the implementation that generated to UUID.
> The uuidv7() function provided PostgreSQL will normally store the actual time of
> generation to in the UUID, but if large batches of UUIDs are generated at the
> same time it's possible that some UUIDs will store a time that is slightly later
> than their actual generation time.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-29 12:01 Jelte Fennema-Nio <[email protected]>
parent: Aleksander Alekseev <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Jelte Fennema-Nio @ 2024-01-29 12:01 UTC (permalink / raw)
To: Aleksander Alekseev <[email protected]>; +Cc: pgsql-hackers; Andrey M. Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; Sergey Prokhorenko <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>; Alexander Lakhin <[email protected]>
On Thu, 25 Jan 2024 at 13:31, Aleksander Alekseev
<[email protected]> wrote:
> PFA v14.
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ Both functions return a version 4 (random) UUID. This is the most commonly
+ used type of UUID and is appropriate when random distribution of keys does
+ not affect performance of an application.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 (time-ordered + random) UUID. This UUID
+ version should be used when application prefers locality of identifiers.
+<synopsis>
I think it would be good to explain the tradeoffs between uuidv4 and
uuidv7 a bit better. How about changing the docs to something like
this:
<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
</synopsis>
Both functions return a version 4 (random) UUID. UUIDv4 is one of the
most commonly used types of UUID. It is appropriate when random
distribution of keys does not affect performance of an application or
when exposing the generation time of a UUID has unacceptable security
or business intelligence implications.
<synopsis>
<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
</synopsis>
This function returns a version 7 (time-ordered + random) UUID. It
provides much better data locality than UUIDv4, which can greatly
improve performance when UUID is used in a BTREE index (the default
index type in PostgreSQL). To achieve this data locality, UUIDv7
embeds its own generation time into the UUID. If exposing such a
timestamp has unacceptable security or business intelligence
implications, then uuidv4() should be used instead.
<synopsis>
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-29 13:58 Junwang Zhao <[email protected]>
parent: Jelte Fennema-Nio <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Junwang Zhao @ 2024-01-29 13:58 UTC (permalink / raw)
To: Jelte Fennema-Nio <[email protected]>; +Cc: Sergey Prokhorenko <[email protected]>; pgsql-hackers; Aleksander Alekseev <[email protected]>; Andrey M. Borodin <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Mon, Jan 29, 2024 at 7:38 PM Jelte Fennema-Nio <[email protected]> wrote:
>
> tl;dr I believe we should remove the uuidv7(timestamp) function from
> this patchset.
>
> On Thu, 25 Jan 2024 at 18:04, Sergey Prokhorenko
> <[email protected]> wrote:
> > In this case the documentation must state that the functions uuid_extract_time() and uuidv7(T) are against the RFC requirements, and that developers may use these functions with caution at their own risk, and these functions are not recommended for production environment.
> >
> > The function uuidv7(T) is not better than uuid_extract_time(). Careless developers may well pass any business date into this function: document date, registration date, payment date, reporting date, start date of the current month, data download date, and even a constant. This would be a profanation of UUIDv7 with very negative consequences.
>
> After re-reading the RFC more diligently, I'm inclined to agree with
> Sergey that uuidv7(timestamp) is quite problematic. And I would even
> say that we should not provide uuidv7(timestamp) at all, and instead
> should only provide uuidv7(). Providing an explicit timestamp for
> UUIDv7 is explicitly against the spec (in my reading):
>
> > Implementations acquire the current timestamp from a reliable
> > source to provide values that are time-ordered and continually
> > increasing. Care must be taken to ensure that timestamp changes
> > from the environment or operating system are handled in a way that
> > is consistent with implementation requirements. For example, if
> > it is possible for the system clock to move backward due to either
> > manual adjustment or corrections from a time synchronization
> > protocol, implementations need to determine how to handle such
> > cases. (See Altering, Fuzzing, or Smearing below.)
> >
> > ...
> >
> > UUID version 1 and 6 both utilize a Gregorian epoch timestamp
> > while UUIDv7 utilizes a Unix Epoch timestamp. If other timestamp
> > sources or a custom timestamp epoch are required, UUIDv8 MUST be
> > used.
> >
> > ...
> >
> > Monotonicity (each subsequent value being greater than the last) is
> > the backbone of time-based sortable UUIDs.
>
> By allowing users to provide a timestamp we're not using a continually
> increasing timestamp for our UUIDv7 generation, and thus it would not
> be a valid UUIDv7 implementation.
>
> I do agree with others however, that being able to pass in an
> arbitrary timestamp for UUID generation would be very useful. For
> example to be able to partition by the timestamp in the UUID and then
> being able to later load data for an older timestamp and have it be
> added to to the older partition. But it's possible to do that while
> still following the spec, by using a UUIDv8 instead of UUIDv7. So for
> this usecase we could make a helper function that generates a UUIDv8
> using the same format as a UUIDv7, but allows storing arbitrary
> timestamps. You might say, why not sligthly change UUIDv7 then? Well
> mainly because of this critical sentence in the RFC:
>
> > UUIDv8's uniqueness will be implementation-specific and MUST NOT be assumed.
>
> That would allow us to say that using this UUIDv8 helper requires
> careful usage and checks if uniqueness is required.
>
> So I believe we should remove the uuidv7(timestamp) function from this patchset.
Agreed, the RFC section 6.1[1] has the following statements:
```
UUID version 1 and 6 both utilize a Gregorian epoch timestamp while
UUIDv7 utilizes a Unix Epoch timestamp. If other timestamp sources or
a custom timestamp epoch are required, UUIDv8 MUST be used.
```
In contrib/uuid-ossp, uuidv1 does not allow the user to supply a
custom timestamp,
so I think it should be the same for uuidv6 and uuidv7.
And I have the same feeling that we should not consider v6 and v8 in
this patch.
[1]: https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-14#section-6.1-2.4.1
>
> I don't see a problem with including uuid_extract_time though. Afaict
> the only thing the RFC says about extracting timestamps is that the
> RFC does not give a requirement or guarantee about how close the
> stored timestamp is to the actual time:
>
> > Implementations MAY alter the actual timestamp. Some examples
> > include security considerations around providing a real clock
> > value within a UUID, to correct inaccurate clocks, to handle leap
> > seconds, or instead of dividing a number of microseconds by 1000
> > to obtain a millisecond value; dividing by 1024 (or some other
> > value) for performance reasons. This specification makes no
> > requirement or guarantee about how close the clock value needs to
> > be to the actual time.
>
> I see no reason why we cannot make stronger guarantees about the
> timestamps that we use to generate UUIDs with our uuidv7() function.
> And then we can update the documentation for
> uuid_extract_time to something like this:
>
> > This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
> > versions and variants this function returns NULL. The extracted timestamp
> > does not necessarily equate to the time of UUID generation. How close it is
> > to the actual time depends on the implementation that generated to UUID.
> > The uuidv7() function provided PostgreSQL will normally store the actual time of
> > generation to in the UUID, but if large batches of UUIDs are generated at the
> > same time it's possible that some UUIDs will store a time that is slightly later
> > than their actual generation time.
>
>
--
Regards
Junwang Zhao
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-29 18:32 Andrey M. Borodin <[email protected]>
parent: Sergey Prokhorenko <[email protected]>
1 sibling, 2 replies; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-29 18:32 UTC (permalink / raw)
To: Sergey Prokhorenko <[email protected]>; +Cc: pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 25 Jan 2024, at 22:04, Sergey Prokhorenko <[email protected]> wrote:
>
> Aleksander,
>
> In this case the documentation must state that the functions uuid_extract_time() and uuidv7(T) are against the RFC requirements, and that developers may use these functions with caution at their own risk, and these functions are not recommended for production environment.
Refining documentation is good. However, saying that these functions are not recommended for production must be based on some real threats.
>
> The function uuidv7(T) is not better than uuid_extract_time(). Careless developers may well pass any business date into this function: document date, registration date, payment date, reporting date, start date of the current month, data download date, and even a constant. This would be a profanation of UUIDv7 with very negative consequences.
Even if the developer pass constant time to uuidv7(T) they will get what they asked for - unique identifier. Moreover - it still will be keeping locality. There will be no negative consequences at all.
On the contrary, experienced developer can leverage parameter when data locality should be reduced. If you have serveral streams of data, you might want to introduce some shift in reduce contention.
For example, you can generate uuidv7(now() + '1 day' * random(0,10)). This will split 1 contention point to 10 and increase ingestion performance 10x-fold.
> On 29 Jan 2024, at 18:58, Junwang Zhao <[email protected]> wrote:
>
> If other timestamp sources or
> a custom timestamp epoch are required, UUIDv8 MUST be used.
Well, yeah. RFC says this... in 4 capital letters :) I believe it's kind of a big deficiency that k-way sortable identifiers are not implementable on top of UUIDv7. Well, let's go without this function. UUIDv7 is still an improvement over previous versions.
Jelte, your documentation corrections looks good to me, I'll include them in next version.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-29 20:38 Jelte Fennema-Nio <[email protected]>
parent: Andrey M. Borodin <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Jelte Fennema-Nio @ 2024-01-29 20:38 UTC (permalink / raw)
To: Andrey M. Borodin <[email protected]>; +Cc: Sergey Prokhorenko <[email protected]>; pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
On Mon, 29 Jan 2024 at 19:32, Andrey M. Borodin <[email protected]> wrote:
> Even if the developer pass constant time to uuidv7(T) they will get what they asked for - unique identifier. Moreover - it still will be keeping locality. There will be no negative consequences at all.
It will be significantly "less unique" than if they wouldn't pass a
constant time. Basically it would become a UUIDv4, but with 74 bits of
random data instead of 122. That might not be enough anymore to
"guarantee" uniqueness. I guess that's why it is required to use
UUIDv8 in these cases, because correct usage is now a requirement for
assuming uniqueness. And for UUIDv8 the spec says this:
> UUIDv8's uniqueness will be implementation-specific and MUST NOT be assumed.
> > On 29 Jan 2024, at 18:58, Junwang Zhao <[email protected]> wrote:
> >
> > If other timestamp sources or
> > a custom timestamp epoch are required, UUIDv8 MUST be used.
>
> Well, yeah. RFC says this... in 4 capital letters :)
As an FYI, there is an RFC that defines these keywords that's why they
are capital letters: https://www.ietf.org/rfc/rfc2119.txt
> I believe it's kind of a big deficiency that k-way sortable identifiers are not implementable on top of UUIDv7. Well, let's go without this function. UUIDv7 is still an improvement over previous versions.
Yeah, I liked the feature to generate UUIDv7 based on timestamp too.
But following the spec seems more important than a nice feature to me.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-30 00:27 Sergey Prokhorenko <[email protected]>
parent: Andrey M. Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-30 00:27 UTC (permalink / raw)
To: Andrey M. Borodin <[email protected]>; +Cc: pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Jelte Fennema-Nio <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Andrey,
I understand and agree with your goals. But instead of dangerous universal functions, it is better to develop safe highly specialized functions that implement only these goals.
There should not be a function uuidv7(T) from an arbitrary timestamp, but there should be a special function that implements your algorithm: uuidv8(now() + '1 century' * random(0,10)).
I replaced 1 day with 1 century because the spread of 1 day is too small. Over time, records will be inserted between existing records, which is undesirable.
Similarly, if we need to calculate the partition id, then we do not need to use the uuid_extract_time() function to provide the extracted timestamp, the accuracy of which cannot be guaranteed. Instead, we need to give exactly the partition id, calculated using the uuidv7 timestamp. For example, partitions may have approximately a month interval between each other.
As for the documentation, it must be indicated that the UUIDv7 structure is not timestamp + random, but timestamp + randomly seeded counter + random, like in all advanced implementations.
Sergey Prokhorenko
[email protected]
______________________________________________________________
On Monday, 29 January 2024 at 09:32:54 pm GMT+3, Andrey M. Borodin <[email protected]> wrote:
> On 25 Jan 2024, at 22:04, Sergey Prokhorenko <[email protected]> wrote:
>
> Aleksander,
>
> In this case the documentation must state that the functions uuid_extract_time() and uuidv7(T) are against the RFC requirements, and that developers may use these functions with caution at their own risk, and these functions are not recommended for production environment.
Refining documentation is good. However, saying that these functions are not recommended for production must be based on some real threats.
>
> The function uuidv7(T) is not better than uuid_extract_time(). Careless developers may well pass any business date into this function: document date, registration date, payment date, reporting date, start date of the current month, data download date, and even a constant. This would be a profanation of UUIDv7 with very negative consequences.
Even if the developer pass constant time to uuidv7(T) they will get what they asked for - unique identifier. Moreover - it still will be keeping locality. There will be no negative consequences at all.
On the contrary, experienced developer can leverage parameter when data locality should be reduced. If you have serveral streams of data, you might want to introduce some shift in reduce contention.
For example, you can generate uuidv7(now() + '1 day' * random(0,10)). This will split 1 contention point to 10 and increase ingestion performance 10x-fold.
> On 29 Jan 2024, at 18:58, Junwang Zhao <[email protected]> wrote:
>
> If other timestamp sources or
> a custom timestamp epoch are required, UUIDv8 MUST be used.
Well, yeah. RFC says this... in 4 capital letters :) I believe it's kind of a big deficiency that k-way sortable identifiers are not implementable on top of UUIDv7. Well, let's go without this function. UUIDv7 is still an improvement over previous versions.
Jelte, your documentation corrections looks good to me, I'll include them in next version.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-30 06:54 Andrey M. Borodin <[email protected]>
parent: Jelte Fennema-Nio <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-30 06:54 UTC (permalink / raw)
To: Jelte Fennema-Nio <[email protected]>; +Cc: Sergey Prokhorenko <[email protected]>; pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 30 Jan 2024, at 01:38, Jelte Fennema-Nio <[email protected]> wrote:
>
> Yeah, I liked the feature to generate UUIDv7 based on timestamp too.
> But following the spec seems more important than a nice feature to me.
PFA v15. Changes: removed timestamp argument, incorporated Jelte’s documentation addons.
Thanks!
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v15-0001-Implement-UUID-v7.patch (16.7K, ../../[email protected]/2-v15-0001-Implement-UUID-v7.patch)
download | inline diff:
From 9dde2582821354d4638c822e8d77eee6ff60cfa5 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v15] Implement UUID v7
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This commit adds function for UUID generation. Most important function here
is uuidv7() which generates new UUID according to the new standard.
This function can optionally accept a timestamp used instead of current time.
This allows implementation of k-way sotable identifiers. For code readability
this commit adds alias uuidv4() to function gen_random_uuid().
Also we add a function to extract timestamp from UUID v1, v6 and v7.
To allow user to distinguish various UUID versions and variants
we add functions uuid_extract_ver() and uuid_extract_var().
Author: Andrey Borodin
Reviewers: Sergey Prokhorenko, Kirk Wolak, Przemysław Sztoch
Reviewers: Nikolay Samokhvalov, Jelte Fennema-Nio, Aleksander Alekseev
Reviewers: Peter Eisentraut, Chris Travers, Lukas Fittl
Discussion: https://postgr.es/m/CAAhFRxitJv%3DyoGnXUgeLB_O%2BM7J2BJAmb5jqAT9gZ3bij3uLDA%40mail.gmail.com
---
doc/src/sgml/func.sgml | 62 +++++++-
src/backend/utils/adt/uuid.c | 179 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 15 ++
src/include/datatype/timestamp.h | 3 +-
src/test/regress/expected/opr_sanity.out | 5 +
src/test/regress/expected/uuid.out | 71 +++++++++
src/test/regress/sql/uuid.sql | 26 ++++
7 files changed, 357 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 6788ba8ef4a..588dd1ffd34 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14128,13 +14128,69 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv4</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_ver</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_var</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes several functions to generate a UUID:
+ <function>gen_random_uuid</function>, <function>uuidv4</function>, and <function>uuidv7</function>.
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ Both functions return a version 4 (random) UUID. UUIDv4 is one of the
+ most commonly used types of UUID. It is appropriate when random
+ distribution of keys does not affect performance of an application or
+ when exposing the generation time of a UUID has unacceptable security
+ or business intelligence implications.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 UUID (UNIX timestamp with 1ms precision +
+ randomly seeded counter + random). It provides much better data locality
+ than UUIDv4, which can greatly improve performance when UUID is used in a
+ B-tree index (the default index type in PostgreSQL). To achieve this data
+ locality, UUIDv7 embeds its own generation time into the UUID. If exposing
+ such a timestamp has unacceptable security or business intelligence
+ implications, then uuidv4() should be used instead.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
+ versions and variants this function returns NULL. The extracted timestamp
+ does not necessarily equate to the time of UUID generation. How close it is
+ to the actual time depends on the implementation that generated to UUID.
+ The uuidv7() function provided by PostgreSQL will normally store the actual time of
+ generation to in the UUID, but if large batches of UUIDs are generated at the
+ same time it's possible that some UUIDs will store a time that is slightly later
+ than their actual generation time.
+<synopsis>
+<function>uuid_extract_ver</function> (uuid) <returnvalue>int2</returnvalue>
+</synopsis>
+ This function extracts a version bits from UUID of variant described by
+ <ulink url="https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis">IETF standard</ulink>
+ (b10xx variant). For other variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_var</function> (uuid) <returnvalue>int2</returnvalue>
</synopsis>
- This function returns a version 4 (random) UUID. This is the most commonly
- used type of UUID and is appropriate for most applications.
+ This function extracts a vartiant bits from UUID.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c73..ef14d6cba27 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,177 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = (tms <= previous_timestamp);
+
+ if (increment_counter)
+ {
+ /* Time did not advance from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ /* read randomly initialized bits of counter */
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 29af4ce65d5..f9be09464be 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,21 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9898', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9899', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 3a37cb661e3..652aeb428e2 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -230,9 +230,10 @@ struct pg_itm_in
((y) < JULIAN_MAXYEAR || \
((y) == JULIAN_MAXYEAR && ((m) < JULIAN_MAXMONTH))))
-/* Julian-date equivalents of Day 0 in Unix and Postgres reckoning */
+/* Julian-date equivalents of Day 0 in Unix, Postgres and Gregorian epochs */
#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define GREGORIAN_EPOCH_JDATE 2299161 /* == date2j(1582,10,15) */
/*
* Range limits for dates and timestamps.
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d68..f4b9ff654ab 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -872,6 +872,11 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d6..f401a550885 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,76 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d2..c7362cf4e13 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,31 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.42.0
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-30 07:28 Sergey Prokhorenko <[email protected]>
parent: Andrey M. Borodin <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-30 07:28 UTC (permalink / raw)
To: Jelte Fennema-Nio <[email protected]>; Andrey M. Borodin <[email protected]>; +Cc: pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Andrey,
I think this phrase is outdated: "This function can optionally accept a timestamp used instead of current time.This allows implementation of k-way sotable identifiers."
This phrase is wrong: "Both functions return a version 4 (random) UUID."
For this phrase the reason is unclear and the phrase is most likely incorrect:
if large batches of UUIDs are generated at the+ same time it's possible that some UUIDs will store a time that is slightly later+ than their actual generation time
Sergey Prokhorenko
[email protected]
On Tuesday, 30 January 2024 at 09:55:04 am GMT+3, Andrey M. Borodin <[email protected]> wrote:
> On 30 Jan 2024, at 01:38, Jelte Fennema-Nio <[email protected]> wrote:
>
> Yeah, I liked the feature to generate UUIDv7 based on timestamp too.
> But following the spec seems more important than a nice feature to me.
PFA v15. Changes: removed timestamp argument, incorporated Jelte’s documentation addons.
Thanks!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-30 09:56 Andrey M. Borodin <[email protected]>
parent: Sergey Prokhorenko <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-30 09:56 UTC (permalink / raw)
To: Sergey Prokhorenko <[email protected]>; +Cc: Jelte Fennema-Nio <[email protected]>; pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 30 Jan 2024, at 12:28, Sergey Prokhorenko <[email protected]> wrote:
>
>
> I think this phrase is outdated: "This function can optionally accept a timestamp used instead of current time.
> This allows implementation of k-way sotable identifiers.”
Fixed.
> This phrase is wrong: "Both functions return a version 4 (random) UUID.”
This applies to functions gen_random_uuid() and uuidv4().
>
> For this phrase the reason is unclear and the phrase is most likely incorrect:
> if large batches of UUIDs are generated at the
> + same time it's possible that some UUIDs will store a time that is slightly later
> + than their actual generation time
I’ve rewritten this phrase, hope it’s more clear now.
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v16-0001-Implement-UUID-v7.patch (16.5K, ../../[email protected]/2-v16-0001-Implement-UUID-v7.patch)
download | inline diff:
From 8d07eec66f93adf1cfa2512f5b89f41fe558dfe6 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v16] Implement UUID v7
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This commit adds function for UUID generation. Most important function here
is uuidv7() which generates new UUID according to the new standard.
For code readability this commit adds alias uuidv4() to function gen_random_uuid().
Also we add a function to extract timestamp from UUID v1, v6 and v7.
To allow user to distinguish various UUID versions and variants
we add functions uuid_extract_ver() and uuid_extract_var().
Author: Andrey Borodin
Reviewers: Sergey Prokhorenko, Kirk Wolak, Przemysław Sztoch
Reviewers: Nikolay Samokhvalov, Jelte Fennema-Nio, Aleksander Alekseev
Reviewers: Peter Eisentraut, Chris Travers, Lukas Fittl
Discussion: https://postgr.es/m/CAAhFRxitJv%3DyoGnXUgeLB_O%2BM7J2BJAmb5jqAT9gZ3bij3uLDA%40mail.gmail.com
---
doc/src/sgml/func.sgml | 61 +++++++-
src/backend/utils/adt/uuid.c | 179 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 15 ++
src/include/datatype/timestamp.h | 3 +-
src/test/regress/expected/opr_sanity.out | 5 +
src/test/regress/expected/uuid.out | 71 +++++++++
src/test/regress/sql/uuid.sql | 26 ++++
7 files changed, 356 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 6788ba8ef4a..97abf7f4c69 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14128,13 +14128,68 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv4</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_ver</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_var</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes several functions to generate a UUID:
+ <function>gen_random_uuid</function>, <function>uuidv4</function>, and <function>uuidv7</function>.
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ These functions return a version 4 (random) UUID. UUIDv4 is one of the
+ most commonly used types of UUID. It is appropriate when random
+ distribution of keys does not affect performance of an application or
+ when exposing the generation time of a UUID has unacceptable security
+ or business intelligence implications.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 UUID (UNIX timestamp with 1ms precision +
+ randomly seeded counter + random). It provides much better data locality
+ than UUIDv4, which can greatly improve performance when UUID is used in a
+ B-tree index (the default index type in PostgreSQL). To achieve this data
+ locality, UUIDv7 embeds its own generation time into the UUID. If exposing
+ such a timestamp has unacceptable security or business intelligence
+ implications, then uuidv4() should be used instead.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
+ versions and variants this function returns NULL. The extracted timestamp
+ does not necessarily equate to the time of UUID generation. How close it is
+ to the actual time depends on the implementation that generated to UUID.
+ The uuidv7() function provided by PostgreSQL will normally store the actual time,
+ with some exceptions: prevention of time leaps backwards and counter overflow
+ being carried to time step.
+<synopsis>
+<function>uuid_extract_ver</function> (uuid) <returnvalue>int2</returnvalue>
+</synopsis>
+ This function extracts a version bits from UUID of variant described by
+ <ulink url="https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis">IETF standard</ulink>
+ (b10xx variant). For other variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_var</function> (uuid) <returnvalue>int2</returnvalue>
</synopsis>
- This function returns a version 4 (random) UUID. This is the most commonly
- used type of UUID and is appropriate for most applications.
+ This function extracts a vartiant bits from UUID.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c73..ef14d6cba27 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,177 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ TimestampTz ts;
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = (tms <= previous_timestamp);
+
+ if (increment_counter)
+ {
+ /* Time did not advance from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ /* read randomly initialized bits of counter */
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 29af4ce65d5..f9be09464be 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,21 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9898', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9899', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 3a37cb661e3..652aeb428e2 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -230,9 +230,10 @@ struct pg_itm_in
((y) < JULIAN_MAXYEAR || \
((y) == JULIAN_MAXYEAR && ((m) < JULIAN_MAXMONTH))))
-/* Julian-date equivalents of Day 0 in Unix and Postgres reckoning */
+/* Julian-date equivalents of Day 0 in Unix, Postgres and Gregorian epochs */
#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define GREGORIAN_EPOCH_JDATE 2299161 /* == date2j(1582,10,15) */
/*
* Range limits for dates and timestamps.
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d68..f4b9ff654ab 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -872,6 +872,11 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d6..f401a550885 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,76 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d2..c7362cf4e13 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,31 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.42.0
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-30 10:33 Junwang Zhao <[email protected]>
parent: Andrey M. Borodin <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Junwang Zhao @ 2024-01-30 10:33 UTC (permalink / raw)
To: Andrey M. Borodin <[email protected]>; +Cc: Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
Hi Andrey,
On Tue, Jan 30, 2024 at 5:56 PM Andrey M. Borodin <[email protected]> wrote:
>
>
>
> > On 30 Jan 2024, at 12:28, Sergey Prokhorenko <[email protected]> wrote:
> >
> >
> > I think this phrase is outdated: "This function can optionally accept a timestamp used instead of current time.
> > This allows implementation of k-way sotable identifiers.”
> Fixed.
>
> > This phrase is wrong: "Both functions return a version 4 (random) UUID.”
> This applies to functions gen_random_uuid() and uuidv4().
> >
> > For this phrase the reason is unclear and the phrase is most likely incorrect:
> > if large batches of UUIDs are generated at the
> > + same time it's possible that some UUIDs will store a time that is slightly later
> > + than their actual generation time
>
> I’ve rewritten this phrase, hope it’s more clear now.
>
>
> Best regards, Andrey Borodin.
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
\ No newline at end of file
It's always good to add a newline at the end of a source file, though
this might be nitpicky.
--
Regards
Junwang Zhao
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-30 13:35 Andrey M. Borodin <[email protected]>
parent: Junwang Zhao <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Andrey M. Borodin @ 2024-01-30 13:35 UTC (permalink / raw)
To: Junwang Zhao <[email protected]>; +Cc: Sergey Prokhorenko <[email protected]>; Jelte Fennema-Nio <[email protected]>; pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
> On 30 Jan 2024, at 15:33, Junwang Zhao <[email protected]> wrote:
>
> It's always good to add a newline at the end of a source file, though
> this might be nitpicky.
Thanks, also fixed warning found by CFBot.
Best regards, Andrey Borodin.
Attachments:
[application/octet-stream] v17-0001-Implement-UUID-v7.patch (16.5K, ../../[email protected]/2-v17-0001-Implement-UUID-v7.patch)
download | inline diff:
From 148750ca11235bc24ef07ceb549b910ba2a862c2 Mon Sep 17 00:00:00 2001
From: "Andrey M. Borodin" <[email protected]>
Date: Sun, 20 Aug 2023 23:55:31 +0300
Subject: [PATCH v17] Implement UUID v7
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit
This commit adds function for UUID generation. Most important function here
is uuidv7() which generates new UUID according to the new standard.
For code readability this commit adds alias uuidv4() to function gen_random_uuid().
Also we add a function to extract timestamp from UUID v1, v6 and v7.
To allow user to distinguish various UUID versions and variants
we add functions uuid_extract_ver() and uuid_extract_var().
Author: Andrey Borodin
Reviewers: Sergey Prokhorenko, Kirk Wolak, Przemysław Sztoch
Reviewers: Nikolay Samokhvalov, Jelte Fennema-Nio, Aleksander Alekseev
Reviewers: Peter Eisentraut, Chris Travers, Lukas Fittl
Discussion: https://postgr.es/m/CAAhFRxitJv%3DyoGnXUgeLB_O%2BM7J2BJAmb5jqAT9gZ3bij3uLDA%40mail.gmail.com
---
doc/src/sgml/func.sgml | 61 +++++++-
src/backend/utils/adt/uuid.c | 178 +++++++++++++++++++++++
src/include/catalog/pg_proc.dat | 15 ++
src/include/datatype/timestamp.h | 3 +-
src/test/regress/expected/opr_sanity.out | 5 +
src/test/regress/expected/uuid.out | 71 +++++++++
src/test/regress/sql/uuid.sql | 26 ++++
7 files changed, 355 insertions(+), 4 deletions(-)
diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml
index 6788ba8ef4a..97abf7f4c69 100644
--- a/doc/src/sgml/func.sgml
+++ b/doc/src/sgml/func.sgml
@@ -14128,13 +14128,68 @@ CREATE TYPE rainbow AS ENUM ('red', 'orange', 'yellow', 'green', 'blue', 'purple
<primary>gen_random_uuid</primary>
</indexterm>
+ <indexterm>
+ <primary>uuidv4</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuidv7</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_time</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_ver</primary>
+ </indexterm>
+
+ <indexterm>
+ <primary>uuid_extract_var</primary>
+ </indexterm>
+
<para>
- <productname>PostgreSQL</productname> includes one function to generate a UUID:
+ <productname>PostgreSQL</productname> includes several functions to generate a UUID:
+ <function>gen_random_uuid</function>, <function>uuidv4</function>, and <function>uuidv7</function>.
<synopsis>
<function>gen_random_uuid</function> () <returnvalue>uuid</returnvalue>
+<function>uuidv4</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ These functions return a version 4 (random) UUID. UUIDv4 is one of the
+ most commonly used types of UUID. It is appropriate when random
+ distribution of keys does not affect performance of an application or
+ when exposing the generation time of a UUID has unacceptable security
+ or business intelligence implications.
+<synopsis>
+<function>uuidv7</function> () <returnvalue>uuid</returnvalue>
+</synopsis>
+ This function returns a version 7 UUID (UNIX timestamp with 1ms precision +
+ randomly seeded counter + random). It provides much better data locality
+ than UUIDv4, which can greatly improve performance when UUID is used in a
+ B-tree index (the default index type in PostgreSQL). To achieve this data
+ locality, UUIDv7 embeds its own generation time into the UUID. If exposing
+ such a timestamp has unacceptable security or business intelligence
+ implications, then uuidv4() should be used instead.
+<synopsis>
+<function>uuid_extract_time</function> (uuid) <returnvalue>timestamptz</returnvalue>
+</synopsis>
+ This function extracts a timestamptz from UUID versions 1, 6 and 7. For other
+ versions and variants this function returns NULL. The extracted timestamp
+ does not necessarily equate to the time of UUID generation. How close it is
+ to the actual time depends on the implementation that generated to UUID.
+ The uuidv7() function provided by PostgreSQL will normally store the actual time,
+ with some exceptions: prevention of time leaps backwards and counter overflow
+ being carried to time step.
+<synopsis>
+<function>uuid_extract_ver</function> (uuid) <returnvalue>int2</returnvalue>
+</synopsis>
+ This function extracts a version bits from UUID of variant described by
+ <ulink url="https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis">IETF standard</ulink>
+ (b10xx variant). For other variants this function returns NULL.
+<synopsis>
+<function>uuid_extract_var</function> (uuid) <returnvalue>int2</returnvalue>
</synopsis>
- This function returns a version 4 (random) UUID. This is the most commonly
- used type of UUID and is appropriate for most applications.
+ This function extracts a vartiant bits from UUID.
</para>
<para>
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 73dfd711c73..a157f69c2b7 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -13,13 +13,18 @@
#include "postgres.h"
+#include <sys/time.h>
+
+#include "access/xlog.h"
#include "common/hashfn.h"
#include "lib/hyperloglog.h"
#include "libpq/pqformat.h"
#include "port/pg_bswap.h"
#include "utils/builtins.h"
+#include "utils/datetime.h"
#include "utils/guc.h"
#include "utils/sortsupport.h"
+#include "utils/timestamp.h"
#include "utils/uuid.h"
/* sortsupport for uuid */
@@ -421,3 +426,176 @@ gen_random_uuid(PG_FUNCTION_ARGS)
PG_RETURN_UUID_P(uuid);
}
+
+static uint32_t sequence_counter;
+static uint64_t previous_timestamp = 0;
+
+
+Datum
+uuidv7(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = palloc(UUID_LEN);
+ uint64_t tms;
+ struct timeval tp;
+ bool increment_counter;
+
+ gettimeofday(&tp, NULL);
+ tms = ((uint64_t)tp.tv_sec) * 1000 + (tp.tv_usec) / 1000;
+ /* time from clock is protected from backward leaps */
+ increment_counter = (tms <= previous_timestamp);
+
+ if (increment_counter)
+ {
+ /* Time did not advance from the previous generation, we must increment counter */
+ ++sequence_counter;
+ if (sequence_counter > 0x3ffff)
+ {
+ /* We only have 18-bit counter */
+ sequence_counter = 0;
+ previous_timestamp++;
+ }
+
+ /* protection from leap backward */
+ tms = previous_timestamp;
+
+ /* fill everything after the timestamp and counter with random bytes */
+ if (!pg_strong_random(&uuid->data[8], UUID_LEN - 8))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /* most significant 4 bits of 18-bit counter */
+ uuid->data[6] = (unsigned char)(sequence_counter >> 14);
+ /* next 8 bits */
+ uuid->data[7] = (unsigned char)(sequence_counter >> 6);
+ /* least significant 6 bits */
+ uuid->data[8] = (unsigned char)(sequence_counter);
+ }
+ else
+ {
+ /* fill everything after the timestamp with random bytes */
+ if (!pg_strong_random(&uuid->data[6], UUID_LEN - 6))
+ ereport(ERROR,
+ (errcode(ERRCODE_INTERNAL_ERROR),
+ errmsg("could not generate random values")));
+
+ /*
+ * Left-most counter bits are initialized as zero for the sole purpose
+ * of guarding against counter rollovers.
+ * See section "Fixed-Length Dedicated Counter Seeding"
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis-09#monotonicity_counters
+ */
+ uuid->data[6] = (uuid->data[6] & 0xf7);
+
+ /* read randomly initialized bits of counter */
+ sequence_counter = ((uint32_t)uuid->data[8] & 0x3f) +
+ (((uint32_t)uuid->data[7]) << 6) +
+ (((uint32_t)uuid->data[6] & 0x0f) << 14);
+
+ previous_timestamp = tms;
+ }
+
+ /* Fill in time part */
+ uuid->data[0] = (unsigned char)(tms >> 40);
+ uuid->data[1] = (unsigned char)(tms >> 32);
+ uuid->data[2] = (unsigned char)(tms >> 24);
+ uuid->data[3] = (unsigned char)(tms >> 16);
+ uuid->data[4] = (unsigned char)(tms >> 8);
+ uuid->data[5] = (unsigned char)tms;
+
+ /*
+ * Set magic numbers for a "version 7" (pseudorandom) UUID, see
+ * https://datatracker.ietf.org/doc/html/draft-ietf-uuidrev-rfc4122bis
+ */
+ /* set version field, top four bits are 0, 1, 1, 1 */
+ uuid->data[6] = (uuid->data[6] & 0x0f) | 0x70;
+ /* set variant field, top two bits are 1, 0 */
+ uuid->data[8] = (uuid->data[8] & 0x3f) | 0x80;
+
+ PG_RETURN_UUID_P(uuid);
+}
+
+Datum
+uuid_extract_time(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ TimestampTz ts;
+ uint64_t tms;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+
+ if ((uuid->data[6] & 0xf0) == 0x70)
+ {
+ tms = uuid->data[5];
+ tms += ((uint64_t)uuid->data[4]) << 8;
+ tms += ((uint64_t)uuid->data[3]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 32;
+ tms += ((uint64_t)uuid->data[0]) << 40;
+
+ ts = (TimestampTz) (tms * 1000) - /* convert ms to us, than adjust */
+ (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x10)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 24;
+ tms += ((uint64_t)uuid->data[1]) << 16;
+ tms += ((uint64_t)uuid->data[2]) << 8;
+ tms += ((uint64_t)uuid->data[3]);
+ tms += ((uint64_t)uuid->data[4]) << 40;
+ tms += ((uint64_t)uuid->data[5]) << 32;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 56;
+ tms += ((uint64_t)uuid->data[7]) << 48;
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ if ((uuid->data[6] & 0xf0) == 0x60)
+ {
+ tms = ((uint64_t)uuid->data[0]) << 52;
+ tms += ((uint64_t)uuid->data[1]) << 44;
+ tms += ((uint64_t)uuid->data[2]) << 36;
+ tms += ((uint64_t)uuid->data[3]) << 28;
+ tms += ((uint64_t)uuid->data[4]) << 20;
+ tms += ((uint64_t)uuid->data[5]) << 12;
+ tms += (((uint64_t)uuid->data[6])&0xf) << 8;
+ tms += ((uint64_t)uuid->data[7]);
+
+ ts = (TimestampTz) (tms / 10) - /* convert 100-ns intervals to us, than adjust */
+ ((uint64_t)POSTGRES_EPOCH_JDATE - GREGORIAN_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
+
+ PG_RETURN_TIMESTAMPTZ(ts);
+ }
+
+ PG_RETURN_NULL();
+}
+
+Datum
+uuid_extract_ver(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+
+ if ((uuid->data[8] & 0xc0) != 0x80)
+ PG_RETURN_NULL();
+ result = uuid->data[6] >> 4;
+
+ PG_RETURN_UINT16(result);
+}
+
+Datum
+uuid_extract_var(PG_FUNCTION_ARGS)
+{
+ pg_uuid_t *uuid = PG_GETARG_UUID_P(0);
+ uint16_t result;
+ result = uuid->data[8] >> 6;
+
+ PG_RETURN_UINT16(result);
+}
diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat
index 29af4ce65d5..f9be09464be 100644
--- a/src/include/catalog/pg_proc.dat
+++ b/src/include/catalog/pg_proc.dat
@@ -9174,6 +9174,21 @@
{ oid => '3432', descr => 'generate random UUID',
proname => 'gen_random_uuid', proleakproof => 't', provolatile => 'v',
prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9895', descr => 'generate random UUID',
+ proname => 'uuidv4', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'gen_random_uuid' },
+{ oid => '9896', descr => 'generate UUID version 7',
+ proname => 'uuidv7', proleakproof => 't', provolatile => 'v',
+ prorettype => 'uuid', proargtypes => '', prosrc => 'uuidv7' },
+{ oid => '9897', descr => 'extract timestamp from UUID version 7',
+ proname => 'uuid_extract_time', proleakproof => 't',
+ prorettype => 'timestamptz', proargtypes => 'uuid', prosrc => 'uuid_extract_time' },
+{ oid => '9898', descr => 'extract version from RFC 4122 UUID',
+ proname => 'uuid_extract_ver', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_ver' },
+{ oid => '9899', descr => 'extract variant from UUID',
+ proname => 'uuid_extract_var', proleakproof => 't',
+ prorettype => 'int2', proargtypes => 'uuid', prosrc => 'uuid_extract_var' },
# pg_lsn
{ oid => '3229', descr => 'I/O',
diff --git a/src/include/datatype/timestamp.h b/src/include/datatype/timestamp.h
index 3a37cb661e3..652aeb428e2 100644
--- a/src/include/datatype/timestamp.h
+++ b/src/include/datatype/timestamp.h
@@ -230,9 +230,10 @@ struct pg_itm_in
((y) < JULIAN_MAXYEAR || \
((y) == JULIAN_MAXYEAR && ((m) < JULIAN_MAXMONTH))))
-/* Julian-date equivalents of Day 0 in Unix and Postgres reckoning */
+/* Julian-date equivalents of Day 0 in Unix, Postgres and Gregorian epochs */
#define UNIX_EPOCH_JDATE 2440588 /* == date2j(1970, 1, 1) */
#define POSTGRES_EPOCH_JDATE 2451545 /* == date2j(2000, 1, 1) */
+#define GREGORIAN_EPOCH_JDATE 2299161 /* == date2j(1582,10,15) */
/*
* Range limits for dates and timestamps.
diff --git a/src/test/regress/expected/opr_sanity.out b/src/test/regress/expected/opr_sanity.out
index 7610b011d68..f4b9ff654ab 100644
--- a/src/test/regress/expected/opr_sanity.out
+++ b/src/test/regress/expected/opr_sanity.out
@@ -872,6 +872,11 @@ xid8ge(xid8,xid8)
xid8eq(xid8,xid8)
xid8ne(xid8,xid8)
xid8cmp(xid8,xid8)
+uuidv4()
+uuidv7()
+uuid_extract_time(uuid)
+uuid_extract_ver(uuid)
+uuid_extract_var(uuid)
-- restore normal output mode
\a\t
-- List of functions used by libpq's fe-lobj.c
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 8e7f21910d6..f401a550885 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -168,5 +168,76 @@ SELECT count(DISTINCT guid_field) FROM guid1;
2
(1 row)
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+ count
+-------
+ 2
+(1 row)
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+ uuid_extract_ver
+------------------
+ 7
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+ uuid_extract_ver
+------------------
+ 5
+(1 row)
+
+SELECT uuid_extract_var(uuidv7());
+ uuid_extract_var
+------------------
+ 2
+(1 row)
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+ uuid_extract_time
+-------------------
+
+(1 row)
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+ ?column?
+----------
+ t
+(1 row)
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 9a8f437c7d2..c7362cf4e13 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -85,5 +85,31 @@ INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
INSERT INTO guid1 (guid_field) VALUES (gen_random_uuid());
SELECT count(DISTINCT guid_field) FROM guid1;
+-- test of uuidv4() alias
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+INSERT INTO guid1 (guid_field) VALUES (uuidv4());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- generation test for v7
+TRUNCATE guid1;
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+INSERT INTO guid1 (guid_field) VALUES (uuidv7());
+SELECT count(DISTINCT guid_field) FROM guid1;
+
+-- support functions for UUID versions and variants
+SELECT uuid_extract_ver(uuidv7());
+SELECT uuid_extract_ver('{11111111-1111-1111-1111-111111111111}') IS NULL;
+SELECT uuid_extract_ver('{11111111-1111-5111-8111-111111111111}');
+SELECT uuid_extract_var(uuidv7());
+
+-- uuid_extract_time() must refuse to accept non-UUIDv7
+SELECT uuid_extract_time(gen_random_uuid());
+
+-- extract UUID v1, v6 and v7 timestamp
+SELECT uuid_extract_time('C232AB00-9414-11EC-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('1EC9414C-232A-6B00-B3C8-9F6BDECED846') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+SELECT uuid_extract_time('017F22E2-79B0-7CC3-98C4-DC0C0C07398F') = 'Tuesday, February 22, 2022 2:22:22.00 PM GMT+05:00';
+
-- clean up
DROP TABLE guid1, guid2 CASCADE;
--
2.42.0
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2024-01-30 18:37 Sergey Prokhorenko <[email protected]>
parent: Andrey M. Borodin <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Sergey Prokhorenko @ 2024-01-30 18:37 UTC (permalink / raw)
To: Junwang Zhao <[email protected]>; Andrey M. Borodin <[email protected]>; +Cc: Jelte Fennema-Nio <[email protected]>; pgsql-hackers; Aleksander Alekseev <[email protected]>; Przemysław Sztoch <[email protected]>; Nikolay Samokhvalov <[email protected]>; David G. Johnston <[email protected]>; Nick Babadzhanian <[email protected]>; Mat Arye <[email protected]>; Peter Eisentraut <[email protected]>; Tom Lane <[email protected]>; Daniel Gustafsson <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Kyzer Davis (kydavis) <[email protected]>; Andres Freund <[email protected]>; [email protected] <[email protected]>; Kirk Wolak <[email protected]>
typo:
being carried to time step
should be:being carried to timestemp
Sergey Prokhorenko [email protected]
On Tuesday, 30 January 2024 at 04:35:45 pm GMT+3, Andrey M. Borodin <[email protected]> wrote:
> On 30 Jan 2024, at 15:33, Junwang Zhao <[email protected]> wrote:
>
> It's always good to add a newline at the end of a source file, though
> this might be nitpicky.
Thanks, also fixed warning found by CFBot.
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2025-03-26 04:32 Masahiko Sawada <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Masahiko Sawada @ 2025-03-26 04:32 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Daniel Verite <[email protected]>; Peter Eisentraut <[email protected]>; Jelte Fennema-Nio <[email protected]>; Sergey Prokhorenko <[email protected]>; Przemysław Sztoch <[email protected]>; Michael Paquier <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; David G. Johnston <[email protected]>; Mat Arye <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Junwang Zhao <[email protected]>; Stepan Neretin <[email protected]>
On Sun, Feb 9, 2025 at 9:07 AM Andrey Borodin <[email protected]> wrote:
>
> I've took into account note from Sergey that "offset" is better name for uuidv7() argument than "shift".
>
> > On 5 Feb 2025, at 03:02, Masahiko Sawada <[email protected]> wrote:
> >
> >>
> >> I was thinking about incorporating test like this.
> >>
> >>>> With this patch we can generate correct UUIDs in a very distant future.
> >>>> postgres=# select x, uuid_extract_timestamp(uuidv7((x::text || ' year'::text)::interval)),
> >>>> (x::text || ' year'::text)::interval
> >>>> from generate_series(1,9000,1000) x;
> >>>> x | uuid_extract_timestamp | interval
> >>>> ------+-----------------------------+------------
> >>>> 1 | 2026-01-31 12:00:53.084+05 | 1 year
> >>>> 1001 | 3026-01-31 12:00:53.084+05 | 1001 years
> >>>> 2001 | 4026-01-31 12:00:53.084+05 | 2001 years
> >>>> 3001 | 5026-01-31 12:00:53.084+05 | 3001 years
> >>>> 4001 | 6026-01-31 12:00:53.084+05 | 4001 years
> >>>> 5001 | 7026-01-31 12:00:53.085+05 | 5001 years
> >>>> 6001 | 8026-01-31 12:00:53.085+05 | 6001 years
> >>>> 7001 | 9026-01-31 12:00:53.085+05 | 7001 years
> >>>> 8001 | 10026-01-31 12:00:53.085+05 | 8001 years
> >>>> (9 rows)
> >>
> >
> > Something like following queries might be workable for example?
> >
> > create table test (c serial, d uuid, t timestamptz generated always as
> > (uuid_extract_timestamp(d)) stored);
> > insert into test (d) select uuidv7((n || 'years')::interval) from
> > generate_series(1, 2000) n;
> > select count(*) from (select t - lag(t) over (order by c) as diff from
> > test) where diff > '10 year' ;
>
> Yeah, makes sense. I reduced tolerance to 366+1 day. Must be stable if we've done all the time offset business right.
>
> > Here are some review comments:
> >
> > #define NS_PER_S INT64CONST(1000000000)
> > #define NS_PER_MS INT64CONST(1000000)
> > +#define US_PER_MS INT64CONST(1000)
> > #define NS_PER_US INT64CONST(1000)
> >
> > I think it's clear if we put US_PER_MS below NS_PER_US.
>
> OK.
>
> > ---
> > *
> > - * ns is a number of nanoseconds since start of the UNIX epoch. This value is
> > + * unix_ts_ms is a number of milliseconds since start of the UNIX epoch,
> > + * ns_in_ms is a number of nanoseconds within millisecond. These values are
> > * used for time-dependent bits of UUID.
> >
> > I think we can mention that the RFC describes that stored unix
> > timestamp as an unsigned integer.
>
> Done. Feel free to adjust my wordings, I've no sense of idiomatic English.
>
> >
> > ---
> > static pg_uuid_t *
> > -generate_uuidv7(int64 ns)
> > +generate_uuidv7(uint64 unix_ts_ms, uint32 ns_in_ms)
> >
> > How about renaming ns_in_ms with sub_ms?
>
> OK.
>
> >
> > ---
> > + /* 64 bits is enough for real time, but not for a time range of UUID */
> >
> > I could not understand the point of this comment. It seems to say that
> > 64-bits is not enough for a time range of UUID, but doesn't the time
> > range of UUIDv7 use only 48 bits? It seems to need more comments.
>
> I've tried to say that acquiring current time as an int64 ns since UNIX epoch is still viable for the code (until year 2262).
>
>
> > ---
> > - ns = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) *
> > SECS_PER_DAY * USECS_PER_SEC)
> > - * NS_PER_US + ns % NS_PER_US;
> > + us = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) *
> > SECS_PER_DAY * USECS_PER_SEC);
> >
> > /* Generate an UUIDv7 */
> > - uuid = generate_uuidv7(ns);
> > + uuid = generate_uuidv7(us / US_PER_MS, (us % US_PER_MS) *
> > NS_PER_US + ns % NS_PER_US);
> >
> > Need to update comments in uuidv7_internval() such as:
> >
> > /*
> > * Shift the current timestamp by the given interval. To calculate time
> > * shift correctly, we convert the UNIX epoch to TimestampTz and use
> > * timestamptz_pl_interval(). Since this calculation is done with
> > * microsecond precision, we carry nanoseconds from original ns value to
> > * shifted ns value.
> > */
> >
> > and
> >
> > /*
> > * Convert a TimestampTz value back to an UNIX epoch and back nanoseconds.
> > */
>
> I've tried. I'm not very satisfied with comments, but could not come up with easier description.
>
Thank you for updating the patch. I had missed to track this patch.
I've updated the patch from your v4 patch. In this version, I excluded
the argument name change (from 'shift' to 'offset') as it's not
related to the bug fix and simplified the regression test case.
Please review it.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v5-0001-Fix-timestamp-overflow-in-UUIDv7-implementation.patch (6.9K, ../../CAD21AoAVU-VnuqvdafqUCHCAo7Msb6_2k7nSzd_fijMbbtKrCg@mail.gmail.com/2-v5-0001-Fix-timestamp-overflow-in-UUIDv7-implementation.patch)
download | inline diff:
From 82ba268b7af93a75c76cf36a85c764761e0dbeb1 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 25 Mar 2025 15:14:42 -0700
Subject: [PATCH v5] Fix timestamp overflow in UUIDv7 implementation.
Previously, the uuidv7_interval() function performed timestamp
shifting calculations using microsecond precision, but then converted
the result back to nanosecond precision. Since the millisecond and
sub-millisecond parts were extracted from this nanosecond timestamp
and stored into the UUIDv7 value, overflow occurred for timestamps
beyond the year 2262.
With this commit, the millisecond and sub-millisecond parts are stored
directly into the UUIDv7 value without being converted back to a
nanosecond precision timestamp. Following RFC 9562, the timestamp is
stored as an unsigned integer, enabling support for dates up to the
year 10889.
Reported and fixed by Andrey Borodin, with cosmetic changes and
regression tests by me.
Reported-by: Andrey Borodin <[email protected]>
Author: Andrey Borodin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
src/backend/utils/adt/uuid.c | 34 +++++++++++++++---------------
src/test/regress/expected/uuid.out | 14 ++++++++++++
src/test/regress/sql/uuid.sql | 11 ++++++++++
3 files changed, 42 insertions(+), 17 deletions(-)
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 4f8402ef925..be0f0f9f1ce 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -29,6 +29,7 @@
#define NS_PER_S INT64CONST(1000000000)
#define NS_PER_MS INT64CONST(1000000)
#define NS_PER_US INT64CONST(1000)
+#define US_PER_MS INT64CONST(1000)
/*
* UUID version 7 uses 12 bits in "rand_a" to store 1/4096 (or 2^12) fractions of
@@ -69,6 +70,7 @@ static bool uuid_abbrev_abort(int memtupcount, SortSupport ssup);
static Datum uuid_abbrev_convert(Datum original, SortSupport ssup);
static inline void uuid_set_version(pg_uuid_t *uuid, unsigned char version);
static inline int64 get_real_time_ns_ascending();
+static pg_uuid_t *generate_uuidv7(uint64 unix_ts_ms, uint32 sub_ms);
Datum
uuid_in(PG_FUNCTION_ARGS)
@@ -523,17 +525,17 @@ get_real_time_ns_ascending()
* described in the RFC. This method utilizes 12 bits from the "rand_a" bits
* to store a 1/4096 (or 2^12) fraction of sub-millisecond precision.
*
- * ns is a number of nanoseconds since start of the UNIX epoch. This value is
+ * unix_ts_ms is a number of milliseconds since start of the UNIX epoch,
+ * and sub_ms is a number of nanoseconds within millisecond. These values are
* used for time-dependent bits of UUID.
+ *
+ * NB: all numbers here are unsigned, unix_ts_ms cannot be negative per RFC.
*/
static pg_uuid_t *
-generate_uuidv7(int64 ns)
+generate_uuidv7(uint64 unix_ts_ms, uint32 sub_ms)
{
pg_uuid_t *uuid = palloc(UUID_LEN);
- int64 unix_ts_ms;
- int32 increased_clock_precision;
-
- unix_ts_ms = ns / NS_PER_MS;
+ uint32 increased_clock_precision;
/* Fill in time part */
uuid->data[0] = (unsigned char) (unix_ts_ms >> 40);
@@ -547,7 +549,7 @@ generate_uuidv7(int64 ns)
* sub-millisecond timestamp fraction (SUBMS_BITS bits, not
* SUBMS_MINIMAL_STEP_BITS)
*/
- increased_clock_precision = ((ns % NS_PER_MS) * (1 << SUBMS_BITS)) / NS_PER_MS;
+ increased_clock_precision = (sub_ms * (1 << SUBMS_BITS)) / NS_PER_MS;
/* Fill the increased clock precision to "rand_a" bits */
uuid->data[6] = (unsigned char) (increased_clock_precision >> 8);
@@ -586,7 +588,8 @@ generate_uuidv7(int64 ns)
Datum
uuidv7(PG_FUNCTION_ARGS)
{
- pg_uuid_t *uuid = generate_uuidv7(get_real_time_ns_ascending());
+ int64 ns = get_real_time_ns_ascending();
+ pg_uuid_t *uuid = generate_uuidv7(ns / NS_PER_MS, ns % NS_PER_MS);
PG_RETURN_UUID_P(uuid);
}
@@ -601,13 +604,13 @@ uuidv7_interval(PG_FUNCTION_ARGS)
TimestampTz ts;
pg_uuid_t *uuid;
int64 ns = get_real_time_ns_ascending();
+ int64 us;
/*
* Shift the current timestamp by the given interval. To calculate time
* shift correctly, we convert the UNIX epoch to TimestampTz and use
- * timestamptz_pl_interval(). Since this calculation is done with
- * microsecond precision, we carry nanoseconds from original ns value to
- * shifted ns value.
+ * timestamptz_pl_interval(). This calculation is done with microsecond
+ * precision.
*/
ts = (TimestampTz) (ns / NS_PER_US) -
@@ -618,14 +621,11 @@ uuidv7_interval(PG_FUNCTION_ARGS)
TimestampTzGetDatum(ts),
IntervalPGetDatum(shift)));
- /*
- * Convert a TimestampTz value back to an UNIX epoch and back nanoseconds.
- */
- ns = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC)
- * NS_PER_US + ns % NS_PER_US;
+ /* Convert a TimestampTz value back to an UNIX epoch timestamp */
+ us = ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
/* Generate an UUIDv7 */
- uuid = generate_uuidv7(ns);
+ uuid = generate_uuidv7(us / US_PER_MS, (us % US_PER_MS) * NS_PER_US + ns % NS_PER_US);
PG_RETURN_UUID_P(uuid);
}
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 798633ad51e..cbd497376c4 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -233,6 +233,20 @@ SELECT array_agg(id ORDER BY guid_field) FROM guid3;
{1,2,3,4,5,6,7,8,9,10}
(1 row)
+-- Check the timestamp offsets for v7.
+--
+-- generate UUIDv7 having timestamps up to 10889 year, which is the maximum year
+-- can be stored in UUIDv7, and then check if the timestamps extracted from UUIDv7
+-- values are not overflowed.
+WITH uuidts AS (
+ SELECT y, ts as ts, lag(ts) OVER (ORDER BY y) AS prev_ts
+ FROM (SELECT y, uuid_extract_timestamp(uuidv7((y || ' years')::interval)) AS ts FROM generate_series(-50, 10889 - extract(year from now())::int) y)
+)
+SELECT y, ts, prev_ts FROM uuidts WHERE ts < prev_ts;
+ y | ts | prev_ts
+---+----+---------
+(0 rows)
+
-- extract functions
-- version
SELECT uuid_extract_version('11111111-1111-5111-8111-111111111111'); -- 5
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 110188361d1..cd0e65d3a8b 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -119,6 +119,17 @@ SELECT count(DISTINCT guid_field) FROM guid1;
INSERT INTO guid3 (guid_field) SELECT uuidv7() FROM generate_series(1, 10);
SELECT array_agg(id ORDER BY guid_field) FROM guid3;
+-- Check the timestamp offsets for v7.
+--
+-- generate UUIDv7 having timestamps up to 10889 year, which is the maximum year
+-- can be stored in UUIDv7, and then check if the timestamps extracted from UUIDv7
+-- values are not overflowed.
+WITH uuidts AS (
+ SELECT y, ts as ts, lag(ts) OVER (ORDER BY y) AS prev_ts
+ FROM (SELECT y, uuid_extract_timestamp(uuidv7((y || ' years')::interval)) AS ts FROM generate_series(-50, 10889 - extract(year from now())::int) y)
+)
+SELECT y, ts, prev_ts FROM uuidts WHERE ts < prev_ts;
+
-- extract functions
-- version
--
2.43.5
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2025-03-26 13:00 Andrey Borodin <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 2 replies; 69+ messages in thread
From: Andrey Borodin @ 2025-03-26 13:00 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Daniel Verite <[email protected]>; Peter Eisentraut <[email protected]>; Jelte Fennema-Nio <[email protected]>; Sergey Prokhorenko <[email protected]>; Przemysław Sztoch <[email protected]>; Michael Paquier <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; David G. Johnston <[email protected]>; Mat Arye <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Junwang Zhao <[email protected]>; Stepan Neretin <[email protected]>
> On 26 Mar 2025, at 08:32, Masahiko Sawada <[email protected]> wrote:
>
> Please review it.
The patch looks good to me except one nit.
+WITH uuidts AS (
+ SELECT y, ts as ts, lag(ts) OVER (ORDER BY y) AS prev_ts
+ FROM (SELECT y, uuid_extract_timestamp(uuidv7((y || ' years')::interval)) AS ts FROM generate_series(-50, 10889 - extract(year from now())::int) y)
+)
+SELECT y, ts, prev_ts FROM uuidts WHERE ts < prev_ts;
if "extract(year from now())::int)" runs slightly before new year and the rest of the test after - the test will fail. How about avoiding overflow by using 10888 instead of 10889?
If we are sure citizen time never will go back, IMO we can safely move other border back to -55.
Also the test is not proof to NTP time drift during New Year's edge, but it's hardly a problem. The test, NTP clock sync and New Year millisecond must coincide for a false failure.
Thank you!
Best regards, Andrey Borodin.
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2025-03-26 17:06 Masahiko Sawada <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 0 replies; 69+ messages in thread
From: Masahiko Sawada @ 2025-03-26 17:06 UTC (permalink / raw)
To: Andrey Borodin <[email protected]>; +Cc: Daniel Verite <[email protected]>; Peter Eisentraut <[email protected]>; Jelte Fennema-Nio <[email protected]>; Sergey Prokhorenko <[email protected]>; Przemysław Sztoch <[email protected]>; Michael Paquier <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; David G. Johnston <[email protected]>; Mat Arye <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Junwang Zhao <[email protected]>; Stepan Neretin <[email protected]>
On Wed, Mar 26, 2025 at 6:00 AM Andrey Borodin <[email protected]> wrote:
>
>
>
> > On 26 Mar 2025, at 08:32, Masahiko Sawada <[email protected]> wrote:
> >
> > Please review it.
>
> The patch looks good to me except one nit.
> +WITH uuidts AS (
> + SELECT y, ts as ts, lag(ts) OVER (ORDER BY y) AS prev_ts
> + FROM (SELECT y, uuid_extract_timestamp(uuidv7((y || ' years')::interval)) AS ts FROM generate_series(-50, 10889 - extract(year from now())::int) y)
> +)
> +SELECT y, ts, prev_ts FROM uuidts WHERE ts < prev_ts;
>
> if "extract(year from now())::int)" runs slightly before new year and the rest of the test after - the test will fail. How about avoiding overflow by using 10888 instead of 10889?
Agreed. I've done this in the attached patch.
> If we are sure citizen time never will go back, IMO we can safely move other border back to -55.
Yes. Or I think we can verify the range from 1970 to 10888 like I did
in the updated patch.
> Also the test is not proof to NTP time drift during New Year's edge, but it's hardly a problem. The test, NTP clock sync and New Year millisecond must coincide for a false failure.
Agreed.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
Attachments:
[application/octet-stream] v6-0001-Fix-timestamp-overflow-in-UUIDv7-implementation.patch (7.1K, ../../CAD21AoCr1aZdEBzDtU5Lg1XssA68L0uRyWC-a4DjmaXGLK++zA@mail.gmail.com/2-v6-0001-Fix-timestamp-overflow-in-UUIDv7-implementation.patch)
download | inline diff:
From 0baca061a9ff508b81d90c514c498d9cb03a63b2 Mon Sep 17 00:00:00 2001
From: Masahiko Sawada <[email protected]>
Date: Tue, 25 Mar 2025 15:14:42 -0700
Subject: [PATCH v6] Fix timestamp overflow in UUIDv7 implementation.
Previously, the uuidv7_interval() function performed timestamp
shifting calculations using microsecond precision, but then converted
the result back to nanosecond precision. Since the millisecond and
sub-millisecond parts were extracted from this nanosecond timestamp
and stored into the UUIDv7 value, overflow occurred for timestamps
beyond the year 2262.
With this commit, the millisecond and sub-millisecond parts are stored
directly into the UUIDv7 value without being converted back to a
nanosecond precision timestamp. Following RFC 9562, the timestamp is
stored as an unsigned integer, enabling support for dates up to the
year 10889.
Reported and fixed by Andrey Borodin, with cosmetic changes and
regression tests by me.
Reported-by: Andrey Borodin <[email protected]>
Author: Andrey Borodin <[email protected]>
Discussion: https://postgr.es/m/[email protected]
---
src/backend/utils/adt/uuid.c | 34 +++++++++++++++---------------
src/test/regress/expected/uuid.out | 15 +++++++++++++
src/test/regress/sql/uuid.sql | 12 +++++++++++
3 files changed, 44 insertions(+), 17 deletions(-)
diff --git a/src/backend/utils/adt/uuid.c b/src/backend/utils/adt/uuid.c
index 4f8402ef925..be0f0f9f1ce 100644
--- a/src/backend/utils/adt/uuid.c
+++ b/src/backend/utils/adt/uuid.c
@@ -29,6 +29,7 @@
#define NS_PER_S INT64CONST(1000000000)
#define NS_PER_MS INT64CONST(1000000)
#define NS_PER_US INT64CONST(1000)
+#define US_PER_MS INT64CONST(1000)
/*
* UUID version 7 uses 12 bits in "rand_a" to store 1/4096 (or 2^12) fractions of
@@ -69,6 +70,7 @@ static bool uuid_abbrev_abort(int memtupcount, SortSupport ssup);
static Datum uuid_abbrev_convert(Datum original, SortSupport ssup);
static inline void uuid_set_version(pg_uuid_t *uuid, unsigned char version);
static inline int64 get_real_time_ns_ascending();
+static pg_uuid_t *generate_uuidv7(uint64 unix_ts_ms, uint32 sub_ms);
Datum
uuid_in(PG_FUNCTION_ARGS)
@@ -523,17 +525,17 @@ get_real_time_ns_ascending()
* described in the RFC. This method utilizes 12 bits from the "rand_a" bits
* to store a 1/4096 (or 2^12) fraction of sub-millisecond precision.
*
- * ns is a number of nanoseconds since start of the UNIX epoch. This value is
+ * unix_ts_ms is a number of milliseconds since start of the UNIX epoch,
+ * and sub_ms is a number of nanoseconds within millisecond. These values are
* used for time-dependent bits of UUID.
+ *
+ * NB: all numbers here are unsigned, unix_ts_ms cannot be negative per RFC.
*/
static pg_uuid_t *
-generate_uuidv7(int64 ns)
+generate_uuidv7(uint64 unix_ts_ms, uint32 sub_ms)
{
pg_uuid_t *uuid = palloc(UUID_LEN);
- int64 unix_ts_ms;
- int32 increased_clock_precision;
-
- unix_ts_ms = ns / NS_PER_MS;
+ uint32 increased_clock_precision;
/* Fill in time part */
uuid->data[0] = (unsigned char) (unix_ts_ms >> 40);
@@ -547,7 +549,7 @@ generate_uuidv7(int64 ns)
* sub-millisecond timestamp fraction (SUBMS_BITS bits, not
* SUBMS_MINIMAL_STEP_BITS)
*/
- increased_clock_precision = ((ns % NS_PER_MS) * (1 << SUBMS_BITS)) / NS_PER_MS;
+ increased_clock_precision = (sub_ms * (1 << SUBMS_BITS)) / NS_PER_MS;
/* Fill the increased clock precision to "rand_a" bits */
uuid->data[6] = (unsigned char) (increased_clock_precision >> 8);
@@ -586,7 +588,8 @@ generate_uuidv7(int64 ns)
Datum
uuidv7(PG_FUNCTION_ARGS)
{
- pg_uuid_t *uuid = generate_uuidv7(get_real_time_ns_ascending());
+ int64 ns = get_real_time_ns_ascending();
+ pg_uuid_t *uuid = generate_uuidv7(ns / NS_PER_MS, ns % NS_PER_MS);
PG_RETURN_UUID_P(uuid);
}
@@ -601,13 +604,13 @@ uuidv7_interval(PG_FUNCTION_ARGS)
TimestampTz ts;
pg_uuid_t *uuid;
int64 ns = get_real_time_ns_ascending();
+ int64 us;
/*
* Shift the current timestamp by the given interval. To calculate time
* shift correctly, we convert the UNIX epoch to TimestampTz and use
- * timestamptz_pl_interval(). Since this calculation is done with
- * microsecond precision, we carry nanoseconds from original ns value to
- * shifted ns value.
+ * timestamptz_pl_interval(). This calculation is done with microsecond
+ * precision.
*/
ts = (TimestampTz) (ns / NS_PER_US) -
@@ -618,14 +621,11 @@ uuidv7_interval(PG_FUNCTION_ARGS)
TimestampTzGetDatum(ts),
IntervalPGetDatum(shift)));
- /*
- * Convert a TimestampTz value back to an UNIX epoch and back nanoseconds.
- */
- ns = (ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC)
- * NS_PER_US + ns % NS_PER_US;
+ /* Convert a TimestampTz value back to an UNIX epoch timestamp */
+ us = ts + (POSTGRES_EPOCH_JDATE - UNIX_EPOCH_JDATE) * SECS_PER_DAY * USECS_PER_SEC;
/* Generate an UUIDv7 */
- uuid = generate_uuidv7(ns);
+ uuid = generate_uuidv7(us / US_PER_MS, (us % US_PER_MS) * NS_PER_US + ns % NS_PER_US);
PG_RETURN_UUID_P(uuid);
}
diff --git a/src/test/regress/expected/uuid.out b/src/test/regress/expected/uuid.out
index 798633ad51e..2e868457d63 100644
--- a/src/test/regress/expected/uuid.out
+++ b/src/test/regress/expected/uuid.out
@@ -233,6 +233,21 @@ SELECT array_agg(id ORDER BY guid_field) FROM guid3;
{1,2,3,4,5,6,7,8,9,10}
(1 row)
+-- Check the timestamp offsets for v7.
+--
+-- generate UUIDv7 values with timestamps ranging from 1970 (the Unix epoch year)
+-- to 10888 (one year before the maximum possible year), and then verify that
+-- the extracted timestamps from these UUIDv7 values have not overflowed.
+WITH uuidts AS (
+ SELECT y, ts as ts, lag(ts) OVER (ORDER BY y) AS prev_ts
+ FROM (SELECT y, uuid_extract_timestamp(uuidv7((y || ' years')::interval)) AS ts
+ FROM generate_series(1970 - extract(year from now())::int, 10888 - extract(year from now())::int) y)
+)
+SELECT y, ts, prev_ts FROM uuidts WHERE ts < prev_ts;
+ y | ts | prev_ts
+---+----+---------
+(0 rows)
+
-- extract functions
-- version
SELECT uuid_extract_version('11111111-1111-5111-8111-111111111111'); -- 5
diff --git a/src/test/regress/sql/uuid.sql b/src/test/regress/sql/uuid.sql
index 110188361d1..241d514eb9c 100644
--- a/src/test/regress/sql/uuid.sql
+++ b/src/test/regress/sql/uuid.sql
@@ -119,6 +119,18 @@ SELECT count(DISTINCT guid_field) FROM guid1;
INSERT INTO guid3 (guid_field) SELECT uuidv7() FROM generate_series(1, 10);
SELECT array_agg(id ORDER BY guid_field) FROM guid3;
+-- Check the timestamp offsets for v7.
+--
+-- generate UUIDv7 values with timestamps ranging from 1970 (the Unix epoch year)
+-- to 10888 (one year before the maximum possible year), and then verify that
+-- the extracted timestamps from these UUIDv7 values have not overflowed.
+WITH uuidts AS (
+ SELECT y, ts as ts, lag(ts) OVER (ORDER BY y) AS prev_ts
+ FROM (SELECT y, uuid_extract_timestamp(uuidv7((y || ' years')::interval)) AS ts
+ FROM generate_series(1970 - extract(year from now())::int, 10888 - extract(year from now())::int) y)
+)
+SELECT y, ts, prev_ts FROM uuidts WHERE ts < prev_ts;
+
-- extract functions
-- version
--
2.43.5
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2025-03-26 19:32 Andrei Borodin <[email protected]>
parent: Andrey Borodin <[email protected]>
1 sibling, 1 reply; 69+ messages in thread
From: Andrei Borodin @ 2025-03-26 19:32 UTC (permalink / raw)
To: Masahiko Sawada <[email protected]>; +Cc: Daniel Verite <[email protected]>; Peter Eisentraut <[email protected]>; Jelte Fennema-Nio <[email protected]>; Sergey Prokhorenko <[email protected]>; Przemysław Sztoch <[email protected]>; Michael Paquier <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; David G. Johnston <[email protected]>; Mat Arye <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Junwang Zhao <[email protected]>; Stepan Neretin <[email protected]>
<br /><br />26.03.2025, 21:06, "Masahiko Sawada" <[email protected]>:<br /><blockquote><p>Agreed. I've done this in the attached patch.</p></blockquote>Great! The patch looks good to me.<div><br /></div><div>Best regards, Andrey Borodin.</div>
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2025-03-28 05:31 Masahiko Sawada <[email protected]>
parent: Andrei Borodin <[email protected]>
0 siblings, 1 reply; 69+ messages in thread
From: Masahiko Sawada @ 2025-03-28 05:31 UTC (permalink / raw)
To: Andrei Borodin <[email protected]>; +Cc: Daniel Verite <[email protected]>; Peter Eisentraut <[email protected]>; Jelte Fennema-Nio <[email protected]>; Sergey Prokhorenko <[email protected]>; Przemysław Sztoch <[email protected]>; Michael Paquier <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; David G. Johnston <[email protected]>; Mat Arye <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Junwang Zhao <[email protected]>; Stepan Neretin <[email protected]>
On Wed, Mar 26, 2025 at 12:32 PM Andrei Borodin <[email protected]> wrote:
>
>
>
> 26.03.2025, 21:06, "Masahiko Sawada" <[email protected]>:
>
> Agreed. I've done this in the attached patch.
>
> Great! The patch looks good to me.
Thank you for reviewing it. I'm going to push the fix tomorrow,
barring further comments.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 69+ messages in thread
* Re: UUID v7
@ 2025-03-29 00:09 Masahiko Sawada <[email protected]>
parent: Masahiko Sawada <[email protected]>
0 siblings, 0 replies; 69+ messages in thread
From: Masahiko Sawada @ 2025-03-29 00:09 UTC (permalink / raw)
To: Andrei Borodin <[email protected]>; +Cc: Daniel Verite <[email protected]>; Peter Eisentraut <[email protected]>; Jelte Fennema-Nio <[email protected]>; Sergey Prokhorenko <[email protected]>; Przemysław Sztoch <[email protected]>; Michael Paquier <[email protected]>; Aleksander Alekseev <[email protected]>; pgsql-hackers; David G. Johnston <[email protected]>; Mat Arye <[email protected]>; Matthias van de Meent <[email protected]>; Nikolay Samokhvalov <[email protected]>; Junwang Zhao <[email protected]>; Stepan Neretin <[email protected]>
On Thu, Mar 27, 2025 at 10:31 PM Masahiko Sawada <[email protected]> wrote:
>
> On Wed, Mar 26, 2025 at 12:32 PM Andrei Borodin <[email protected]> wrote:
> >
> >
> >
> > 26.03.2025, 21:06, "Masahiko Sawada" <[email protected]>:
> >
> > Agreed. I've done this in the attached patch.
> >
> > Great! The patch looks good to me.
>
> Thank you for reviewing it. I'm going to push the fix tomorrow,
> barring further comments.
Pushed.
Regards,
--
Masahiko Sawada
Amazon Web Services: https://aws.amazon.com
^ permalink raw reply [nested|flat] 69+ messages in thread
end of thread, other threads:[~2025-03-29 00:09 UTC | newest]
Thread overview: 69+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2022-11-13 20:03 [PATCH 07/10] cirrus/ccache: use G rather than GB suffix.. Justin Pryzby <[email protected]>
2024-01-05 10:57 Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-16 12:15 ` Re: UUID v7 Andrey Borodin <[email protected]>
2024-01-16 13:00 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-16 14:44 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-16 16:49 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-18 13:17 ` Re: UUID v7 Andrey Borodin <[email protected]>
2024-01-18 20:26 ` Re: UUID v7 Przemysław Sztoch <[email protected]>
2024-01-19 02:58 ` Re: UUID v7 Lukas Fittl <[email protected]>
2024-01-16 20:10 ` Re: UUID v7 Jelte Fennema-Nio <[email protected]>
2024-01-16 21:09 ` Re: UUID v7 Przemysław Sztoch <[email protected]>
2024-01-18 14:20 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-18 15:39 ` Re: UUID v7 Andrey Borodin <[email protected]>
2024-01-18 16:21 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-18 18:28 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-18 18:31 ` Re: UUID v7 Andrey Borodin <[email protected]>
2024-01-18 18:59 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-19 03:24 ` Re: UUID v7 David G. Johnston <[email protected]>
2024-01-19 08:25 ` Re: UUID v7 Andrey Borodin <[email protected]>
2024-01-19 18:07 ` Re: UUID v7 Andrey Borodin <[email protected]>
2024-01-22 04:22 ` Re: UUID v7 Nikolay Samokhvalov <[email protected]>
2024-01-22 04:24 ` Re: UUID v7 Nikolay Samokhvalov <[email protected]>
2024-01-22 15:02 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-24 12:31 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-24 12:40 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-24 13:02 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-24 13:16 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-24 13:29 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-24 15:29 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-24 15:46 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-24 16:54 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-24 17:00 ` Re: UUID v7 Marcos Pegoraro <[email protected]>
2024-01-24 17:51 ` Re: UUID v7 Andrey Borodin <[email protected]>
2024-01-24 20:47 ` Re: UUID v7 Marcos Pegoraro <[email protected]>
2024-01-24 21:15 ` Re: UUID v7 Jelte Fennema-Nio <[email protected]>
2024-01-24 21:30 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-24 13:16 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-24 21:49 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-25 04:40 ` Re: UUID v7 Nikolay Samokhvalov <[email protected]>
2024-01-25 04:41 ` Re: UUID v7 Nikolay Samokhvalov <[email protected]>
2024-01-25 06:51 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-25 08:09 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-25 11:14 ` Re: UUID v7 Przemysław Sztoch <[email protected]>
2024-01-25 12:06 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-25 12:31 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-29 12:01 ` Re: UUID v7 Jelte Fennema-Nio <[email protected]>
2024-01-25 17:04 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-29 11:38 ` Re: UUID v7 Jelte Fennema-Nio <[email protected]>
2024-01-29 13:58 ` Re: UUID v7 Junwang Zhao <[email protected]>
2024-01-29 18:32 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-29 20:38 ` Re: UUID v7 Jelte Fennema-Nio <[email protected]>
2024-01-30 06:54 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-30 07:28 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-30 09:56 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-30 10:33 ` Re: UUID v7 Junwang Zhao <[email protected]>
2024-01-30 13:35 ` Re: UUID v7 Andrey M. Borodin <[email protected]>
2024-01-30 18:37 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-30 00:27 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-28 12:42 ` Re: UUID v7 Sergey Prokhorenko <[email protected]>
2024-01-18 20:49 ` Re: UUID v7 Przemysław Sztoch <[email protected]>
2024-01-18 20:39 ` Re: UUID v7 Przemysław Sztoch <[email protected]>
2024-01-19 11:07 ` Re: UUID v7 Aleksander Alekseev <[email protected]>
2024-01-16 20:20 ` Re: UUID v7 Przemysław Sztoch <[email protected]>
2025-03-26 04:32 Re: UUID v7 Masahiko Sawada <[email protected]>
2025-03-26 13:00 ` Re: UUID v7 Andrey Borodin <[email protected]>
2025-03-26 17:06 ` Re: UUID v7 Masahiko Sawada <[email protected]>
2025-03-26 19:32 ` Re: UUID v7 Andrei Borodin <[email protected]>
2025-03-28 05:31 ` Re: UUID v7 Masahiko Sawada <[email protected]>
2025-03-29 00:09 ` Re: UUID v7 Masahiko Sawada <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox