Subject: Re: LLVM 16 (opaque pointers)

From: Ronan Dunklau <[email-masked]>
Date: Wed, 11 Oct 2023 11:31:43 +0200

Le mercredi 11 octobre 2023, 10:59:50 CEST Thomas Munro a écrit :
> The back-patch to 12 was a little trickier than anticipated, but after
> taking a break and trying again I now have PG 12...17 patches that
> I've tested against LLVM 10...18 (that's 54 combinations), in every
> case only with the clang corresponding to LLVM.

Thank you Thomas for those patches, and the extensive testing, I will run my 
own and let you know.

> I've attached only the patches for master, but the 12-16 versions are
> available at https://github.com/macdice/postgres/tree/llvm16-$N in
> case anyone has comments on those.

For PG13 and PG12, it looks like the ExecEvalBoolSubroutineTemplate is not 
used anywhere, as ExecEvalBoolSubroutine was introduced in PG14 if I'm not 
mistaken. 

Best regards,

--
Ronan Dunklau

---

From: Thomas Munro <[email-masked]>
Date: Sat, 14 Oct 2023 09:32:13 +1300

> Thank you Thomas for those patches, and the extensive testing, I will run my

Thanks!  No news is good news, I hope?  I'm hoping to commit this today.

> used anywhere, as ExecEvalBoolSubroutine was introduced in PG14 if I'm not

Right, looks like I can remove that in those branches.

---

From: Ronan Dunklau <[email-masked]>
Date: Mon, 16 Oct 2023 10:31:32 +0200

Le vendredi 13 octobre 2023, 22:32:13 CEST Thomas Munro a écrit :
> > Thank you Thomas for those patches, and the extensive testing, I will run
> > my own and let you know.
> 

Oh sorry I thought I followed up. I ran the same stress testing involving 
several hours of sqlsmith with all jit costs set to zero and didn't notice 
anything with LLVM16.

Thank you !

--
Ronan Dunklau

---

From: Thomas Munro <[email-masked]>
Date: Thu, 19 Oct 2023 00:02:11 +1300

I pushed the first patch, for LLVM 16, and the build farm told me that
some old LLVM versions don't like it.  The problem seems to be the
function LLVMGlobalGetValueType().  I can see that that was only added
to the C API in 2018, so it looks like I may need to back-port that
(trivial) wrapper into our own llvmjit_wrap.cpp for LLVM < 8.

---

From: Thomas Munro <[email-masked]>
Date: Thu, 19 Oct 2023 01:06:11 +1300

> (trivial) wrapper into our own llvmjit_wrap.cpp for LLVM < 8.

Concretely something like the attached should probably fix it, but
it'll take me a little while to confirm that...

---

From: Thomas Munro <[email-masked]>
Date: Thu, 19 Oct 2023 06:20:26 +1300

> it'll take me a little while to confirm that...

Pushed, after digging up some old LLVM skeletons to test, and those
"old LLVM" animals are turning green now.  I went ahead and pushed the
much smaller and simpler patch for LLVM 17.

Interestingly, a new problem just showed up on the the RHEL9 s390x
machine "lora", where a previously reported problem [1] apparently
re-appeared.  It complains about incompatible layout, previously
blamed on mismatch between clang and LLVM versions.  I can see that
its clang is v15 from clues in the conflig log, but I don't know which
version of LLVM is being used.  However, I see now that --with-llvm
was literally just turned on, so there is no reason to think that this
would have worked before or this work is relevant.  Strange though --
we must be able to JIT further than that on s390x because we have
crash reports in other threads (ie we made it past this and into other
more advanced brokenness).

[1] https://www.postgresql.org/message-id/flat/20210319190047.7o4bwhbp5dzkqif3%40alap3.anarazel.de#ec51b488ca8eac8c603d91c0439d38b2

---

From: Andres Freund <[email-masked]>
Date: Wed, 18 Oct 2023 11:11:32 -0700

Hi,

> much smaller and simpler patch for LLVM 17.

I enabled a new set of buildfarm animals to test LLVM 16 and 17. I initially
forgot to disable them for 11, which means we'll have those failed build on 11
until they age out :/.

Particularly for the LLVM debug builds it'll take a fair bit to run on all
branches. Each branch takes about 3h.

Greetings,

Andres Freund

---

From: Thomas Munro <[email-masked]>
Date: Sat, 21 Oct 2023 10:48:47 +1300

> more advanced brokenness).

I see that Mark has also just enabled --with-llvm on some POWER Linux
animals, and they have failed in various ways.  The failures are
strangely lacking in detail.  It seems we didn't have coverage before,
and I recall that there were definitely versions of LLVM that *didn't*
work for our usage in the past, which I'll need to dredge out of the
archives.  I will try to get onto a cfarm POWER machine and see if I
can reproduce that, before and after these commits, and whose bug is
it etc.

I doubt I can get anywhere near an s390x though, and we definitely had
pre-existing problems on that arch.

---

From: Mark Wong <[email-masked]>
Date: Fri, 20 Oct 2023 15:00:40 -0700

> can reproduce that, before and after these commits, and whose bug is

Yeah, I'm slowing enabling --with-llvm on POWER, s390x, and aarch64 (but
none here yet as I write this)...

> pre-existing problems on that arch.

If you want to send me your ssh key, I have access to these systems
through OSUOSL and LinuxFoundation programs.

Regards,
Mark

--
Mark Wong
EDB https://enterprisedb.com

---

From: Tom Lane <[email-masked]>
Date: Fri, 20 Oct 2023 18:03:07 -0400

> pre-existing problems on that arch.

Yeah.  Too bad there's no s390x in the gcc compile farm.
(I'm wondering how straight a line to draw between that fact
and llvm's evident shortcomings on s390x.)

I'm missing my old access to Red Hat's dev machines.  But in
the meantime, Mark's clearly got beaucoup access to s390
machines, so I wonder if he can let you into any of them?

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Fri, 20 Oct 2023 15:12:13 -0700

Hi,

> can reproduce that, before and after these commits, and whose bug is

I'm quite sure that jiting did pass on ppc64 at some point.

> pre-existing problems on that arch.

IMO an LLVM bug, rather than a postgres bug, but I guess it's all a matter of
perspective.
https://github.com/llvm/llvm-project/issues/53009#issuecomment-1042748553

I had made another bug report about this issue at some point, but I can't
refind it right now.

Greetings,

Andres Freund

---

From: Thomas Munro <[email-masked]>
Date: Sat, 21 Oct 2023 12:02:51 +1300

> I'm quite sure that jiting did pass on ppc64 at some point.

Yeah, I remember debugging it on EDB's POWER machine.  First off, we
know that LLVM < 7 doesn't work for us on POWER, because:

https://www.postgresql.org/message-id/CAEepm%3D39F_B3Ou8S3OrUw%2BhJEUP3p%3DwCu0ug-TTW67qKN53g3w%40mail.gmail.com

That was fixed:

https://github.com/llvm/llvm-project/commit/a95b0df5eddbe7fa1e9f8fe0b1ff62427e1c0318

So I think that means that we'd first have to go through those animals
and figure out which ones have older LLVM, and ignore those results --
they just can't use --with-llvm.  Unfortunately there doesn't seem to
be any clue on the version from the paths used by OpenSUSE.  Mark, do
you know?

> https://github.com/llvm/llvm-project/issues/53009#issuecomment-1042748553

Ah, good to know about that.  But there are also reports of crashes in
released versions that manage to get passed that ABI-wobble business:

https://www.postgresql.org/message-id/flat/CAF1DzPXjpPxnsgySz2Zjm8d2dx7%3DJ070C%2BMQBFh%2B9NBNcBKCAg%40mail.gmail.com

---

From: Andres Freund <[email-masked]>
Date: Fri, 20 Oct 2023 16:07:28 -0700

Hi,

> https://www.postgresql.org/message-id/flat/CAF1DzPXjpPxnsgySz2Zjm8d2dx7%3DJ070C%2BMQBFh%2B9NBNcBKCAg%40mail.gmail.com

Trying to debug that now, using access to an s390x box provided by Mark...

Greetings,

Andres Freund

---

From: Thomas Munro <[email-masked]>
Date: Sat, 21 Oct 2023 13:45:31 +1300

> be any clue on the version from the paths used by OpenSUSE.  Mark, do

Adding Mark to this subthread.  Concretely, could you please disable
--with-llvm on any of those machines running LLVM < 7?  And report
what version any remaining animals are running?  (It'd be nice if the
build farm logged "$LLVM_CONFIG --version" somewhere.)  One of them
seems to have clang 5 which is a clue -- if the LLVM is also 5 it's
just not going to work, as LLVM is one of those forwards-only projects
that doesn't back-patch fixes like that.

---

From: Tom Lane <[email-masked]>
Date: Fri, 20 Oct 2023 21:45:28 -0400

Thomas Munro <[email-masked]> writes:
> (It'd be nice if the
> build farm logged "$LLVM_CONFIG --version" somewhere.)

It's not really the buildfarm script's responsibility to do that,
but feel free to make configure do so.

			regards, tom lane

---

From: Andres Freund <[email-masked]>
Date: Fri, 20 Oct 2023 23:08:05 -0700

Hi,

On 2023-10-19 06:20:26 +1300, Thomas Munro wrote:
> blamed on mismatch between clang and LLVM versions.

I've attached a patch revision that I spent the last couple hours working
on. It's very very roughly based on a patch Tom Stellard had written (which I
think a few rpm packages use). But instead of encoding details about specific
layout details, I made the code check if the data layout works and fall back
to the cpu / features used for llvmjit_types.bc.  This way it's not s390x
specific, future odd architecture behaviour would "automatically" be handled
the same.

With that at least the main regression tests pass on s390x, even with
jit_above_cost=0.

> I can see that its clang is v15 from clues in the conflig log, but I don't
> know which version of LLVM is being used.  However, I see now that
> --with-llvm was literally just turned on, so there is no reason to think
> that this would have worked before or this work is relevant.  Strange though
> -- we must be able to JIT further than that on s390x because we have crash
> reports in other threads (ie we made it past this and into other more
> advanced brokenness).

You can avoid the borkedness by a) running on an older cpu b) adding
compilation flags to change the code generation target
(e.g. -march=native). And some RPM packages have applied the patch by Tom
Stellard.

Greetings,

Andres Freund
>From e7ab9eb11576ef85e4d2f6e1ade0a6028279634d Mon Sep 17 00:00:00 2001
From: Andres Freund <[email-masked]>
Date: Fri, 20 Oct 2023 23:03:53 -0700
Subject: [PATCH v2] jit: Add fallback in case of runtime/compile time ABI
 mismatch

LLVM's s390x target uses a different ABI (called data layout in LLVM) for z13
and newer processors.  If IR files (like llvmjit_types.bc) are compiled to
target a processor older than z13, which is the default, and JIT occurs on a
z13 or newer processor, the ABI mismatch will cause JIT to fail at runtime.

To deal with that, check if data layouts match during JIT initialization. If
the runtime detected cpu / features result in a different layout, try if the
cpu/features recorded in in llvmjit_types.bc work.

Author: Andres Freund <[email-masked]>
Author: Tom Stellard <[email-masked]> (in an older version)
Discussion: [email-masked]
Discussion: [email-masked]
---
 src/backend/jit/llvm/llvmjit.c | 191 ++++++++++++++++++++++++++++-----
 1 file changed, 166 insertions(+), 25 deletions(-)

diff --git a/src/backend/jit/llvm/llvmjit.c b/src/backend/jit/llvm/llvmjit.c
index 58f638859a4..fa2a982991d 100644
--- a/src/backend/jit/llvm/llvmjit.c
+++ b/src/backend/jit/llvm/llvmjit.c
@@ -101,6 +101,8 @@ static size_t llvm_jit_context_in_use_count = 0;
 static size_t llvm_llvm_context_reuse_count = 0;
 static const char *llvm_triple = NULL;
 static const char *llvm_layout = NULL;
+static char *llvm_cpu = NULL;
+static char *llvm_cpu_features = NULL;
 static LLVMContextRef llvm_context;

@@ -123,6 +125,7 @@ static void llvm_optimize_module(LLVMJitContext *context, LLVMModuleRef module);
 
 static void llvm_create_types(void);
 static void llvm_set_target(void);
+static void llvm_set_cpu_and_features(void);
 static void llvm_recreate_llvm_context(void);
 static uint64_t llvm_resolve_symbol(const char *name, void *ctx);
 
@@ -884,9 +887,6 @@ static void
 llvm_session_initialize(void)
 {
 	MemoryContext oldcontext;
-	char	   *error = NULL;
-	char	   *cpu = NULL;
-	char	   *features = NULL;
 	LLVMTargetMachineRef opt0_tm;
 	LLVMTargetMachineRef opt3_tm;
 
@@ -931,38 +931,21 @@ llvm_session_initialize(void)
 	 */
 	llvm_set_target();
 
-	if (LLVMGetTargetFromTriple(llvm_triple, &llvm_targetref, &error) != 0)
-	{
-		elog(FATAL, "failed to query triple %s", error);
-	}
-
-	/*
-	 * We want the generated code to use all available features. Therefore
-	 * grab the host CPU string and detect features of the current CPU. The
-	 * latter is needed because some CPU architectures default to enabling
-	 * features not all CPUs have (weird, huh).
-	 */
-	cpu = LLVMGetHostCPUName();
-	features = LLVMGetHostCPUFeatures();
-	elog(DEBUG2, "LLVMJIT detected CPU \"%s\", with features \"%s\"",
-		 cpu, features);
+	llvm_set_cpu_and_features();
 
 	opt0_tm =
-		LLVMCreateTargetMachine(llvm_targetref, llvm_triple, cpu, features,
+		LLVMCreateTargetMachine(llvm_targetref, llvm_triple,
+								llvm_cpu, llvm_cpu_features,
 								LLVMCodeGenLevelNone,
 								LLVMRelocDefault,
 								LLVMCodeModelJITDefault);
 	opt3_tm =
-		LLVMCreateTargetMachine(llvm_targetref, llvm_triple, cpu, features,
+		LLVMCreateTargetMachine(llvm_targetref, llvm_triple,
+								llvm_cpu, llvm_cpu_features,
 								LLVMCodeGenLevelAggressive,
 								LLVMRelocDefault,
 								LLVMCodeModelJITDefault);
 
-	LLVMDisposeMessage(cpu);
-	cpu = NULL;
-	LLVMDisposeMessage(features);
-	features = NULL;
-
 	/* force symbols in main binary to be loaded */
 	LLVMLoadLibraryPermanently(NULL);
 
@@ -1092,6 +1075,73 @@ load_return_type(LLVMModuleRef mod, const char *name)
 	return typ;
 }
 
+/*
+ * Copies a string that needs to be freed with LLVMDisposeMessage() and then
+ * frees the source string.
+ */
+static char *
+llvm_to_pg_str(char *str)
+{
+	char	   *ret = pstrdup(str);
+
+	LLVMDisposeMessage(str);
+
+	return ret;
+}
+
+/*
+ * Return data layout for a target machine created with cpu and features
+ *
+ * The return value is a palloc'd string.
+ */
+static char *
+determine_data_layout(const char *cpu, const char *features)
+{
+	LLVMTargetMachineRef tm;
+	LLVMTargetDataRef layout;
+	char	   *layout_str;
+
+	tm = LLVMCreateTargetMachine(llvm_targetref, llvm_triple, cpu, features,
+								 LLVMCodeGenLevelNone,
+								 LLVMRelocDefault,
+								 LLVMCodeModelJITDefault);
+	layout = LLVMCreateTargetDataLayout(tm);
+	layout_str = LLVMCopyStringRepOfTargetData(layout);
+
+	LLVMDisposeTargetData(layout);
+	LLVMDisposeTargetMachine(tm);
+
+	return llvm_to_pg_str(layout_str);
+}
+
+/*
+ * Convenience wrapper around LLVMGetStringAttributeAtIndex &
+ * LLVMGetStringAttributeValue.
+ *
+ * The return value is a zero-terminated, palloc'd string.
+ */
+static char *
+get_string_attribute_value(LLVMValueRef v, uint32 index,
+						   const char *name, const char *fallback)
+{
+	LLVMAttributeRef attr;
+	const char *val;
+	unsigned	len;
+
+	attr = LLVMGetStringAttributeAtIndex(v, index, name, strlen(name));
+	if (!attr)
+		return fallback ? pstrdup(fallback) : NULL;
+
+	val = LLVMGetStringAttributeValue(attr, &len);
+
+	/*
+	 * LLVMGetStringAttributeValue() returns values not zero terminated, which
+	 * inconvenient to work with. Also has the advantage that the return value
+	 * is freed by memory context cleanup etc.
+	 */
+	return psprintf("%.*s", len, val);
+}
+
 /*
  * Load triple & layout from clang emitted file so we're guaranteed to be
  * compatible.
@@ -1099,14 +1149,105 @@ load_return_type(LLVMModuleRef mod, const char *name)
 static void
 llvm_set_target(void)
 {
+	char	   *error = NULL;
+
 	if (!llvm_types_module)
 		elog(ERROR, "failed to extract target information, llvmjit_types.c not loaded");
 
+	/* can get called again after partial initialization */
+
 	if (llvm_triple == NULL)
 		llvm_triple = pstrdup(LLVMGetTarget(llvm_types_module));
 
 	if (llvm_layout == NULL)
 		llvm_layout = pstrdup(LLVMGetDataLayoutStr(llvm_types_module));
+
+	if (llvm_targetref == NULL)
+	{
+		if (LLVMGetTargetFromTriple(llvm_triple, &llvm_targetref, &error) != 0)
+			elog(FATAL, "failed to query triple %s", error);
+	}
+}
+
+/*
+ * Determine CPU and features to use for JIT.
+ *
+ * We want the generated code to use all available features. Therefore
+ * grab the host CPU string and detect features of the current CPU. The
+ * latter is needed because some CPU architectures default to enabling
+ * features not all CPUs have (weird, huh).
+ *
+ * Unfortunately there is at least one architecture on which LLVM doesn't play
+ * fair - on s390, LLVM will use a different ABI for the same triple,
+ * depending on host CPU (IMO not a sane decision, but ...). To work around
+ * that, if the layout of llvmjit_types.bc does not match what we get using
+ * the host cpu / features, try target-cpu/target-features that clang recorded
+ * in llvmjit_types.bc at compile time.
+ */
+static void
+llvm_set_cpu_and_features(void)
+{
+	char	   *host_cpu;
+	char	   *host_cpu_features;
+	char	   *host_layout_str;
+
+	/* can get called again after partial initialization */
+	if (llvm_cpu != NULL)
+		return;
+
+	/* determine runtime CPU / feature */
+	host_cpu = llvm_to_pg_str(LLVMGetHostCPUName());
+	host_cpu_features = llvm_to_pg_str(LLVMGetHostCPUFeatures());
+	host_layout_str = determine_data_layout(host_cpu, host_cpu_features);
+
+	elog(DEBUG2, "detected CPU \"%s\", with features \"%s\", resulting in layout \"%s\"",
+		 host_cpu, host_cpu_features, host_layout_str);
+
+	/* check if we can use detected values or if we need to fall back */
+	if (strcmp(host_layout_str, llvm_layout) == 0)
+	{
+		llvm_cpu = host_cpu;
+		llvm_cpu_features = host_cpu_features;
+		pfree(host_layout_str);
+	}
+	else
+	{
+		char	   *module_cpu;
+		char	   *module_cpu_features;
+		char	   *module_layout_str;
+
+		/* incompatible, try to fall back to module cpu / features */
+
+		module_cpu = get_string_attribute_value(AttributeTemplate,
+												LLVMAttributeFunctionIndex,
+												"target-cpu", "generic");
+		module_cpu_features = get_string_attribute_value(AttributeTemplate,
+														 LLVMAttributeFunctionIndex,
+														 "target-features", "");
+		module_layout_str = determine_data_layout(module_cpu, module_cpu_features);
+
+		if (strcmp(module_layout_str, llvm_layout) != 0)
+		{
+			/* leaking a few strings, this isn't expected to ever be hit */
+			ereport(ERROR,
+					errmsg_internal("could not determine working CPU / feature comination for JIT compilation"),
+					errdetail_internal("compile time data layout: \"%s\", host layout \"%s\", fallback layout \"%s\"",
+									   llvm_layout, host_layout_str, module_layout_str));
+		}
+
+		llvm_cpu = module_cpu;
+		llvm_cpu_features = module_cpu_features;
+
+		ereport(DEBUG2,
+				errmsg_internal("detected CPU / features yield incompatible data layout, using values from module instead"),
+				errdetail_internal("module CPU \"%s\", features \"%s\", resulting in layout \"%s\"",
+								   module_cpu, module_cpu_features, module_layout_str));
+
+		pfree(host_cpu);
+		pfree(host_cpu_features);
+		pfree(host_layout_str);
+		pfree(module_layout_str);
+	}
 }
 
 /*
-- 
2.38.0

---

From: Thomas Munro <[email-masked]>
Date: Sun, 22 Oct 2023 12:16:54 +1300

On Sat, Oct 21, 2023 at 7:08 PM Andres Freund <[email-masked]> wrote:
> the same

The explanation makes sense and this seems like a solid plan to deal
with it.  I didn't try on a s390x, but I tested locally on our master
branch with LLVM 7, 10, 17, 18, and then I hacked your patch to take
the fallback path as if a layout mismatch had been detected, and it
worked fine:

2023-10-22 11:49:55.663 NZDT [12000] DEBUG:  detected CPU "skylake",
with features "...", resulting in layout
"e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"
2023-10-22 11:49:55.664 NZDT [12000] DEBUG:  detected CPU / features
yield incompatible data layout, using values from module instead
2023-10-22 11:49:55.664 NZDT [12000] DETAIL:  module CPU "x86-64",
features "...", resulting in layout
"e-m:e-p270:32:32-p271:32:32-p272:64:64-i64:64-f80:128-n8:16:32:64-S128"

+    To deal with that, check if data layouts match during JIT
initialization. If
+    the runtime detected cpu / features result in a different layout,
try if the
+    cpu/features recorded in in llvmjit_types.bc work.

s|try |check |
s| in in | in |

+                                       errmsg_internal("could not
determine working CPU / feature comination for JIT compilation"),

s|comination|combination|
s| / |/|g

---

From: Thomas Munro <[email-masked]>
Date: Sun, 22 Oct 2023 14:44:31 +1300

> but feel free to make configure do so.

Done, copying the example of how we do it for perl and various other things.

---

From: Thomas Munro <[email-masked]>
Date: Mon, 23 Oct 2023 13:15:04 +1300

> Done, copying the example of how we do it for perl and various other things.

Build farm measles update:

With that we can see that nicator (LLVM 15 on POWER) is green.  We can
see that cavefish (LLVM 6 on POWER) is red as expected.  We can also
see that bonito (LLVM 7 on POWER) is red, so my earlier theory that
this might be due to the known bug we got fixed in LLVM 7 is not
enough.  Maybe there are other things fixed on POWER somewhere between
those LLVM versions?  I suspect it'll be hard to figure out without
debug builds and backtraces.

One thing is definitely our fault, though.  xenodermus shows failures
on REL_12_STABLE and REL_13_STABLE, using debug LLVM 6 on x86.  I
couldn't reproduce this locally on (newer) debug LLVM, so I bugged
Andres for access to the host/libraries and chased it down.  There is
some type punning for a function parameter REL_13_STABLE and earlier,
removed by Andres in REL_14_STABLE, and when I back-patched my
refactoring I effectively back-patched a few changes from his commit
df99ddc70b97 that removed the type punning, but I should have brought
one more line from that commit to remove another trace of it.  See
attached.

---

From: Mark Wong <[email-masked]>
Date: Mon, 23 Oct 2023 08:27:23 -0700

> debug builds and backtraces.

I haven't gotten around to disabling llvm on any of my animals with llvm
< 7 yet.  Do you still want to hold on that?

> one more line from that commit to remove another trace of it.  See

Here are my list of llvm-config versions and distros for s390x and POWER
(didn't see any issues on aarch64 but I grabbed all the info at the same
time.)

s390x:

branta: 10.0.0 Ubuntu 20.04.4 LTS
cotinga: 6.0.0 Ubuntu 18.04.6 LTS
perch: 6.0.0 Ubuntu 18.04.6 LTS
sarus: 14.0.0 Ubuntu 22.04.1 LTS
aracari: 15.0.7 Red Hat Enterprise Linux 8.6
pipit: 15.0.7 Red Hat Enterprise Linux 8.6
lora: 15.0.7 Red Hat Enterprise Linux 9.2
mamushi: 15.0.7 Red Hat Enterprise Linux 9.2
pike: 11.0.1 Debian GNU/Linux 11
rinkhals: 11.0.1 Debian GNU/Linux 11

POWER:

bonito: 7.0.1 Fedora 29
cavefish: 6.0.0 Ubuntu 18.04.6 LTS
demoiselle: 5.0.1 openSUSE Leap 15.0
elasmobranch: 5.0.1 openSUSE Leap 15.0
babbler: 15.0.7 AlmaLinux 8.8
pytilia: 15.0.7 AlmaLinux 8.8
nicator: 15.0.7 AlmaLinux 9.2
twinspot: 15.0.7 AlmaLinux 9.2
cascabel: 11.0.1 Debian GNU/Linux 11
habu: 16.0.6 Fedora Linux 38
kingsnake: 16.0.6 Fedora Linux 38
krait: CentOS 16.0.6 Stream 8
lancehead: CentOS 16.0.6 Stream 8

aarch64:

boiga: 14.0.5 Fedora Linux 36
corzo: 14.0.5 Fedora Linux 36
desman: 16.0.6 Fedora Linux 38
motmot: 16.0.6 Fedora Linux 38
whinchat: 11.0.1 Debian GNU/Linux 11
jackdaw: 11.0.1 Debian GNU/Linux 11
blackneck: 7.0.1 Debian GNU/Linux 10
alimoche: 7.0.1 Debian GNU/Linux 10
bulbul: 15.0.7 AlmaLinux 8.8
broadbill: 15.0.7 AlmaLinux 8.8
oystercatcher: 15.0.7 AlmaLinux 9.2
potoo: 15.0.7 AlmaLinux 9.2
whiting: 6.0.0 Ubuntu 18.04.5 LTS
vimba: 6.0.0 Ubuntu 18.04.5 LTS
splitfin: 10.0.0 Ubuntu 20.04.6 LTS
rudd: 10.0.0 Ubuntu 20.04.6 LTS
turbot: 14.0.0 Ubuntu 22.04.3 LTS
shiner: 14.0.0 Ubuntu 22.04.3 LTS
ziege: 16.0.6 CentOS Stream 8
chevrotain: 11.0.1 Debian GNU/Linux 11

Regards,
Mark