pg.ddx.io  pgvector/pgvector GitHub issues and pull requests  
help / color / mirror / Atom feed
From: abhijain1204fujitsu (@abhijain1204fujitsu) <noreply+abhijain1204fujitsu@github.com>
To: pgvector/pgvector <noreply+pgvector-pgvector@github.com>
Subject: [pgvector/pgvector] PR #1067: [Arm CPU] Add SVE2 distance functions for halfvec on AArch64
Date: Wed, 18 Nov 2026 06:28:07 +0000
Message-ID: <gh-pgvector-pgvector-1067@github.com> (raw)

### Description

This adds SVE2 implementations of the four halfvec distance functions (L2, inner product, cosine, L1), selected at runtime on AArch64 Linux CPUs that support SVE2 (e.g. AWS Graviton4, Neoverse V2/V3). Other CPUs, compilers, and platforms keep the existing code.

### Approach
- No FP16→FP32 conversion instructions. FMLALB/FMLALT multiply the even/odd P16 lanes and accumulate into FP32 in one instruction. An FP16×FP16 product is exact in FP32, so results match convert-then-FMA.
- L2 / L1 differences: FMLALB(0, a, 1.0) followed by FMLSLB(·, b, 1.0) gives (float)a - (float)b with a single rounding, bit-identical to converting both values and subtracting.
- Multiple independent accumulators; WHILELT-predicated tail (no scalar loop).
- For dim < 32 the existing code is used, since the fixed per-call overhead outweighs the gain (measured).

### Dispatch
Follows the existing F16C pattern in HalfvecInit():

- Kernels use __attribute__((target("+sve2"))) ("sve2" for Clang), so no global -march flag is needed and the extension still runs on any AArch64 CPU.
- Selected at runtime with getauxval(AT_HWCAP2) & HWCAP2_SVE2.
- Enabled only for GCC >= 12 / Clang >= 16 on Linux. Older Clang rejects <arm_sve.h> without SVE enabled for the whole translation unit (verified with Clang 15). GCC 11 also worked in testing, but I kept the cutoff conservative.
- DISABLE_DISPATCH disables it, as for x86.

The SVE detection macros are added to halfvec.h next to the existing dispatch macros. They also define TARGET_SVE / SVE_RUNTIME_SUPPORTED() for follow-up changes (SVE for bit and vector distances); happy to drop those here if you'd prefer them added with those changes.

### Testing
- make installcheck passes.
- Standalone test against a double-precision reference for dims 1–1100, on hardware and under QEMU at 128/256/512/2048-bit vector lengths.
- Dispatch verified for every combination of GCC 11/12/14, Clang 15/16 and CPU models Neoverse N1 (no SVE), V1 (SVE only), N2 (SVE2), A64FX: SVE2 is used only when both compiler and CPU support it.
- x86 build unaffected.

### Benchmarks
AWS EC2, Neoverse V3 (192 vCPUs), PostgreSQL 17, both builds with OPTFLAGS="", NUMA-pinned. Baseline is current master (auto-vectorized NEON).

**Kernels** (ns per call, data in cache)

### Performance Results

| Dims | L2 | Inner Product | Cosine | L1 |
|---:|---:|---:|---:|---:|
| 384  | 64.7 → 35.1 **(1.84x)** | 61.5 → 16.9 **(3.63x)** | 80.9 → 39.8 **(2.03x)** | 58.7 → 40.6 **(1.45x)** |
| 768  | 129.2 → 61.2 **(2.11x)** | 122.8 → 33.2 **(3.69x)** | 159.3 → 67.7 **(2.35x)** | 117.0 → 77.5 **(1.51x)** |
| 1536 | 258.2 → 112.7 **(2.29x)** | 245.0 → 69.7 **(3.52x)** | 315.9 → 127.1 **(2.49x)** | 233.5 → 155.3 **(1.50x)** |

**End-to-end:** DBpedia OpenAI embeddings, 1M × 1536, halfvec table + HNSW (m=16, ef_construction=64), 1,000 held-out queries, ef_search=100, k=10, median of 3 runs

### Performance Summary

| Benchmark | Cosine | L2 | Inner Product |
|---|---:|---:|---:|
| Exact Scan | **1.38x** | **1.21x** | **1.34x** |
| HNSW Index Build | **1.10x** | **1.11x** | **1.11x** |
| QPS, 32 Clients | **1.06x** | **1.06x** | **1.09x** |
| Recall@10 | Unchanged | Unchanged | Unchanged |

**VectorDBBench** Performance1536D500K (OpenAI 500K × 1536, halfvec table + index, ef_search=200): index build 1.10x, max QPS 1.09x, QPS 1.07–1.17x across 1–64 clients and ef_search 40/100/200; recall unchanged.

This PR is Co-Authored by @mohakgoel1

inline patch 1: src/halfutils.c, src/halfvec.h (+271/-1 lines) download inline-1.patchdiff --git a/src/halfutils.c b/src/halfutils.c index 2df4995c0..fe5d56802 100644 --- a/src/halfutils.c +++ b/src/halfutils.c @@ -239,6 +239,230 @@ HalfvecL1DistanceF16c(int dim, half * ax, half * bx) } #endif +#ifdef USE_SVE +#include <arm_sve.h> + +#define SVE_HSUM4(pg, a0, a1, a2, a3) \ + svaddv_f32((pg), svadd_f32_x((pg), svadd_f32_x((pg), (a0), (a1)), svadd_f32_x((pg), (a2), (a3)))) + +#define SVE2_MIN_DIM 32 + +#define SVE2_DIFF_BOT(z, one, a, b) svmlslb_f32(svmlalb_f32((z), (a), (one)), (b), (one)) +#define SVE2_DIFF_TOP(z, one, a, b) svmlslt_f32(svmlalt_f32((z), (a), (one)), (b), (one)) + +TARGET_SVE2 static float +HalfvecL2SquaredDistanceSve2(int dim, half * ax, half * bx) +{ + const float16_t *a = (const float16_t *) ax; + const float16_t *b = (const float16_t *) bx; + const svbool_t all16 = svptrue_b16(); + const svbool_t all32 = svptrue_b32(); + const svfloat16_t one = svdup_n_f16(1.0); + const svfloat32_t z = svdup_n_f32(0.0f); + const int vl = (int) svcnth(); + int i = 0; + svfloat32_t acc0 = z; + svfloat32_t acc1 = z; + svfloat32_t acc2 = z; + svfloat32_t acc3 = z; + + if (dim < SVE2_MIN_DIM) + return HalfvecL2SquaredDistanceDefault(dim, ax, bx); + + for (; i + 2 * vl <= dim; i += 2 * vl) + { + svfloat16_t a0 = svld1_f16(all16, a + i); + svfloat16_t b0 = svld1_f16(all16, b + i); + svfloat16_t a1 = svld1_f16(all16, a + i + vl); + svfloat16_t b1 = svld1_f16(all16, b + i + vl); + svfloat32_t d0 = SVE2_DIFF_BOT(z, one, a0, b0); + svfloat32_t d1 = SVE2_DIFF_TOP(z, one, a0, b0); + svfloat32_t d2 = SVE2_DIFF_BOT(z, one, a1, b1); + svfloat32_t d3 = SVE2_DIFF_TOP(z, one, a1, b1); + + acc0 = svmla_f32_x(all32, acc0, d0, d0); + acc1 = svmla_f32_x(all32, acc1, d1, d1); + acc2 = svmla_f32_x(all32, acc2, d2, d2); + acc3 = svmla_f32_x(all32, acc3, d3, d3); + } + + for (; i < dim; i += vl) + { + svbool_t pg = svwhilelt_b16_s32(i, dim); + svfloat16_t a0 = svld1_f16(pg, a + i); + svfloat16_t b0 = svld1_f16(pg, b + i); + svfloat32_t d0 = SVE2_DIFF_BOT(z, one, a0, b0); + svfloat32_t d1 = SVE2_DIFF_TOP(z, one, a0, b0); + + acc0 = svmla_f32_x(all32, acc0, d0, d0); + acc1 = svmla_f32_x(all32, acc1, d1, d1); + } + + return SVE_HSUM4(all32, acc0, acc1, acc2, acc3); +} + +TARGET_SVE2 static float +HalfvecInnerProductSve2(int dim, half * ax, half * bx) +{ + const float16_t *a = (const float16_t *) ax; + const float16_t *b = (const float16_t *) bx; + const svbool_t all16 = svptrue_b16(); + const svbool_t all32 = svptrue_b32(); + const int vl = (int) svcnth(); + int i = 0; + svfloat32_t z = svdup_n_f32(0.0f); + svfloat32_t acc0 = z, acc1 = z, acc2 = z, acc3 = z; + svfloat32_t acc4 = z, acc5 = z, acc6 = z, acc7 = z; + + if (dim < SVE2_MIN_DIM) + return HalfvecInnerProductDefault(dim, ax, bx); + + for (; i + 4 * vl <= dim; i += 4 * vl) + { + svfloat16_t a0 = svld1_f16(all16, a + i); + svfloat16_t b0 = svld1_f16(all16, b + i); + svfloat16_t a1 = svld1_f16(all16, a + i + vl); + svfloat16_t b1 = svld1_f16(all16, b + i + vl); + svfloat16_t a2 = svld1_f16(all16, a + i + 2 * vl); + svfloat16_t b2 = svld1_f16(all16, b + i + 2 * vl); + svfloat16_t a3 = svld1_f16(all16, a + i + 3 * vl); + svfloat16_t b3 = svld1_f16(all16, b + i + 3 * vl); + + acc0 = svmlalb_f32(acc0, a0, b0); + acc1 = svmlalt_f32(acc1, a0, b0); + acc2 = svmlalb_f32(acc2, a1, b1); + acc3 = svmlalt_f32(acc3, a1, b1); + acc4 = svmlalb_f32(acc4, a2, b2); + acc5 = svmlalt_f32(acc5, a2, b2); + acc6 = svmlalb_f32(acc6, a3, b3); + acc7 = svmlalt_f32(acc7, a3, b3); + } + + for (; i < dim; i += vl) + { + svbool_t pg = svwhilelt_b16_s32(i, dim); + svfloat16_t a0 = svld1_f16(pg, a + i); + svfloat16_t b0 = svld1_f16(pg, b + i); + + acc0 = svmlalb_f32(acc0, a0, b0); + acc1 = svmlalt_f32(acc1, a0, b0); + } + + acc0 = svadd_f32_x(all32, acc0, acc4); + acc1 = svadd_f32_x(all32, acc1, acc5); + acc2 = svadd_f32_x(all32, acc2, acc6); + acc3 = svadd_f32_x(all32, acc3, acc7); + return SVE_HSUM4(all32, acc0, acc1, acc2, acc3); +} + +TARGET_SVE2 static double +HalfvecCosineSimilaritySve2(int dim, half * ax, half * bx) +{ + const float16_t *a = (const float16_t *) ax; + const float16_t *b = (const float16_t *) bx; + const svbool_t all16 = svptrue_b16(); + const svbool_t all32 = svptrue_b32(); + const int vl = (int) svcnth(); + int i = 0; + svfloat32_t zero = svdup_n_f32(0.0f); + svfloat32_t s0 = zero, s1 = zero, s2 = zero, s3 = zero; + svfloat32_t na0 = zero, na1 = zero, na2 = zero, na3 = zero; + svfloat32_t nb0 = zero, nb1 = zero, nb2 = zero, nb3 = zero; + float similarity; + float norma; + float normb; + + if (dim < SVE2_MIN_DIM) + return HalfvecCosineSimilarityDefault(dim, ax, bx); + + for (; i + 2 * vl <= dim; i += 2 * vl) + { + svfloat16_t a0 = svld1_f16(all16, a + i); + svfloat16_t b0 = svld1_f16(all16, b + i); + svfloat16_t a1 = svld1_f16(all16, a + i + vl); + svfloat16_t b1 = svld1_f16(all16, b + i + vl); + + s0 = svmlalb_f32(s0, a0, b0); + s1 = svmlalt_f32(s1, a0, b0); + s2 = svmlalb_f32(s2, a1, b1); + s3 = svmlalt_f32(s3, a1, b1); + na0 = svmlalb_f32(na0, a0, a0); + na1 = svmlalt_f32(na1, a0, a0); + na2 = svmlalb_f32(na2, a1, a1); + na3 = svmlalt_f32(na3, a1, a1); + nb0 = svmlalb_f32(nb0, b0, b0); + nb1 = svmlalt_f32(nb1, b0, b0); + nb2 = svmlalb_f32(nb2, b1, b1); + nb3 = svmlalt_f32(nb3, b1, b1); + } + + for (; i < dim; i += vl) + { + svbool_t pg = svwhilelt_b16_s32(i, dim); + svfloat16_t a0 = svld1_f16(pg, a + i); + svfloat16_t b0 = svld1_f16(pg, b + i); + + s0 = svmlalb_f32(s0, a0, b0); + s1 = svmlalt_f32(s1, a0, b0); + na0 = svmlalb_f32(na0, a0, a0); + na1 = svmlalt_f32(na1, a0, a0); + nb0 = svmlalb_f32(nb0, b0, b0); + nb1 = svmlalt_f32(nb1, b0, b0); + } + + similarity = SVE_HSUM4(all32, s0, s1, s2, s3); + norma = SVE_HSUM4(all32, na0, na1, na2, na3); + normb = SVE_HSUM4(all32, nb0, nb1, nb2, nb3); + + return (double) similarity / sqrt((double) norma * (double) normb); +} + +TARGET_SVE2 static float +HalfvecL1DistanceSve2(int dim, half * ax, half * bx) +{ + const float16_t *a = (const float16_t *) ax; + const float16_t *b = (const float16_t *) bx; + const svbool_t all16 = svptrue_b16(); + const svbool_t all32 = svptrue_b32(); + const svfloat16_t one = svdup_n_f16(1.0); + const svfloat32_t z = svdup_n_f32(0.0f); + const int vl = (int) svcnth(); + int i = 0; + svfloat32_t acc0 = z; + svfloat32_t acc1 = z; + svfloat32_t acc2 = z; + svfloat32_t acc3 = z; + + if (dim < SVE2_MIN_DIM) + return HalfvecL1DistanceDefault(dim, ax, bx); + + for (; i + 2 * vl <= dim; i += 2 * vl) + { + svfloat16_t a0 = svld1_f16(all16, a + i); + svfloat16_t b0 = svld1_f16(all16, b + i); + svfloat16_t a1 = svld1_f16(all16, a + i + vl); + svfloat16_t b1 = svld1_f16(all16, b + i + vl); + + acc0 = svadd_f32_x(all32, acc0, svabs_f32_x(all32, SVE2_DIFF_BOT(z, one, a0, b0))); + acc1 = svadd_f32_x(all32, acc1, svabs_f32_x(all32, SVE2_DIFF_TOP(z, one, a0, b0))); + acc2 = svadd_f32_x(all32, acc2, svabs_f32_x(all32, SVE2_DIFF_BOT(z, one, a1, b1))); + acc3 = svadd_f32_x(all32, acc3, svabs_f32_x(all32, SVE2_DIFF_TOP(z, one, a1, b1))); + } + + for (; i < dim; i += vl) + { + svbool_t pg = svwhilelt_b16_s32(i, dim); + svfloat16_t a0 = svld1_f16(pg, a + i); + svfloat16_t b0 = svld1_f16(pg, b + i); + + acc0 = svadd_f32_x(all32, acc0, svabs_f32_x(all32, SVE2_DIFF_BOT(z, one, a0, b0))); + acc1 = svadd_f32_x(all32, acc1, svabs_f32_x(all32, SVE2_DIFF_TOP(z, one, a0, b0))); + } + + return SVE_HSUM4(all32, acc0, acc1, acc2, acc3); +} +#endif /* USE_SVE */ + #ifdef HALFVEC_DISPATCH #define CPU_FEATURE_FMA (1 << 12) #define CPU_FEATURE_OSXSAVE (1 << 27) @@ -297,4 +521,14 @@ HalfvecInit(void) HalfvecL1Distance = HalfvecL1DistanceF16c; } #endif -} + +#ifdef USE_SVE + if (SVE2_RUNTIME_SUPPORTED()) + { + HalfvecL2SquaredDistance = HalfvecL2SquaredDistanceSve2; + HalfvecInnerProduct = HalfvecInnerProductSve2; + HalfvecCosineSimilarity = HalfvecCosineSimilaritySve2; + HalfvecL1Distance = HalfvecL1DistanceSve2; + } +#endif +} \ No newline at end of file diff --git a/src/halfvec.h b/src/halfvec.h index 8ff2cbfbf..cd00e02e7 100644 --- a/src/halfvec.h +++ b/src/halfvec.h @@ -38,6 +38,42 @@ #define USE__GET_CPUID #endif +#ifndef DISABLE_DISPATCH +#if defined(__aarch64__) && defined(__linux__) && defined(__has_include) +#if __has_include(<arm_sve.h>) && __has_include(<sys/auxv.h>) +#if (defined(__clang__) && __clang_major__ >= 16) || \ + (!defined(__clang__) && defined(__GNUC__) && __GNUC__ >= 12) +#define USE_SVE +#endif +#endif +#endif +#endif + +#ifdef USE_SVE +#include <sys/auxv.h> + +#ifndef AT_HWCAP2 +#define AT_HWCAP2 26 +#endif +#ifndef HWCAP_SVE +#define HWCAP_SVE (1 << 22) +#endif +#ifndef HWCAP2_SVE2 +#define HWCAP2_SVE2 (1 << 1) +#endif + +#if defined(__clang__) +#define TARGET_SVE __attribute__((target("sve"))) +#define TARGET_SVE2 __attribute__((target("sve2"))) +#else +#define TARGET_SVE __attribute__((target("+sve"))) +#define TARGET_SVE2 __attribute__((target("+sve2"))) +#endif + +#define SVE_RUNTIME_SUPPORTED() ((getauxval(AT_HWCAP) & HWCAP_SVE) != 0) +#define SVE2_RUNTIME_SUPPORTED() ((getauxval(AT_HWCAP2) & HWCAP2_SVE2) != 0) +#endif + #if defined(USE_DISPATCH) #define HALFVEC_DISPATCH #endif

view thread (7+ messages)  latest in thread

Message-ID: <gh-pgvector-pgvector-1067@github.com>
Permalink:  ../gh-pgvector-pgvector-1067@github.com/

 ·  · 


This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox