pg.ddx.io psycopg/psycopg GitHub issues and pull requests
help / color / mirror / Atom feed[psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
4+ messages / 2 participants
[nested] [flat]
* [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-16 23:41 "youdie006 (@youdie006)" <noreply+youdie006@github.com>
0 siblings, 0 replies; 4+ messages in thread
From: youdie006 (@youdie006) @ 2026-11-16 23:41 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
The C binary dumpers for int2 and int4 cast the value to `int16_t`/`int32_t` without a range check, so an int that doesn't fit is sent as a different, valid number. The Python implementation fails on the same value with a struct error. It is the situation 82a799a fixed for decimals, so this fixes it the same way: both implementations now raise `DataError`.
Against PostgreSQL 16, `PSYCOPG_IMPL=c`:
```
before after
COPY binary, set_types(["int2"]), 40000 stored -25536 DataError: int out of range for int2: 40000
insert %b Int2(40000) stored -25536 DataError: int out of range for int2: 40000
insert %b Int4(2**31) stored -2**31 DataError: int out of range for int4: 2147483648
insert %b Int4Range(-2**32+1, 10) stored [1,10) DataError: int out of range for int4: -4294967295
```
With `PSYCOPG_IMPL=python` the same calls raised `struct.error` before and raise the same `DataError` now. Over text the server already rejects these values.
The new tests in `tests/types/test_numeric.py` check the out-of-range values and the exact limits without a database; the out-of-range rows fail on master with both implementations. The full test suite passes against PostgreSQL 16 with both implementations, except a few `test_waiting.py` timing cases that fail on master too on this machine (WSL), a different set on each run. The pre-commit hooks pass on the changed files.
Written with AI assistance (Claude); I have reviewed the change.
inline patch 1: docs/news.rst, psycopg/psycopg/types/numeric.py, psycopg_c/psycopg_c/types/numeric.pyx and 1 more (+52/-3 lines) download inline-1.patch
diff --git a/docs/news.rst b/docs/news.rst
index 16ed74769..2829cc755 100644
--- a/docs/news.rst
+++ b/docs/news.rst
@@ -7,6 +7,17 @@
``psycopg`` release notes
=========================
+Future releases
+---------------
+
+Psycopg 3.3.10 (unreleased)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+- Raise `!DataError` dumping out-of-range ints to binary int2 or int4: the C
+ implementation sent a wrong value, the Python one failed with a struct error
+ (:ticket:`#1498`).
+
+
Current release
---------------
diff --git a/psycopg/psycopg/types/numeric.py b/psycopg/psycopg/types/numeric.py
index 58d356ecc..f009fdaca 100644
--- a/psycopg/psycopg/types/numeric.py
+++ b/psycopg/psycopg/types/numeric.py
@@ -168,6 +168,8 @@ class Int2BinaryDumper(Int2Dumper):
format = Format.BINARY
def dump(self, obj: int) -> Buffer | None:
+ if not -(2**15) <= obj < 2**15:
+ raise e.DataError(f"int out of range for int2: {obj}")
return pack_int2(obj)
@@ -175,6 +177,8 @@ class Int4BinaryDumper(Int4Dumper):
format = Format.BINARY
def dump(self, obj: int) -> Buffer | None:
+ if not -(2**31) <= obj < 2**31:
+ raise e.DataError(f"int out of range for int4: {obj}")
return pack_int4(obj)
diff --git a/psycopg_c/psycopg_c/types/numeric.pyx b/psycopg_c/psycopg_c/types/numeric.pyx
index e6a0f41fb..3315c4896 100644
--- a/psycopg_c/psycopg_c/types/numeric.pyx
+++ b/psycopg_c/psycopg_c/types/numeric.pyx
@@ -822,7 +822,10 @@ cdef Py_ssize_t dump_int_or_sub_to_text(
cdef Py_ssize_t dump_int_to_int2_binary(
obj, bytearray rv, Py_ssize_t offset
) except -1:
- cdef int16_t val = <int16_t>PyLong_AsLongLong(obj)
+ cdef long long lval = PyLong_AsLongLong(obj)
+ if not INT16_MIN <= lval <= INT16_MAX:
+ raise e.DataError(f"int out of range for int2: {obj}")
+ cdef int16_t val = <int16_t>lval
cdef int16_t *buf = <int16_t *>CDumper.ensure_size(rv, offset, sizeof(val))
cdef uint16_t beval = endian.htobe16(val) # swap bytes if needed
memcpy(buf, &beval, sizeof(beval))
@@ -832,7 +835,10 @@ cdef Py_ssize_t dump_int_to_int2_binary(
cdef Py_ssize_t dump_int_to_int4_binary(
obj, bytearray rv, Py_ssize_t offset
) except -1:
- cdef int32_t val = <int32_t>PyLong_AsLongLong(obj)
+ cdef long long lval = PyLong_AsLongLong(obj)
+ if not INT32_MIN <= lval <= INT32_MAX:
+ raise e.DataError(f"int out of range for int4: {obj}")
+ cdef int32_t val = <int32_t>lval
cdef int32_t *buf = <int32_t *>CDumper.ensure_size(rv, offset, sizeof(val))
cdef uint32_t beval = endian.htobe32(val) # swap bytes if needed
memcpy(buf, &beval, sizeof(beval))
diff --git a/tests/types/test_numeric.py b/tests/types/test_numeric.py
index 97003f84c..d616d6207 100644
--- a/tests/types/test_numeric.py
+++ b/tests/types/test_numeric.py
@@ -10,7 +10,8 @@
from psycopg import pq, sql
from psycopg.abc import Buffer
from psycopg.adapt import PyFormat, Transformer
-from psycopg.types.numeric import FloatLoader, Int8, Int8BinaryDumper, Int8Dumper
+from psycopg.types.numeric import FloatLoader, Int2, Int4, Int8, Int8BinaryDumper
+from psycopg.types.numeric import Int8Dumper
from ..fix_crdb import is_crdb, skip_crdb
@@ -91,6 +92,33 @@ def test_dump_int_numeric_binary_limits():
Transformer().dump_sequence([val], [PyFormat.AUTO])
+@pytest.mark.parametrize(
+ "wrapper, val",
+ [
+ (Int2, 2**15),
+ (Int2, -(2**15) - 1),
+ (Int2, 2**31),
+ (Int4, 2**31),
+ (Int4, -(2**31) - 1),
+ (Int4, 2**32 + 5),
+ ],
+)
+def test_dump_int_binary_out_of_range(wrapper, val):
+ name = wrapper.__name__.lower()
+ with pytest.raises(psycopg.DataError, match=f"out of range for {name}"):
+ Transformer().dump_sequence([wrapper(val)], [PyFormat.BINARY])
+
+
+@pytest.mark.parametrize(
+ "wrapper, val",
+ [(Int2, 2**15 - 1), (Int2, -(2**15)), (Int4, 2**31 - 1), (Int4, -(2**31))],
+)
+def test_dump_int_binary_range_limits(wrapper, val):
+ data = Transformer().dump_sequence([wrapper(val)], [PyFormat.BINARY])[0]
+ assert data is not None
+ assert int.from_bytes(data, "big", signed=True) == val
+
+
def test_dump_decimal_numeric_binary_limits():
# Largest number of digits before the decimal point (131072 = 32768 * 4)
for val, sign in [("1E+131071", 0x0000), ("-1E+131071", 0x4000)]:
^ permalink raw [nested|flat] 4+ messages in thread
* Re: [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-17 14:26 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2 siblings, 0 replies; 4+ messages in thread
From: dvarrazzo (@dvarrazzo) @ 2026-11-17 14:26 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
(on psycopg/psycopg/types/numeric.py:172)
Better try and raise `DataError ... from None` because setting the exception and not raising is a cheap operation in Python.
^ permalink raw [nested|flat] 4+ messages in thread
* Re: [psycopg/psycopg] PR #1498 (changes requested): fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-17 14:27 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2 siblings, 0 replies; 4+ messages in thread
From: dvarrazzo (@dvarrazzo) @ 2026-11-17 14:27 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
^ permalink raw [nested|flat] 4+ messages in thread
* Re: [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-20 00:01 ` "youdie006 (@youdie006)" <noreply+youdie006@github.com>
2 siblings, 0 replies; 4+ messages in thread
From: youdie006 (@youdie006) @ 2026-11-20 00:01 UTC (permalink / raw)
To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>
(on psycopg/psycopg/types/numeric.py)
Done in 6fd9977b, for both int2 and int4. The tests and the binary COPY check against PostgreSQL 16 still give the same `DataError` with both implementations.
^ permalink raw [nested|flat] 4+ messages in thread
end of thread, other threads:[~2026-11-20 00:01 UTC | newest]
Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 23:41 [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4 "youdie006 (@youdie006)" <noreply+youdie006@github.com>
2026-11-17 14:26 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2026-11-17 14:27 ` Re: [psycopg/psycopg] PR #1498 (changes requested): fix: raise DataError dumping out-of-range ints to binary int2 or int4 "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2026-11-20 00:01 ` "youdie006 (@youdie006)" <noreply+youdie006@github.com>
This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox