pg.ddx.io  psycopg/psycopg GitHub issues and pull requests  
help / color / mirror / Atom feed
[psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
4+ messages / 2 participants
[nested] [flat]

* [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-16 23:41 "youdie006 (@youdie006)" <noreply+youdie006@github.com>
  0 siblings, 0 replies; 4+ messages in thread

From: youdie006 (@youdie006) @ 2026-11-16 23:41 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

The C binary dumpers for int2 and int4 cast the value to `int16_t`/`int32_t` without a range check, so an int that doesn't fit is sent as a different, valid number. The Python implementation fails on the same value with a struct error. It is the situation 82a799a fixed for decimals, so this fixes it the same way: both implementations now raise `DataError`.

Against PostgreSQL 16, `PSYCOPG_IMPL=c`:

```
                                         before            after
COPY binary, set_types(["int2"]), 40000  stored -25536     DataError: int out of range for int2: 40000
insert %b Int2(40000)                    stored -25536     DataError: int out of range for int2: 40000
insert %b Int4(2**31)                    stored -2**31     DataError: int out of range for int4: 2147483648
insert %b Int4Range(-2**32+1, 10)        stored [1,10)     DataError: int out of range for int4: -4294967295
```

With `PSYCOPG_IMPL=python` the same calls raised `struct.error` before and raise the same `DataError` now. Over text the server already rejects these values.

The new tests in `tests/types/test_numeric.py` check the out-of-range values and the exact limits without a database; the out-of-range rows fail on master with both implementations. The full test suite passes against PostgreSQL 16 with both implementations, except a few `test_waiting.py` timing cases that fail on master too on this machine (WSL), a different set on each run. The pre-commit hooks pass on the changed files.

Written with AI assistance (Claude); I have reviewed the change.


inline patch 1: docs/news.rst, psycopg/psycopg/types/numeric.py, psycopg_c/psycopg_c/types/numeric.pyx and 1 more (+52/-3 lines) download inline-1.patchdiff --git a/docs/news.rst b/docs/news.rst index 16ed74769..2829cc755 100644 --- a/docs/news.rst +++ b/docs/news.rst @@ -7,6 +7,17 @@ ``psycopg`` release notes ========================= +Future releases +--------------- + +Psycopg 3.3.10 (unreleased) +^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +- Raise `!DataError` dumping out-of-range ints to binary int2 or int4: the C + implementation sent a wrong value, the Python one failed with a struct error + (:ticket:`#1498`). + + Current release --------------- diff --git a/psycopg/psycopg/types/numeric.py b/psycopg/psycopg/types/numeric.py index 58d356ecc..f009fdaca 100644 --- a/psycopg/psycopg/types/numeric.py +++ b/psycopg/psycopg/types/numeric.py @@ -168,6 +168,8 @@ class Int2BinaryDumper(Int2Dumper): format = Format.BINARY def dump(self, obj: int) -> Buffer | None: + if not -(2**15) <= obj < 2**15: + raise e.DataError(f"int out of range for int2: {obj}") return pack_int2(obj) @@ -175,6 +177,8 @@ class Int4BinaryDumper(Int4Dumper): format = Format.BINARY def dump(self, obj: int) -> Buffer | None: + if not -(2**31) <= obj < 2**31: + raise e.DataError(f"int out of range for int4: {obj}") return pack_int4(obj) diff --git a/psycopg_c/psycopg_c/types/numeric.pyx b/psycopg_c/psycopg_c/types/numeric.pyx index e6a0f41fb..3315c4896 100644 --- a/psycopg_c/psycopg_c/types/numeric.pyx +++ b/psycopg_c/psycopg_c/types/numeric.pyx @@ -822,7 +822,10 @@ cdef Py_ssize_t dump_int_or_sub_to_text( cdef Py_ssize_t dump_int_to_int2_binary( obj, bytearray rv, Py_ssize_t offset ) except -1: - cdef int16_t val = <int16_t>PyLong_AsLongLong(obj) + cdef long long lval = PyLong_AsLongLong(obj) + if not INT16_MIN <= lval <= INT16_MAX: + raise e.DataError(f"int out of range for int2: {obj}") + cdef int16_t val = <int16_t>lval cdef int16_t *buf = <int16_t *>CDumper.ensure_size(rv, offset, sizeof(val)) cdef uint16_t beval = endian.htobe16(val) # swap bytes if needed memcpy(buf, &beval, sizeof(beval)) @@ -832,7 +835,10 @@ cdef Py_ssize_t dump_int_to_int2_binary( cdef Py_ssize_t dump_int_to_int4_binary( obj, bytearray rv, Py_ssize_t offset ) except -1: - cdef int32_t val = <int32_t>PyLong_AsLongLong(obj) + cdef long long lval = PyLong_AsLongLong(obj) + if not INT32_MIN <= lval <= INT32_MAX: + raise e.DataError(f"int out of range for int4: {obj}") + cdef int32_t val = <int32_t>lval cdef int32_t *buf = <int32_t *>CDumper.ensure_size(rv, offset, sizeof(val)) cdef uint32_t beval = endian.htobe32(val) # swap bytes if needed memcpy(buf, &beval, sizeof(beval)) diff --git a/tests/types/test_numeric.py b/tests/types/test_numeric.py index 97003f84c..d616d6207 100644 --- a/tests/types/test_numeric.py +++ b/tests/types/test_numeric.py @@ -10,7 +10,8 @@ from psycopg import pq, sql from psycopg.abc import Buffer from psycopg.adapt import PyFormat, Transformer -from psycopg.types.numeric import FloatLoader, Int8, Int8BinaryDumper, Int8Dumper +from psycopg.types.numeric import FloatLoader, Int2, Int4, Int8, Int8BinaryDumper +from psycopg.types.numeric import Int8Dumper from ..fix_crdb import is_crdb, skip_crdb @@ -91,6 +92,33 @@ def test_dump_int_numeric_binary_limits(): Transformer().dump_sequence([val], [PyFormat.AUTO]) +@pytest.mark.parametrize( + "wrapper, val", + [ + (Int2, 2**15), + (Int2, -(2**15) - 1), + (Int2, 2**31), + (Int4, 2**31), + (Int4, -(2**31) - 1), + (Int4, 2**32 + 5), + ], +) +def test_dump_int_binary_out_of_range(wrapper, val): + name = wrapper.__name__.lower() + with pytest.raises(psycopg.DataError, match=f"out of range for {name}"): + Transformer().dump_sequence([wrapper(val)], [PyFormat.BINARY]) + + +@pytest.mark.parametrize( + "wrapper, val", + [(Int2, 2**15 - 1), (Int2, -(2**15)), (Int4, 2**31 - 1), (Int4, -(2**31))], +) +def test_dump_int_binary_range_limits(wrapper, val): + data = Transformer().dump_sequence([wrapper(val)], [PyFormat.BINARY])[0] + assert data is not None + assert int.from_bytes(data, "big", signed=True) == val + + def test_dump_decimal_numeric_binary_limits(): # Largest number of digits before the decimal point (131072 = 32768 * 4) for val, sign in [("1E+131071", 0x0000), ("-1E+131071", 0x4000)]:
^ permalink raw [nested|flat] 4+ messages in thread

* Re: [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-17 14:26 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: dvarrazzo (@dvarrazzo) @ 2026-11-17 14:26 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

(on psycopg/psycopg/types/numeric.py:172)

Better try and raise `DataError ... from None` because setting the exception and not raising is a cheap operation in Python.

^ permalink  raw  [nested|flat] 4+ messages in thread

* Re: [psycopg/psycopg] PR #1498 (changes requested): fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-17 14:27 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: dvarrazzo (@dvarrazzo) @ 2026-11-17 14:27 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>



^ permalink  raw  [nested|flat] 4+ messages in thread

* Re: [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4
@ 2026-11-20 00:01 ` "youdie006 (@youdie006)" <noreply+youdie006@github.com>
  2 siblings, 0 replies; 4+ messages in thread

From: youdie006 (@youdie006) @ 2026-11-20 00:01 UTC (permalink / raw)
  To: psycopg/psycopg <noreply+psycopg-psycopg@github.com>

(on psycopg/psycopg/types/numeric.py)

Done in 6fd9977b, for both int2 and int4. The tests and the binary COPY check against PostgreSQL 16 still give the same `DataError` with both implementations.

^ permalink  raw  [nested|flat] 4+ messages in thread


end of thread, other threads:[~2026-11-20 00:01 UTC | newest]

Thread overview: 4+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2026-11-16 23:41 [psycopg/psycopg] PR #1498: fix: raise DataError dumping out-of-range ints to binary int2 or int4 "youdie006 (@youdie006)" <noreply+youdie006@github.com>
2026-11-17 14:26 ` "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2026-11-17 14:27 ` Re: [psycopg/psycopg] PR #1498 (changes requested): fix: raise DataError dumping out-of-range ints to binary int2 or int4 "dvarrazzo (@dvarrazzo)" <noreply+dvarrazzo@github.com>
2026-11-20 00:01 ` "youdie006 (@youdie006)" <noreply+youdie006@github.com>

This inbox is served by DDX for PostgreSQL; see mirroring instructions
for how to clone and mirror all data and code used for this inbox