Date: Wed, 23 Sep 2020 23:54:44 +0200
From: Karsten Hilbert <Karsten.Hilbert@gmx.net>
To: psycopg@lists.postgresql.org
Subject: iterating over DictRow
Message-ID: <20200923215444.GA25406@hermes.hilbert.loc>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Ma_X_il-Followup-to: d
Re_X_turn-receipt-to: Karsten.Hilbert@gmx.net
Di_X_sposition-Notification-To: Karsten.Hilbert@gmx.net
X-Confi_X_rm-Reading-To: Karsten.Hilbert@gmx.net
X-Pri_X_ority: 2 (High)
Sender: <karsten.hilbert@gmx.net>
X-Provags-ID: V03:K1:QTlXMAT9J16ftWm/Ns82LDlg8EUDTJcebHt3x1LENPRkTYgIoxX
 gpqnuISXcDsa3zLrH3HwGVXGSbkWtEmsOf8J9akShhjVe8xI4hvUbiN0udkDjRqWCcAFXt5
 /m7d20R5gcHfa7u4lIDd4htXCwYckRhSAfF7dmYL18KOM4Sh2P1v8uQdYiQIw3m4ie6A3tp
 seygjPHXa2EmYuayy6alQ==
X-Spam-Flag: NO
X-UI-Out-Filterresults: notjunk:1;V03:K0:WnU7gh7wigE=:D8FF7PDP8RNQIZtNqP+EXU
 fhzJyAj94GpWdim+W6Y9iDglWVlXVrAkiSDtowluC7DwQAinr8bX7fPstjgb/xnRPbuxN5TLS
 zl/FP0ge0yISqI6Cs36ezviP0AvRgW534VQ0OzckMBLVX+U4FFIJISZXnULGpTERshUgQpB5o
 G6c0ktCjsCRSqla+m8uyZWWWIqFmq8Ze8KNugq59S2wFCKDXhSVlWlchOL2SbvV2ay21Nkuu/
 eB+nq6I8n1xRiO24Xg85lR5zuJ9fib4Zk3odlCZ9ZOsrv1Od1184vg0B0YH4fiVxA71URpWCm
 m/ffQoYWqnDwPvii7SUdiPGEkPbysln8lgKKuRyEOsbKIAkyh+x5P3jGPIEP9H8i/GaTEKZ0Y
 GdTDxeP4pQGDc5nW2IaNc6yIgu6gcuEI/97q+UTllVuyVkJHPbtZFZR3vo8G8hUpRm25EY+ri
 ghvxqacXroDv4Ufar6TdLb3J5B/Pum2JAzw1dAzlEFjqiXNe/JCssMrdA1lJujgk7dqPLpKC+
 J5Y/TcOFVipDq8d1id8zq+pp1Ldufu97BTLg3miZg+UUvFuupR9HZBrl72XXAMeUL9BlD/zS1
 768gHeNl9qfCfCij0hKcjBC4R9KDzbVOXvz/RDbJm2thDxD79E75kpvxwOYo/E8DgV8h87Jg2
 QvQHxHqP3Pi7qztj1EVHCCM9rjxbFnPWkmTLudiiz5mOeWK9Btr6ENRKBULqrZtWj9tstp/d1
 rDgaKORglF7xI2JddvFYSiIdATlS0hBHMTV0jlV9iUsaIA4FYc93ktTBcqsGmNQe+LM3jusxn
 KU9EigJndBtStEvTJMDYCVLJs38mTt/4f4E5gs9PdSJqnfRg7x0C2ER15HChSwkSJaHH7GArK
 UsE9eMR+n9gbsabgc9Fz9t6Y5UWxzXyPhHWfn3kfgmKB5IoYuPZVLSr5Rgtrx4UDlKY+md74N
 3EYZDR8CcJjmMvhmL3WdUSzO+PhlsN/5G5u+F/2JL8L3+XlvNN+Fux1aIsRSXQMO3rT6eBRKL
 b+3Hk3alrkDzMwfdaKhcH78bGUbGZ78RkEt/sJlX7t6TG8ASrHRbCQvNIWXKk1lWS4eJbchT4
 9C4f4nkWQoWUFkOwK7rvRHG+xG+KvwI99gcMKS3cbTXc56MRIBoc5oppELoMFDv9ZiD6ImMkm
 oI98PdpV1mWGLkmzx9QSptbLz4r5dhL03kHGEsvL9bNPTh36WbCLu+DvK13HcmZDpA6TXkaML
 3iLYnv+ch5NnR+LFyfGPOKovbWpauD28cd0Fczg==
Content-Transfer-Encoding: quoted-printable
List-Id: <psycopg.lists.postgresql.org>
List-Help: <https://lists.postgresql.org/manage/>
List-Subscribe: <https://lists.postgresql.org/manage/>
List-Post: <mailto:psycopg@lists.postgresql.org>
List-Owner: <mailto:psycopg-owner@lists.postgresql.org>
List-Archive: <https://www.postgresql.org/list/psycopg>
Precedence: bulk

Dear all,

I cannot currently wrap my head around why I am seeing this:

	2020-09-23 23:30:23 gmConnectionPool.py::<module>() #87): psycopg2 module=
 version: 2.8.5 (dt dec pq3 ext)
	2020-09-23 23:30:23 gmConnectionPool.py::<module>() #88): PostgreSQL via =
DB-API module "<module 'psycopg2' from '/usr/lib/python3/dist-packages/psy=
copg2/__init__.py'>": API level 2.0, thread safety 2, parameter style "pyf=
ormat"
	2020-09-23 23:30:23 gmConnectionPool.py::<module>() #89): libpq version (=
compiled in): 120002
	2020-09-23 23:30:23 gmConnectionPool.py::<module>() #90): libpq version (=
loaded now) : 120004
	...
	2020-09-23 23:30:28 gmConnectionPool.py::__log_on_first_contact() #445): =
heed Prime Directive
	2020-09-23 23:30:28 gmConnectionPool.py::__log_on_first_contact() #457): =
PostgreSQL version (numeric): 11.7
	...
	2020-09-23 23:31:02 gmMacro.py::_get_variant_diagnoses() #2442): [[260, '=
L: Bewegungsapparat', False, 'B', 'clin.health_issue'], [260, 'K: aHT/HI',=
 False, 'B', 'clin.health_issue'], [260, 'D: Verdauung', False, None, 'cli=
n.health_issue']]
	2020-09-23 23:31:02 gmMacro.py::_get_variant_diagnoses() #2443): <class '=
psycopg2.extras.DictRow'>
	2020-09-23 23:31:02 gmMacro.py::_escape_dict() #2851): 260
	2020-09-23 23:31:02 gmMacro.py::__getitem__() #880): placeholder handling=
 error: diagnoses::		\item %(diagnosis)s::
	Traceback (most recent call last):
	  File "/home/ncq/Projekte/gm/git/gnumed/gnumed/Gnumed/wxpython/gmMacro.p=
y", line 869, in __getitem__
	    val =3D handler(data =3D options)
	  File "/home/ncq/Projekte/gm/git/gnumed/gnumed/Gnumed/wxpython/gmMacro.p=
y", line 2445, in _get_variant_diagnoses
	    return '\n'.join(template % self._escape_dict(dx, none_string =3D '?'=
, bool_strings =3D [_('yes'), _('no')]) for dx in selected)
	  File "/home/ncq/Projekte/gm/git/gnumed/gnumed/Gnumed/wxpython/gmMacro.p=
y", line 2445, in <genexpr>
	    return '\n'.join(template % self._escape_dict(dx, none_string =3D '?'=
, bool_strings =3D [_('yes'), _('no')]) for dx in selected)
	  File "/home/ncq/Projekte/gm/git/gnumed/gnumed/Gnumed/wxpython/gmMacro.p=
y", line 2852, in _escape_dict
	    val =3D the_dict[field]
	  File "/usr/lib/python3/dist-packages/psycopg2/extras.py", line 169, in =
__getitem__
	    return super(DictRow, self).__getitem__(x)
	IndexError: list index out of range

This line logs a list of DictRow's:

	2020-09-23 23:31:02 gmMacro.py::_get_variant_diagnoses() #2442): [[260, '=
L: Bewegungsapparat', False, 'B', 'clin.health_issue'], [260, 'K: aHT/HI',=
 False, 'B', 'clin.health_issue'], [260, 'D: Verdauung', False, None, 'cli=
n.health_issue']]

as evidenced here:

	2020-09-23 23:31:02 gmMacro.py::_get_variant_diagnoses() #2443): <class '=
psycopg2.extras.DictRow'>

The logging code:

	_log.debug('%s', selected)
	_log.debug('%s', type(selected[0]))

and then iterates over the list of DictRow's as per list
comprehension like so:

	return '\n'.join(template % self._escape_dict(dx, none_string =3D '?', bo=
ol_strings =3D [_('yes'), _('no')]) for dx in selected)

where _escape_dict() does this:

	def _escape_dict(self, the_dict=3DNone, date_format=3D'%Y %b %d  %H:%M', =
none_string=3D'', bool_strings=3DNone):
		data =3D {}
		for field in the_dict:
			_log.debug('%s', field)
			val =3D the_dict[field]
			if val is None:
				...
			if isinstance(val, bool):
				...
			if isinstance(val, datetime.datetime):
				...
			if self.__esc_style in ['latex', 'tex']:
				...
			elif self.__esc_style in ['xetex', 'xelatex']:
				...
		return data

Iterating over the_dict should work for lists (produces
values) OR dicts (produces keys, under py3). It seems as if
the line

	for field in the_dict:			# I hoped to iterate over the keys =3D column na=
mes

somehow "decides": "currently we are treating the_dict as a
list (after all, it is a DictRow, which _can_ be treated as a
list) which then "turns" the dict-style access in

	val =3D the_dict[field]

into a list access.

Somehow I have a feeling that due to the type of "field"
=2D- it being integer (namely, a primary key) -- forces

	the_dict[field]

to attempt a list-style index based access (which fails, due
to there not being 260 columns in the SQL query result :-)

Solutions that come to mind:

	_Must_ I use RealDictCursor() to safely avoid this trap ?

	(or else dict()ify DictRow's as needed)

Any insights to be had ?

Thanks,
Karsten
=2D-
GPG  40BE 5B0E C98E 1713 AFA6  5BC0 3BEA AC80 7D4F C89B