From: Stas Kelvich <s.kelvich@postgrespro.ru>
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_120EA140-8B74-46F3-B06C-3CACD872D423"
Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\))
Subject: unused_oids script is broken with bsd sed
Message-Id: <37D774E4-FE1F-437E-B3D2-593F314B7505@postgrespro.ru>
Date: Wed, 25 Apr 2018 13:22:41 +0300
To: PostgreSQL Hackers <pgsql-hackers@postgresql.org>
Precedence: bulk


--Apple-Mail=_120EA140-8B74-46F3-B06C-3CACD872D423
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8

Hi.

BSD sed in macOS doesn't understand word boundary operator "\b". So =
after
372728b0d49 unused_oids doesn=E2=80=99t match all oids in new *.dat =
files marking
all of them as unused.

It is possible to write more portable regexps, e.g. change "\b" to
something like "^.*{*.*", but it seems easier for feature use to just
rewrite unused_oids in perl to match duplicate_oids. Also add in-place
complain about duplicates instead of running uniq through oids array.

--
Stas Kelvich
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


--Apple-Mail=_120EA140-8B74-46F3-B06C-3CACD872D423
Content-Disposition: attachment;
	filename=0001-Rewrite-unused_oids-in-perl.patch
Content-Type: application/octet-stream;
	x-unix-mode=0644;
	name="0001-Rewrite-unused_oids-in-perl.patch"
Content-Transfer-Encoding: quoted-printable

=46rom=20756aeca9f6bf48c795cb84de7a131e5db8a4b33e=20Mon=20Sep=2017=20=
00:00:00=202001=0AFrom:=20Stas=20Kelvich=20<stanconn@gmail.com>=0ADate:=20=
Wed,=2025=20Apr=202018=2013:00:28=20+0300=0ASubject:=20[PATCH]=20Rewrite=20=
unused_oids=20in=20perl=0A=0Aunused_oids=20script=20was=20broken=20with=20=
bsd=20sed=20after=20372728b0d49.=20It=20was=0Apossible=20to=20write=20=
more=20portable=20regexps=20but=20it=20seems=20easier=20to=20just=20=
rewrite=0Aunused_oids=20in=20perl=20to=20match=20duplicate_oids.=20Also=20=
add=20in-place=20complain=0Aabout=20duplicates=20instead=20of=20running=20=
uniq=20through=20oids=20array.=0A---=0A=20=
src/include/catalog/unused_oids=20|=2075=20=
++++++++++++++++++++++++-----------------=0A=201=20file=20changed,=2045=20=
insertions(+),=2030=20deletions(-)=0A=0Adiff=20--git=20=
a/src/include/catalog/unused_oids=20b/src/include/catalog/unused_oids=0A=
index=20f71222d50d..c225cbaa70=20100755=0A---=20=
a/src/include/catalog/unused_oids=0A+++=20=
b/src/include/catalog/unused_oids=0A@@=20-1,4=20+1,4=20@@=0A-#!/bin/sh=0A=
+#!/usr/bin/perl=0A=20#=0A=20#=20unused_oids=0A=20#=0A@@=20-16,42=20=
+16,57=20@@=0A=20#=09run=20this=20script=20in=20src/include/catalog.=0A=20=
#=0A=20=0A+use=20strict;=0A+use=20warnings;=0A=20=0A-AWK=3D"awk"=0A=
+BEGIN=0A+{=0A+=09@ARGV=20=3D=20(glob("pg_*.h"),=20glob("pg_*.dat"),=20=
qw(indexing.h=20toasting.h));=0A+}=0A=20=0A-#=20Get=20=
FirstBootstrapObjectId=20from=20access/transam.h=0A-FIRSTOBJECTID=3D`grep=20=
'#define[=20=09]*FirstBootstrapObjectId'=20../access/transam.h=20|=20=
$AWK=20'{=20print=20$3=20}'`=0A-export=20FIRSTOBJECTID=0A+my=20$AWK=20=3D=20=
'awk';=0A+my=20$FirstObjectId=20=3D`grep=20'#define[=20=09=
]*FirstBootstrapObjectId'=20../access/transam.h=20|=20$AWK=20'{=20print=20=
\$3=20}'`;=0A+if=20($FirstObjectId=20!~=20/^\d+$/)=0A+{=0A+=09die=20=
"Counld=20not=20find=20FirstBootstrapObjectId=20in=20=
../access/transam.h";=0A+}=0A+my=20@oids=20=3D=20($FirstObjectId);=0A=20=0A=
=20#=20this=20part=20(down=20to=20the=20uniq=20step)=20should=20match=20=
the=20duplicate_oids=20script=0A=20#=20note:=20we=20exclude=20=
BKI_BOOTSTRAP=20relations=20since=20they=20are=20expected=20to=20have=0A=20=
#=20matching=20data=20entries=20in=20pg_class.dat=20and=20pg_type.dat=0A=
+while=20(<>)=0A+{=0A+=09next=20if=20/^CATALOG\(.*BKI_BOOTSTRAP/;=0A+=09=
next=0A+=09=20=20unless=20/\boid=20*=3D>=20*'(\d+)'/=0A+=09=09=20=20||=20=
/^CATALOG\([^,]*,=20*(\d+).*BKI_ROWTYPE_OID\((\d+),/=0A+=09=09=20=20||=20=
/^CATALOG\([^,]*,=20*(\d+)/=0A+=09=09=20=20||=20/^DECLARE_INDEX\([^,]*,=20=
*(\d+)/=0A+=09=09=20=20||=20/^DECLARE_UNIQUE_INDEX\([^,]*,=20*(\d+)/=0A+=09=
=09=20=20||=20/^DECLARE_TOAST\([^,]*,=20*(\d+),=20*(\d+)/;=0A+=09push=20=
@oids,=20$1;=0A+=09push=20@oids,=20$2=20if=20$2;=0A+}=0A=20=0A-cat=20=
pg_*.h=20pg_*.dat=20toasting.h=20indexing.h=20|=0A-egrep=20-v=20-e=20=
'^CATALOG\(.*BKI_BOOTSTRAP'=20|=20\=0A-sed=20-n=09-e=20's/.*\boid=20*=3D>=20=
*'\''\([0-9][0-9]*\)'\''.*$/\1/p'=20\=0A-=09-e=20's/^CATALOG([^,]*,=20=
*\([0-9][0-9]*\).*BKI_ROWTYPE_OID(\([0-9][0-9]*\),.*$/\1,\2/p'=20\=0A-=09=
-e=20's/^CATALOG([^,]*,=20*\([0-9][0-9]*\).*$/\1/p'=20\=0A-=09-e=20=
's/^DECLARE_INDEX([^,]*,=20*\([0-9][0-9]*\).*$/\1/p'=20\=0A-=09-e=20=
's/^DECLARE_UNIQUE_INDEX([^,]*,=20*\([0-9][0-9]*\).*$/\1/p'=20\=0A-=09-e=20=
's/^DECLARE_TOAST([^,]*,=20*\([0-9][0-9]*\),=20=
*\([0-9][0-9]*\).*$/\1,\2/p'=20|=20\=0A-tr=20','=20'\n'=20|=20\=0A-sort=20=
-n=20|=20\=0A-uniq=20|=20\=0A-$AWK=20'=0A-BEGIN=20{=0A-=09last=20=3D=20=
0;=0A-}=0A-/^[0-9]/=20{=0A-=09if=20($1=20>=20last=20+=201)=20{=0A-=09=09=
if=20($1=20>=20last=20+=202)=20{=0A-=09=09=09print=20last=20+=201,=20=
"-",=20$1=20-=201;=0A-=09=09}=20else=20{=0A-=09=09=09print=20last=20+=20=
1;=0A+my=20$prev_oid=20=3D=200;=0A+foreach=20my=20$oid=20(sort=20{=20$a=20=
<=3D>=20$b=20}=20@oids)=0A+{=0A+=09if=20($oid=20>=20$prev_oid=20+=201)=0A=
+=09{=0A+=09=09if=20($oid=20>=20$prev_oid=20+=202)=0A+=09=09{=0A+=09=09=09=
print=20"@{[=20$prev_oid=20+=201=20]}=20-=20@{[=20$oid=20-=201=20]}\n";=0A=
+=09=09}=0A+=09=09else=0A+=09=09{=0A+=09=09=09print=20"@{[=20$prev_oid=20=
+=201=20]}\n";=0A=20=09=09}=0A=20=09}=0A-=09last=20=3D=20$1;=0A+=09elsif=20=
($oid=20=3D=3D=20$prev_oid)=0A+=09{=0A+=09=09print=20"Duplicate=20oid=20=
detected:=20$oid\n";=0A+=09=09exit=201;=0A+=09}=0A+=09$prev_oid=20=3D=20=
$oid;=0A=20}=0A-END=20{=0A-=09print=20last=20+=201,=20"-",=20=
ENVIRON["FIRSTOBJECTID"]-1;=0A-}'=0A--=20=0A2.16.2=0A=0A=

--Apple-Mail=_120EA140-8B74-46F3-B06C-3CACD872D423--