From: Stas Kelvich <s.kelvich@postgrespro.ru>
Message-Id: <26728623-7783-4103-9BED-08D443C1729F@postgrespro.ru>
Content-Type: multipart/mixed;
	boundary="Apple-Mail=_9F5CE368-5136-4E6D-A961-464B64B168AE"
Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\))
Subject: Re: unused_oids script is broken with bsd sed
Date: Wed, 25 Apr 2018 20:55:08 +0300
In-Reply-To: 
 <CAJVSVGU2w6rvTXghDe7-Y3x9Ca0833VmnC3HpaYKGLw4xznpJA@mail.gmail.com>
Cc: andreas@proxel.se,
 Tom Lane <tgl@sss.pgh.pa.us>,
 Andrew Dunstan <andrew.dunstan@2ndquadrant.com>,
 PostgreSQL Hackers <pgsql-hackers@postgresql.org>,
 David Fetter <david@fetter.org>
To: John Naylor <jcnaylor@gmail.com>
References: <37D774E4-FE1F-437E-B3D2-593F314B7505@postgrespro.ru>
 <34a54c54-9132-1254-2bc3-d3489fb79728@2ndQuadrant.com>
 <4473.1524665917@sss.pgh.pa.us>
 <4DE444D1-DDE9-460F-AC2F-EB1B73D94A2D@postgrespro.ru>
 <CAJVSVGU2w6rvTXghDe7-Y3x9Ca0833VmnC3HpaYKGLw4xznpJA@mail.gmail.com>
Precedence: bulk


--Apple-Mail=_9F5CE368-5136-4E6D-A961-464B64B168AE
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=us-ascii


> On 25 Apr 2018, at 17:55, John Naylor <jcnaylor@gmail.com> wrote:
>=20
> On 4/25/18, Stas Kelvich <s.kelvich@postgrespro.ru> wrote:
>>> On 25 Apr 2018, at 17:18, Tom Lane <tgl@sss.pgh.pa.us> wrote:
>>> I think we should rewrite
>>> both of them to use the Catalog.pm infrastructure.
>>=20
>> Okay, seems reasonable. I'll put shared code in Catalog.pm and
>> update patch.
>=20
> I don't think you need any new code in Catalog.pm, I believe the
> suggestion was just to use that module as a stable interface to the
> data. Looking at your patch, I'll mention that we have an idiom for
> extracting #define'd OID symbols, e.g.:
>=20
> my $FirstBootstrapObjectId =3D Catalog::FindDefinedSymbol(
> 	'access/transam.h', \@include_path, 'FirstBootstrapObjectId');
>=20
> This is preferred over using awk, which would have its own portability
> issues (Windows for starters).
>=20
> While I'm thinking out loud, it might be worthwhile to patch genbki.pl
> for the duplicate test, since they're run at the same time anyway (see
> catalog/Makefile), and we've already read all the data.
>=20
> -John Naylor
>=20


New version is attached. I've put iterator into Catalog.pm to avoid =
copy-pasting
it over two scripts. Also instead of greping through *.dat and *.h files =
I've
used parsers provided in Catalog module. That way should be more =
sustainable
to format changes.

Anyone who wanted to help with scripts -- feel free to rewrite.


--Apple-Mail=_9F5CE368-5136-4E6D-A961-464B64B168AE
Content-Disposition: attachment;
	filename=0001-Rewrite-unused_oids-in-perl.patch
Content-Type: application/octet-stream;
	x-unix-mode=0644;
	name="0001-Rewrite-unused_oids-in-perl.patch"
Content-Transfer-Encoding: quoted-printable

=46rom=20a86a01be7f35d2f71cdfc3e887d29e2d5edf820a=20Mon=20Sep=2017=20=
00:00:00=202001=0AFrom:=20Stas=20Kelvich=20<stanconn@gmail.com>=0ADate:=20=
Wed,=2025=20Apr=202018=2013:00:28=20+0300=0ASubject:=20[PATCH]=20Rewrite=20=
unused_oids=20in=20perl=0A=0Aunused_oids=20script=20was=20broken=20with=20=
bsd=20sed=20after=20372728b0d49.=20It=20was=0Apossible=20to=20write=20=
more=20portable=20regexps=20but=20it=20seems=20easier=20to=20just=20=
rewrite=0Aunused_oids=20in=20perl=20to=20match=20duplicate_oids.=20Also=20=
add=20in-place=20complain=0Aabout=20duplicates=20instead=20of=20running=20=
uniq=20through=20oids=20array.=0A---=0A=20src/backend/catalog/Catalog.pm=20=
=20=20=20=20|=2047=20++++++++++++++++++++++++++++=0A=20=
src/include/catalog/duplicate_oids=20|=2023=20+++++---------=0A=20=
src/include/catalog/unused_oids=20=20=20=20|=2063=20=
++++++++++++++++++--------------------=0A=203=20files=20changed,=2085=20=
insertions(+),=2048=20deletions(-)=0A=0Adiff=20--git=20=
a/src/backend/catalog/Catalog.pm=20b/src/backend/catalog/Catalog.pm=0A=
index=206305a2b362..5caf428b6a=20100644=0A---=20=
a/src/backend/catalog/Catalog.pm=0A+++=20=
b/src/backend/catalog/Catalog.pm=0A@@=20-40,7=20+40,9=20@@=20sub=20=
ParseHeader=0A=20=0A=20=09=09$catalog{columns}=20=3D=20[];=0A=20=09=09=
$catalog{toasting}=20=3D=20[];=0A+=09=09$catalog{toasting_oids}=20=3D=20=
[];=0A=20=09=09$catalog{indexing}=20=3D=20[];=0A+=09=09=
$catalog{indexing_oids}=20=3D=20[];=0A=20=09=09$catalog{client_code}=20=3D=
=20[];=0A=20=0A=20=09=09open(my=20$ifh,=20'<',=20$input_file)=20||=20die=20=
"$input_file:=20$!";=0A@@=20-89,6=20+91,7=20@@=20sub=20ParseHeader=0A=20=09=
=09=09=09my=20($toast_name,=20$toast_oid,=20$index_oid)=20=3D=20($1,=20=
$2,=20$3);=0A=20=09=09=09=09push=20@{=20$catalog{toasting}=20},=0A=20=09=09=
=09=09=20=20"declare=20toast=20$toast_oid=20$index_oid=20on=20=
$toast_name\n";=0A+=09=09=09=09push=20@{=20$catalog{toasting_oids}=20},=20=
($toast_oid,=20$index_oid);=0A=20=09=09=09}=0A=20=09=09=09elsif=20=
(/^DECLARE_(UNIQUE_)?INDEX\(\s*(\w+),\s*(\d+),\s*(.+)\)/)=0A=20=09=09=09=
{=0A@@=20-99,6=20+102,7=20@@=20sub=20ParseHeader=0A=20=09=09=09=09=09=
"declare=20%sindex=20%s=20%s=20%s\n",=0A=20=09=09=09=09=09$is_unique=20?=20=
'unique=20'=20:=20'',=0A=20=09=09=09=09=09$index_name,=20$index_oid,=20=
$using);=0A+=09=09=09=09push=20@{=20$catalog{indexing_oids}=20},=20=
($index_oid);=0A=20=09=09=09}=0A=20=09=09=09elsif=20(/^BUILD_INDICES/)=0A=
=20=09=09=09{=0A@@=20-389,4=20+393,47=20@@=20sub=20=
FindDefinedSymbolFromData=0A=20=09die=20"no=20definition=20found=20for=20=
$symbol\n";=0A=20}=0A=20=0A+sub=20FindAllOidsFromHeaders=0A+{=0A+=09my=20=
@input_files=20=3D=20@_;=0A+=09my=20@oids=20=3D=20();=0A+=0A+=09foreach=20=
my=20$header=20(@input_files)=0A+=09{=0A+=09=09$header=20=3D~=20=
/(.+)\.h$/=0A+=09=09or=20die=20"Input=20files=20need=20to=20be=20header=20=
files.\n";=0A+=09=09my=20$datfile=20=3D=20"$1.dat";=0A+=0A+=09=09my=20=
$catalog=20=3D=20Catalog::ParseHeader($header);=0A+=0A+=09=09push=20=
@oids,=20$catalog->{rowtype_oid}=20if=20($catalog->{rowtype_oid});=0A+=09=
=09push=20@oids,=20$catalog->{relation_oid}=20if=20=
($catalog->{relation_oid});=0A+=0A+=09=09#=20pg_class=20row=20oids=20are=20=
duplicated=20in=20typrelid=20of=20pg_type=0A+=09=09if=20=
(exists($catalog->{catname})=20and=20($catalog->{catname}=20eq=20=
'pg_class'))=0A+=09=09{=0A+=09=09=09next;=0A+=09=09}=0A+=0A+=09=09if=20=
(-e=20$datfile)=0A+=09=09{=0A+=09=09=09my=20$catdata=20=3D=20=
Catalog::ParseData($datfile,=20$catalog->{columns},=200);=0A+=09=09=09=
foreach=20my=20$row=20(@$catdata)=0A+=09=09=09{=0A+=09=09=09=09#=20=
filter=20entries=20for=20bootstrapped=20catalogs=20entries=20in=20=
pg_type=0A+=09=09=09=09if=20(($catalog->{catname}=20eq=20'pg_type')=20=
and=20($row->{typcategory}=20eq=20'C'))=0A+=09=09=09=09{=0A+=09=09=09=09=09=
next;=0A+=09=09=09=09}=0A+=09=09=09=09push=20@oids,=20$row->{oid}=20if=20=
defined=20$row->{oid};=0A+=09=09=09}=0A+=09=09}=0A+=0A+=09=09push=20=
@oids,=20@{=20$catalog->{toasting_oids}=20};=0A+=09=09push=20@oids,=20@{=20=
$catalog->{indexing_oids}=20};=0A+=09}=0A+=0A+=09return=20\@oids;=0A+}=0A=
+=0A=201;=0Adiff=20--git=20a/src/include/catalog/duplicate_oids=20=
b/src/include/catalog/duplicate_oids=0Aindex=208c143cf06f..c818dc8a67=20=
100755=0A---=20a/src/include/catalog/duplicate_oids=0A+++=20=
b/src/include/catalog/duplicate_oids=0A@@=20-1,27=20+1,20=20@@=0A=20=
#!/usr/bin/perl=0A=20=0A+use=20lib=20'../../backend/catalog/';=0A+use=20=
Catalog;=0A+=0A=20use=20strict;=0A=20use=20warnings;=0A=20=0A-BEGIN=0A-{=0A=
-=09@ARGV=20=3D=20(glob("pg_*.h"),=20glob("pg_*.dat"),=20qw(indexing.h=20=
toasting.h));=0A-}=0A+my=20@input_files=20=3D=20=20(glob("pg_*.h"),=20=
qw(indexing.h=20toasting.h));=0A+=0A+my=20@oids=20=3D=20@{=20=
Catalog::FindAllOidsFromHeaders(@input_files)=20};=0A=20=0A=20my=20=
%oidcounts;=0A=20=0A-while=20(<>)=0A+foreach=20my=20$oid=20(@oids)=0A=20=
{=0A-=09next=20if=20/^CATALOG\(.*BKI_BOOTSTRAP/;=0A-=09next=0A-=09=20=20=
unless=20/\boid=20*=3D>=20*'(\d+)'/=0A-=09=09=20=20||=20=
/^CATALOG\([^,]*,=20*(\d+).*BKI_ROWTYPE_OID\((\d+),/=0A-=09=09=20=20||=20=
/^CATALOG\([^,]*,=20*(\d+)/=0A-=09=09=20=20||=20/^DECLARE_INDEX\([^,]*,=20=
*(\d+)/=0A-=09=09=20=20||=20/^DECLARE_UNIQUE_INDEX\([^,]*,=20*(\d+)/=0A-=09=
=09=20=20||=20/^DECLARE_TOAST\([^,]*,=20*(\d+),=20*(\d+)/;=0A-=09=
$oidcounts{$1}++;=0A-=09$oidcounts{$2}++=20if=20$2;=0A+=09=
$oidcounts{$oid}++;=0A=20}=0A=20=0A=20my=20$found=20=3D=200;=0Adiff=20=
--git=20a/src/include/catalog/unused_oids=20=
b/src/include/catalog/unused_oids=0Aindex=20f71222d50d..4b0e16cc02=20=
100755=0A---=20a/src/include/catalog/unused_oids=0A+++=20=
b/src/include/catalog/unused_oids=0A@@=20-1,4=20+1,4=20@@=0A-#!/bin/sh=0A=
+#!/usr/bin/perl=0A=20#=0A=20#=20unused_oids=0A=20#=0A@@=20-15,43=20=
+15,40=20@@=0A=20#=0A=20#=09run=20this=20script=20in=20=
src/include/catalog.=0A=20#=0A+use=20lib=20'../../backend/catalog/';=0A=
+use=20Catalog;=0A=20=0A+use=20strict;=0A+use=20warnings;=0A=20=0A=
-AWK=3D"awk"=0A+my=20@input_files=20=3D=20(glob("pg_*.h"),=20=
qw(indexing.h=20toasting.h));=0A+my=20@oids;=0A=20=0A-#=20Get=20=
FirstBootstrapObjectId=20from=20access/transam.h=0A-FIRSTOBJECTID=3D`grep=20=
'#define[=20=09]*FirstBootstrapObjectId'=20../access/transam.h=20|=20=
$AWK=20'{=20print=20$3=20}'`=0A-export=20FIRSTOBJECTID=0A+push=20@oids,=20=
@{=20Catalog::FindAllOidsFromHeaders(@input_files)=20};=0A=20=0A-#=20=
this=20part=20(down=20to=20the=20uniq=20step)=20should=20match=20the=20=
duplicate_oids=20script=0A-#=20note:=20we=20exclude=20BKI_BOOTSTRAP=20=
relations=20since=20they=20are=20expected=20to=20have=0A-#=20matching=20=
data=20entries=20in=20pg_class.dat=20and=20pg_type.dat=0A+my=20=
$FirstBootstrapObjectId=20=3D=20Catalog::FindDefinedSymbol(=0A+=09=
'access/transam.h',=20[".."],=20'FirstBootstrapObjectId');=0A+push=20=
@oids,=20$FirstBootstrapObjectId;=0A=20=0A-cat=20pg_*.h=20pg_*.dat=20=
toasting.h=20indexing.h=20|=0A-egrep=20-v=20-e=20=
'^CATALOG\(.*BKI_BOOTSTRAP'=20|=20\=0A-sed=20-n=09-e=20's/.*\boid=20*=3D>=20=
*'\''\([0-9][0-9]*\)'\''.*$/\1/p'=20\=0A-=09-e=20's/^CATALOG([^,]*,=20=
*\([0-9][0-9]*\).*BKI_ROWTYPE_OID(\([0-9][0-9]*\),.*$/\1,\2/p'=20\=0A-=09=
-e=20's/^CATALOG([^,]*,=20*\([0-9][0-9]*\).*$/\1/p'=20\=0A-=09-e=20=
's/^DECLARE_INDEX([^,]*,=20*\([0-9][0-9]*\).*$/\1/p'=20\=0A-=09-e=20=
's/^DECLARE_UNIQUE_INDEX([^,]*,=20*\([0-9][0-9]*\).*$/\1/p'=20\=0A-=09-e=20=
's/^DECLARE_TOAST([^,]*,=20*\([0-9][0-9]*\),=20=
*\([0-9][0-9]*\).*$/\1,\2/p'=20|=20\=0A-tr=20','=20'\n'=20|=20\=0A-sort=20=
-n=20|=20\=0A-uniq=20|=20\=0A-$AWK=20'=0A-BEGIN=20{=0A-=09last=20=3D=20=
0;=0A-}=0A-/^[0-9]/=20{=0A-=09if=20($1=20>=20last=20+=201)=20{=0A-=09=09=
if=20($1=20>=20last=20+=202)=20{=0A-=09=09=09print=20last=20+=201,=20=
"-",=20$1=20-=201;=0A-=09=09}=20else=20{=0A-=09=09=09print=20last=20+=20=
1;=0A+=0A+my=20$prev_oid=20=3D=200;=0A+foreach=20my=20$oid=20(sort=20{=20=
$a=20<=3D>=20$b=20}=20@oids)=0A+{=0A+=09if=20($oid=20>=20$prev_oid=20+=20=
1)=0A+=09{=0A+=09=09if=20($oid=20>=20$prev_oid=20+=202)=0A+=09=09{=0A+=09=
=09=09print=20"@{[=20$prev_oid=20+=201=20]}=20-=20@{[=20$oid=20-=201=20=
]}\n";=0A=20=09=09}=0A+=09=09else=0A+=09=09{=0A+=09=09=09print=20"@{[=20=
$prev_oid=20+=201=20]}\n";=0A+=09=09}=0A+=09}=0A+=09elsif=20($oid=20=3D=3D=
=20$prev_oid)=0A+=09{=0A+=09=09print=20"Duplicate=20oid=20detected:=20=
$oid\n";=0A+=09=09exit=201;=0A=20=09}=0A-=09last=20=3D=20$1;=0A+=09=
$prev_oid=20=3D=20$oid;=0A=20}=0A-END=20{=0A-=09print=20last=20+=201,=20=
"-",=20ENVIRON["FIRSTOBJECTID"]-1;=0A-}'=0A--=20=0A2.16.2=0A=0A=

--Apple-Mail=_9F5CE368-5136-4E6D-A961-464B64B168AE
Content-Transfer-Encoding: 7bit
Content-Type: text/plain;
	charset=us-ascii


--
Stas Kelvich
Postgres Professional: http://www.postgrespro.com
The Russian Postgres Company


--Apple-Mail=_9F5CE368-5136-4E6D-A961-464B64B168AE--