Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1f5Ept-0001X5-8t for pgsql-committers@arkaria.postgresql.org; Sun, 08 Apr 2018 18:14:41 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1f5Epr-0004B5-V4 for pgsql-committers@arkaria.postgresql.org; Sun, 08 Apr 2018 18:14:39 +0000 Received: from makus.postgresql.org ([2001:4800:1501:1::229]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1f5Epr-0004Aw-I1 for pgsql-committers@lists.postgresql.org; Sun, 08 Apr 2018 18:14:39 +0000 Received: from hook.sigaev.ru ([93.180.27.50]) by makus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA384:256) (Exim 4.89) (envelope-from ) id 1f5Epk-00085X-65 for pgsql-committers@lists.postgresql.org; Sun, 08 Apr 2018 18:14:38 +0000 Received: from zorg.sigaev.ru (95-25-2-27.broadband.corbina.ru [95.25.2.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: teodor) by hook.sigaev.ru (Postfix) with ESMTPSA id C7255B19192; Sun, 8 Apr 2018 21:14:29 +0300 (MSK) Subject: Re: pgsql: Replace our traditional initial-catalog-data format with a bette To: Tom Lane , pgsql-committers@lists.postgresql.org References: From: Teodor Sigaev Message-ID: Date: Sun, 8 Apr 2018 21:14:29 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=KOI8-R; format=flowed Content-Transfer-Encoding: 7bit List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk Seems, gaur animal doesn't too long shell command. Could "find | xargs" help here? cp ./*.h '/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/ || exit; \ chmod 644 '/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/*.h || exit; \ for dir in access bootstrap catalog commands common datatype executor fe_utils foreign lib libpq mb nodes optimizer parser postmaster regex replication rewrite statistics storage tcop snowball snowball/libstemmer tsearch tsearch/dicts utils port port/atomics port/win32 port/win32_msvc port/win32_msvc/sys port/win32/arpa port/win32/netinet port/win32/sys portability; do \ cp ./$dir/*.h '/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/$dir/ || exit; \ chmod 644 '/home/bfarm/bf-data/HEAD/pgsql.build/tmp_install/home/bfarm/bf-data/HEAD/inst/include/postgresql/server'/$dir/*.h || exit; \ done /bin/sh[5]: /usr/bin/chmod: The parameter list is too long. https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=gaur&dt=2018-04-08%2017%3A20%3A57 Tom Lane wrote: > Replace our traditional initial-catalog-data format with a better design. > > Historically, the initial catalog data to be installed during bootstrap > has been written in DATA() lines in the catalog header files. This had > lots of disadvantages: the format was badly underdocumented, it was > very difficult to edit the data in any mechanized way, and due to the > lack of any abstraction the data was verbose, hard to read/understand, > and easy to get wrong. > > Hence, move this data into separate ".dat" files and represent it in a way > that can easily be read and rewritten by Perl scripts. The new format is > essentially "key => value" for each column; while it's a bit repetitive, > explicit labeling of each value makes the data far more readable and less > error-prone. Provide a way to abbreviate entries by omitting field values > that match a specified default value for their column. This allows removal > of a large amount of repetitive boilerplate and also lowers the barrier to > adding new columns. > > Also teach genbki.pl how to translate symbolic OID references into > numeric OIDs for more cases than just "regproc"-like pg_proc references. > It can now do that for regprocedure-like references (thus solving the > problem that regproc is ambiguous for overloaded functions), operators, > types, opfamilies, opclasses, and access methods. Use this to turn > nearly all OID cross-references in the initial data into symbolic form. > This represents a very large step forward in readability and error > resistance of the initial catalog data. It should also reduce the > difficulty of renumbering OID assignments in uncommitted patches. > > Also, solve the longstanding problem that frontend code that would like to > use OID macros and other information from the catalog headers often had > difficulty with backend-only code in the headers. To do this, arrange for > all generated macros, plus such other declarations as we deem fit, to be > placed in "derived" header files that are safe for frontend inclusion. > (Once clients migrate to using these pg_*_d.h headers, it will be possible > to get rid of the pg_*_fn.h headers, which only exist to quarantine code > away from clients. That is left for follow-on patches, however.) > > The now-automatically-generated macros include the Anum_xxx and Natts_xxx > constants that we used to have to update by hand when adding or removing > catalog columns. > > Replace the former manual method of generating OID macros for pg_type > entries with an automatic method, ensuring that all built-in types have > OID macros. (But note that this patch does not change the way that > OID macros for pg_proc entries are built and used. It's not clear that > making that match the other catalogs would be worth extra code churn.) > > Add SGML documentation explaining what the new data format is and how to > work with it. > > Despite being a very large change in the catalog headers, there is no > catversion bump here, because postgres.bki and related output files > haven't changed at all. > > John Naylor, based on ideas from various people; review and minor > additional coding by me; previous review by Alvaro Herrera > > Discussion: https://postgr.es/m/CAJVSVGWO48JbbwXkJz_yBFyGYW-M9YWxnPdxJBUosDC9ou_F0Q@mail.gmail.com > > Branch > ------ > master > > Details > ------- > https://git.postgresql.org/pg/commitdiff/372728b0d49552641f0ea83d9d2e08817de038fa > > Modified Files > -------------- > doc/src/sgml/bki.sgml | 661 +- > doc/src/sgml/libpq.sgml | 2 +- > src/Makefile | 5 + > src/backend/Makefile | 34 +- > src/backend/catalog/.gitignore | 2 + > src/backend/catalog/Catalog.pm | 313 +- > src/backend/catalog/Makefile | 87 +- > src/backend/catalog/README | 111 - > src/backend/catalog/genbki.pl | 478 +- > src/backend/utils/Gen_fmgrtab.pl | 62 +- > src/backend/utils/Makefile | 15 +- > src/include/Makefile | 6 +- > src/include/catalog/.gitignore | 2 + > src/include/catalog/Makefile | 31 + > src/include/catalog/duplicate_oids | 6 +- > src/include/catalog/genbki.h | 29 +- > src/include/catalog/indexing.h | 2 +- > src/include/catalog/pg_aggregate.dat | 598 ++ > src/include/catalog/pg_aggregate.h | 336 +- > src/include/catalog/pg_am.dat | 34 + > src/include/catalog/pg_am.h | 62 +- > src/include/catalog/pg_amop.dat | 2472 ++++++ > src/include/catalog/pg_amop.h | 1148 +-- > src/include/catalog/pg_amproc.dat | 1277 +++ > src/include/catalog/pg_amproc.h | 554 +- > src/include/catalog/pg_attrdef.h | 20 +- > src/include/catalog/pg_attribute.h | 59 +- > src/include/catalog/pg_auth_members.h | 22 +- > src/include/catalog/pg_authid.dat | 67 + > src/include/catalog/pg_authid.h | 71 +- > src/include/catalog/pg_cast.dat | 535 ++ > src/include/catalog/pg_cast.h | 368 +- > src/include/catalog/pg_class.dat | 68 + > src/include/catalog/pg_class.h | 76 +- > src/include/catalog/pg_collation.dat | 31 + > src/include/catalog/pg_collation.h | 42 +- > src/include/catalog/pg_constraint.h | 56 +- > src/include/catalog/pg_conversion.h | 29 +- > src/include/catalog/pg_database.dat | 26 + > src/include/catalog/pg_database.h | 34 +- > src/include/catalog/pg_db_role_setting.h | 29 +- > src/include/catalog/pg_default_acl.h | 27 +- > src/include/catalog/pg_depend.h | 49 +- > src/include/catalog/pg_description.h | 37 +- > src/include/catalog/pg_enum.h | 27 +- > src/include/catalog/pg_event_trigger.h | 22 +- > src/include/catalog/pg_extension.h | 29 +- > src/include/catalog/pg_foreign_data_wrapper.h | 23 +- > src/include/catalog/pg_foreign_server.h | 23 +- > src/include/catalog/pg_foreign_table.h | 19 +- > src/include/catalog/pg_index.h | 38 +- > src/include/catalog/pg_inherits.h | 24 +- > src/include/catalog/pg_init_privs.h | 42 +- > src/include/catalog/pg_language.dat | 31 + > src/include/catalog/pg_language.h | 39 +- > src/include/catalog/pg_largeobject.h | 19 +- > src/include/catalog/pg_largeobject_metadata.h | 18 +- > src/include/catalog/pg_namespace.dat | 25 + > src/include/catalog/pg_namespace.h | 37 +- > src/include/catalog/pg_opclass.dat | 357 + > src/include/catalog/pg_opclass.h | 222 +- > src/include/catalog/pg_operator.dat | 3340 ++++++++ > src/include/catalog/pg_operator.h | 1854 +---- > src/include/catalog/pg_opfamily.dat | 248 + > src/include/catalog/pg_opfamily.h | 170 +- > src/include/catalog/pg_partitioned_table.h | 24 +- > src/include/catalog/pg_pltemplate.dat | 51 + > src/include/catalog/pg_pltemplate.h | 39 +- > src/include/catalog/pg_policy.h | 29 +- > src/include/catalog/pg_proc.dat | 10210 ++++++++++++++++++++++++ > src/include/catalog/pg_proc.h | 5637 +------------ > src/include/catalog/pg_publication.h | 24 +- > src/include/catalog/pg_publication_rel.h | 19 +- > src/include/catalog/pg_range.dat | 31 + > src/include/catalog/pg_range.h | 61 +- > src/include/catalog/pg_replication_origin.h | 22 +- > src/include/catalog/pg_rewrite.h | 23 +- > src/include/catalog/pg_seclabel.h | 16 +- > src/include/catalog/pg_sequence.h | 20 +- > src/include/catalog/pg_shdepend.h | 42 +- > src/include/catalog/pg_shdescription.h | 36 +- > src/include/catalog/pg_shseclabel.h | 18 +- > src/include/catalog/pg_statistic.h | 44 +- > src/include/catalog/pg_statistic_ext.h | 26 +- > src/include/catalog/pg_subscription.h | 20 +- > src/include/catalog/pg_subscription_rel.h | 19 +- > src/include/catalog/pg_tablespace.dat | 22 + > src/include/catalog/pg_tablespace.h | 26 +- > src/include/catalog/pg_transform.h | 25 +- > src/include/catalog/pg_trigger.h | 35 +- > src/include/catalog/pg_ts_config.dat | 19 + > src/include/catalog/pg_ts_config.h | 29 +- > src/include/catalog/pg_ts_config_map.dat | 35 + > src/include/catalog/pg_ts_config_map.h | 47 +- > src/include/catalog/pg_ts_dict.dat | 20 + > src/include/catalog/pg_ts_dict.h | 31 +- > src/include/catalog/pg_ts_parser.dat | 20 + > src/include/catalog/pg_ts_parser.h | 58 +- > src/include/catalog/pg_ts_template.dat | 30 + > src/include/catalog/pg_ts_template.h | 49 +- > src/include/catalog/pg_type.dat | 942 +++ > src/include/catalog/pg_type.h | 565 +- > src/include/catalog/pg_user_mapping.h | 19 +- > src/include/catalog/reformat_dat_file.pl | 311 + > src/include/catalog/toasting.h | 2 +- > src/include/catalog/unused_oids | 8 +- > src/interfaces/ecpg/ecpglib/pg_type.h | 65 +- > src/tools/msvc/Solution.pm | 49 +- > src/tools/msvc/clean.bat | 2 + > src/tools/pgindent/pgindent | 8 +- > 110 files changed, 22928 insertions(+), 12480 deletions(-) > -- Teodor Sigaev E-mail: teodor@sigaev.ru WWW: http://www.sigaev.ru/