Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKQ0F-00064Y-G2 for pgsql-hackers@arkaria.postgresql.org; Thu, 11 Mar 2021 18:25:44 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1lKQ0E-00071C-3g for pgsql-hackers@arkaria.postgresql.org; Thu, 11 Mar 2021 18:25:42 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1lKQ0D-00070p-1k for pgsql-hackers@lists.postgresql.org; Thu, 11 Mar 2021 18:25:41 +0000 Received: from mail-il1-x12e.google.com ([2607:f8b0:4864:20::12e]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1lKQ02-000867-IK for pgsql-hackers@lists.postgresql.org; Thu, 11 Mar 2021 18:25:39 +0000 Received: by mail-il1-x12e.google.com with SMTP id r7so234150ilb.0 for ; Thu, 11 Mar 2021 10:25:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=telsasoft-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to:user-agent; bh=rXpzD2xmRo4abePEgxIT6c6LKzyvFOLJPdAGWpPqn3g=; b=IT2Gwn/BMaPqa/uDML/X4wHJo8dLKCadPPYXsR6x10pq0WHFh2T1YaO23pCJvaH+vf elz++kQiFBc28+Qz8p2d8PiHvTnwOdj48hufi9vyOwW4DUiTjAEuxq1laQ+ttbk/9qNL jQyOsXXyZqX4/psAm6dwaZaGGvRgC/mUvGkBPNJt7bvzQlVEE1TXdbJD1JrqBIuwgS3X L1/h0wm4lioToRRnuLQdFPljTnmE+D4V31F19H2JecYZMElX5Ixb3n1DCayzKreu8yaY p0xqMvbmauLnoWnjzOuaQt1ejL+nq3y0yyAkAbVO84CvpaBGUR6G2PEMXpN/SshhXDUU 9gIg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to:user-agent; bh=rXpzD2xmRo4abePEgxIT6c6LKzyvFOLJPdAGWpPqn3g=; b=t/lSqa+hAQ7GnY46c8pCkEob+OqJAbmt1ofvbZAPeubY2RTWcVSufXVqcgqPYkl+ib Jr7C6xZDaZNxCypUCUF9Wxxzsz37kNWmWX1mptVk8LxfGoDcYwimkEEWgAHn2oIl7IFC qufMQE7TOcikqYVumNeFGXrlsEbGzR3E7dokfpMd5X+/X4SAX8qfgNoIM/iym9fJBbvL huVn+d7Snmq9ABc4BvxUovd4iJm12vCHVn/bpcRqZUkrTAmCxlivmGcRHYo4W32qvYh6 MqiALgMrIUzc8xYTxxcOmabpdT35HVulBQuI+O5wntWmX+PJ+aro5VPmmewNSyzIgwRV FdVA== X-Gm-Message-State: AOAM533ZCJUU1sibd19sHqoOZeLAQrley9LnJJDeGTFI2QzfylvHga1v I7ZUKbKACfv/ohCU2QzVBLNhBA== X-Google-Smtp-Source: ABdhPJxYLYQhqD2ciGZjxGTa8UmaVGrdfKgFn5JTRPkUukycFqMyYcmAbhRq2EHJaFRl7UNmGxktcg== X-Received: by 2002:a05:6e02:20ee:: with SMTP id q14mr7402558ilv.259.1615487129602; Thu, 11 Mar 2021 10:25:29 -0800 (PST) Received: from pryzbyj.telsasoft (charmander.telsasoft.com. [50.244.222.1]) by smtp.gmail.com with ESMTPSA id q21sm1701667ioh.41.2021.03.11.10.25.27 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Mar 2021 10:25:28 -0800 (PST) Received: by pryzbyj.telsasoft (Postfix, from userid 1000) id C6AD480094D; Thu, 11 Mar 2021 12:25:26 -0600 (CST) Date: Thu, 11 Mar 2021 12:25:26 -0600 From: Justin Pryzby To: Robert Haas Cc: Dilip Kumar , Tomas Vondra , Alexander Korotkov , David Steele , Ildus Kurbangaliev , Dmitry Dolgov <9erthalion6@gmail.com>, Thomas Munro , pgsql-hackers@lists.postgresql.org Subject: Re: [HACKERS] Custom compression methods Message-ID: <20210311182526.GC8796@telsasoft.com> References: <20210306195751.GW29832@telsasoft.com> <20210311022858.GY2021@telsasoft.com> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="cvVnyQ+4j833TQvp" Content-Disposition: inline In-Reply-To: <20210311022858.GY2021@telsasoft.com> User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --cvVnyQ+4j833TQvp Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Wed, Mar 10, 2021 at 08:28:58PM -0600, Justin Pryzby wrote: > This includes a patch to use pkgconfig, in an attempt to build on mac, which > currently fails like: > > https://cirrus-ci.com/task/5993712963551232?command=build#L126 > checking for LZ4_compress in -llz4... no > configure: error: library 'lz4' is required for LZ4 support This includes a 2nd attempt to use pkg-config to build on mac. If this doesn't work, we should ask for help from a mac user who wants to take on a hopefully-quick project. --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0001-Get-datum-from-tuple-which-doesn-t-contain-external-.patch" From db1e2cbe5cd008d1e042f409705e43b992eb9b7d Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Thu, 4 Mar 2021 13:30:47 +0530 Subject: [PATCH 1/8] Get datum from tuple which doesn't contain external data Currently, we use HeapTupleGetDatum and HeapTupleHeaderGetDatum whenever we need to convert a tuple to datum. Introduce another version of these routine HeapTupleGetRawDatum and HeapTupleHeaderGetRawDatum respectively so that all the callers who doesn't expect any external data in the tuple can call these new routines. Likewise similar treatment for heap_copy_tuple_as_datum and PG_RETURN_HEAPTUPLEHEADER as well. Dilip Kumar based on the idea by Robert Haas --- contrib/dblink/dblink.c | 2 +- contrib/hstore/hstore_io.c | 4 ++-- contrib/hstore/hstore_op.c | 2 +- contrib/old_snapshot/time_mapping.c | 2 +- contrib/pageinspect/brinfuncs.c | 2 +- contrib/pageinspect/btreefuncs.c | 6 +++--- contrib/pageinspect/ginfuncs.c | 6 +++--- contrib/pageinspect/gistfuncs.c | 2 +- contrib/pageinspect/hashfuncs.c | 8 ++++---- contrib/pageinspect/heapfuncs.c | 6 +++--- contrib/pageinspect/rawpage.c | 2 +- contrib/pg_buffercache/pg_buffercache_pages.c | 2 +- contrib/pg_stat_statements/pg_stat_statements.c | 3 ++- contrib/pg_visibility/pg_visibility.c | 13 ++++++++----- contrib/pgcrypto/pgp-pgsql.c | 2 +- contrib/pgstattuple/pgstatapprox.c | 2 +- contrib/pgstattuple/pgstatindex.c | 6 +++--- contrib/pgstattuple/pgstattuple.c | 2 +- contrib/sslinfo/sslinfo.c | 2 +- src/backend/access/common/heaptuple.c | 17 +++++++++++++++-- src/backend/access/transam/commit_ts.c | 4 ++-- src/backend/access/transam/multixact.c | 2 +- src/backend/access/transam/twophase.c | 2 +- src/backend/access/transam/xlogfuncs.c | 2 +- src/backend/catalog/objectaddress.c | 6 +++--- src/backend/commands/sequence.c | 2 +- src/backend/executor/execExprInterp.c | 15 ++++++++++----- src/backend/replication/slotfuncs.c | 8 ++++---- src/backend/replication/walreceiver.c | 3 ++- src/backend/statistics/mcv.c | 2 +- src/backend/tsearch/wparser.c | 4 ++-- src/backend/utils/adt/acl.c | 2 +- src/backend/utils/adt/datetime.c | 2 +- src/backend/utils/adt/genfile.c | 2 +- src/backend/utils/adt/lockfuncs.c | 4 ++-- src/backend/utils/adt/misc.c | 4 ++-- src/backend/utils/adt/partitionfuncs.c | 2 +- src/backend/utils/adt/pgstatfuncs.c | 6 ++++-- src/backend/utils/adt/rowtypes.c | 4 ++-- src/backend/utils/adt/tsvector_op.c | 4 ++-- src/backend/utils/misc/guc.c | 2 +- src/backend/utils/misc/pg_controldata.c | 8 ++++---- src/include/access/htup_details.h | 1 + src/include/fmgr.h | 1 + src/include/funcapi.h | 15 ++++++++++++++- src/pl/plperl/plperl.c | 4 ++-- src/pl/tcl/pltcl.c | 4 ++-- src/test/modules/test_predtest/test_predtest.c | 3 ++- 48 files changed, 125 insertions(+), 84 deletions(-) diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c index 3a0beaa88e..5a41116e8e 100644 --- a/contrib/dblink/dblink.c +++ b/contrib/dblink/dblink.c @@ -1630,7 +1630,7 @@ dblink_get_pkey(PG_FUNCTION_ARGS) tuple = BuildTupleFromCStrings(attinmeta, values); /* make the tuple into a datum */ - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/contrib/hstore/hstore_io.c b/contrib/hstore/hstore_io.c index b3304ff844..cc7a8e0eef 100644 --- a/contrib/hstore/hstore_io.c +++ b/contrib/hstore/hstore_io.c @@ -1137,14 +1137,14 @@ hstore_populate_record(PG_FUNCTION_ARGS) * check domain constraints before deciding we're done. */ if (argtype != tupdesc->tdtypeid) - domain_check(HeapTupleGetDatum(rettuple), false, + domain_check(HeapTupleGetRawDatum(rettuple), false, argtype, &my_extra->domain_info, fcinfo->flinfo->fn_mcxt); ReleaseTupleDesc(tupdesc); - PG_RETURN_DATUM(HeapTupleGetDatum(rettuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(rettuple->t_data); } diff --git a/contrib/hstore/hstore_op.c b/contrib/hstore/hstore_op.c index dd79d01cac..d466f6cc46 100644 --- a/contrib/hstore/hstore_op.c +++ b/contrib/hstore/hstore_op.c @@ -1067,7 +1067,7 @@ hstore_each(PG_FUNCTION_ARGS) } tuple = heap_form_tuple(funcctx->tuple_desc, dvalues, nulls); - res = HeapTupleGetDatum(tuple); + res = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, PointerGetDatum(res)); } diff --git a/contrib/old_snapshot/time_mapping.c b/contrib/old_snapshot/time_mapping.c index 3df07177ed..5d517c8afc 100644 --- a/contrib/old_snapshot/time_mapping.c +++ b/contrib/old_snapshot/time_mapping.c @@ -72,7 +72,7 @@ pg_old_snapshot_time_mapping(PG_FUNCTION_ARGS) tuple = MakeOldSnapshotTimeMappingTuple(funcctx->tuple_desc, mapping); ++mapping->current_index; - SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funcctx, HeapTupleGetRawDatum(tuple)); } SRF_RETURN_DONE(funcctx); diff --git a/contrib/pageinspect/brinfuncs.c b/contrib/pageinspect/brinfuncs.c index 0e3c2deb66..626294685a 100644 --- a/contrib/pageinspect/brinfuncs.c +++ b/contrib/pageinspect/brinfuncs.c @@ -366,7 +366,7 @@ brin_metapage_info(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } /* diff --git a/contrib/pageinspect/btreefuncs.c b/contrib/pageinspect/btreefuncs.c index b7725b572f..f0028e4cc4 100644 --- a/contrib/pageinspect/btreefuncs.c +++ b/contrib/pageinspect/btreefuncs.c @@ -263,7 +263,7 @@ bt_page_stats_internal(PG_FUNCTION_ARGS, enum pageinspect_version ext_version) tuple = BuildTupleFromCStrings(TupleDescGetAttInMetadata(tupleDesc), values); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); PG_RETURN_DATUM(result); } @@ -436,7 +436,7 @@ bt_page_print_tuples(struct user_args *uargs) /* Build and return the result tuple */ tuple = heap_form_tuple(uargs->tupd, values, nulls); - return HeapTupleGetDatum(tuple); + return HeapTupleGetRawDatum(tuple); } /*------------------------------------------------------- @@ -766,7 +766,7 @@ bt_metap(PG_FUNCTION_ARGS) tuple = BuildTupleFromCStrings(TupleDescGetAttInMetadata(tupleDesc), values); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); UnlockReleaseBuffer(buffer); relation_close(rel, AccessShareLock); diff --git a/contrib/pageinspect/ginfuncs.c b/contrib/pageinspect/ginfuncs.c index e425cbcdb8..77dd559c7e 100644 --- a/contrib/pageinspect/ginfuncs.c +++ b/contrib/pageinspect/ginfuncs.c @@ -82,7 +82,7 @@ gin_metapage_info(PG_FUNCTION_ARGS) /* Build and return the result tuple. */ resultTuple = heap_form_tuple(tupdesc, values, nulls); - return HeapTupleGetDatum(resultTuple); + return HeapTupleGetRawDatum(resultTuple); } @@ -150,7 +150,7 @@ gin_page_opaque_info(PG_FUNCTION_ARGS) /* Build and return the result tuple. */ resultTuple = heap_form_tuple(tupdesc, values, nulls); - return HeapTupleGetDatum(resultTuple); + return HeapTupleGetRawDatum(resultTuple); } typedef struct gin_leafpage_items_state @@ -254,7 +254,7 @@ gin_leafpage_items(PG_FUNCTION_ARGS) /* Build and return the result tuple. */ resultTuple = heap_form_tuple(inter_call_data->tupd, values, nulls); - result = HeapTupleGetDatum(resultTuple); + result = HeapTupleGetRawDatum(resultTuple); inter_call_data->seg = GinNextPostingListSegment(cur); diff --git a/contrib/pageinspect/gistfuncs.c b/contrib/pageinspect/gistfuncs.c index eb9f6303df..dbf34f87d4 100644 --- a/contrib/pageinspect/gistfuncs.c +++ b/contrib/pageinspect/gistfuncs.c @@ -89,7 +89,7 @@ gist_page_opaque_info(PG_FUNCTION_ARGS) /* Build and return the result tuple. */ resultTuple = heap_form_tuple(tupdesc, values, nulls); - return HeapTupleGetDatum(resultTuple); + return HeapTupleGetRawDatum(resultTuple); } Datum diff --git a/contrib/pageinspect/hashfuncs.c b/contrib/pageinspect/hashfuncs.c index ff01119474..957ee5bc14 100644 --- a/contrib/pageinspect/hashfuncs.c +++ b/contrib/pageinspect/hashfuncs.c @@ -273,7 +273,7 @@ hash_page_stats(PG_FUNCTION_ARGS) tuple = heap_form_tuple(tupleDesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(tuple->t_data); } /* @@ -368,7 +368,7 @@ hash_page_items(PG_FUNCTION_ARGS) values[j] = Int64GetDatum((int64) hashkey); tuple = heap_form_tuple(fctx->attinmeta->tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); uargs->offset = uargs->offset + 1; @@ -499,7 +499,7 @@ hash_bitmap_info(PG_FUNCTION_ARGS) tuple = heap_form_tuple(tupleDesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(tuple->t_data); } /* ------------------------------------------------ @@ -577,5 +577,5 @@ hash_metapage_info(PG_FUNCTION_ARGS) tuple = heap_form_tuple(tupleDesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(tuple->t_data); } diff --git a/contrib/pageinspect/heapfuncs.c b/contrib/pageinspect/heapfuncs.c index f6760eb31e..3a050ee88c 100644 --- a/contrib/pageinspect/heapfuncs.c +++ b/contrib/pageinspect/heapfuncs.c @@ -282,7 +282,7 @@ heap_page_items(PG_FUNCTION_ARGS) /* Build and return the result tuple. */ resultTuple = heap_form_tuple(inter_call_data->tupd, values, nulls); - result = HeapTupleGetDatum(resultTuple); + result = HeapTupleGetRawDatum(resultTuple); inter_call_data->offset++; @@ -540,7 +540,7 @@ heap_tuple_infomask_flags(PG_FUNCTION_ARGS) values[0] = PointerGetDatum(construct_empty_array(TEXTOID)); values[1] = PointerGetDatum(construct_empty_array(TEXTOID)); tuple = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(tuple->t_data); } /* build set of raw flags */ @@ -618,5 +618,5 @@ heap_tuple_infomask_flags(PG_FUNCTION_ARGS) /* Returns the record as Datum */ tuple = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(tuple->t_data); } diff --git a/contrib/pageinspect/rawpage.c b/contrib/pageinspect/rawpage.c index 7272b21016..e7aab866b8 100644 --- a/contrib/pageinspect/rawpage.c +++ b/contrib/pageinspect/rawpage.c @@ -328,7 +328,7 @@ page_header(PG_FUNCTION_ARGS) memset(nulls, 0, sizeof(nulls)); tuple = heap_form_tuple(tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); PG_RETURN_DATUM(result); } diff --git a/contrib/pg_buffercache/pg_buffercache_pages.c b/contrib/pg_buffercache/pg_buffercache_pages.c index 1bd579fcbb..1fd405e5df 100644 --- a/contrib/pg_buffercache/pg_buffercache_pages.c +++ b/contrib/pg_buffercache/pg_buffercache_pages.c @@ -230,7 +230,7 @@ pg_buffercache_pages(PG_FUNCTION_ARGS) /* Build and return the tuple. */ tuple = heap_form_tuple(fctx->tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/contrib/pg_stat_statements/pg_stat_statements.c b/contrib/pg_stat_statements/pg_stat_statements.c index 62cccbfa44..7eb80d5b78 100644 --- a/contrib/pg_stat_statements/pg_stat_statements.c +++ b/contrib/pg_stat_statements/pg_stat_statements.c @@ -1922,7 +1922,8 @@ pg_stat_statements_info(PG_FUNCTION_ARGS) values[0] = Int64GetDatum(stats.dealloc); values[1] = TimestampTzGetDatum(stats.stats_reset); - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } /* diff --git a/contrib/pg_visibility/pg_visibility.c b/contrib/pg_visibility/pg_visibility.c index dd0c124e62..43bb2ab852 100644 --- a/contrib/pg_visibility/pg_visibility.c +++ b/contrib/pg_visibility/pg_visibility.c @@ -97,7 +97,8 @@ pg_visibility_map(PG_FUNCTION_ARGS) relation_close(rel, AccessShareLock); - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } /* @@ -156,7 +157,8 @@ pg_visibility(PG_FUNCTION_ARGS) relation_close(rel, AccessShareLock); - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } /* @@ -197,7 +199,7 @@ pg_visibility_map_rel(PG_FUNCTION_ARGS) info->next++; tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funcctx, HeapTupleGetRawDatum(tuple)); } SRF_RETURN_DONE(funcctx); @@ -243,7 +245,7 @@ pg_visibility_rel(PG_FUNCTION_ARGS) info->next++; tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funcctx, HeapTupleGetRawDatum(tuple)); } SRF_RETURN_DONE(funcctx); @@ -303,7 +305,8 @@ pg_visibility_map_summary(PG_FUNCTION_ARGS) values[0] = Int64GetDatum(all_visible); values[1] = Int64GetDatum(all_frozen); - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } /* diff --git a/contrib/pgcrypto/pgp-pgsql.c b/contrib/pgcrypto/pgp-pgsql.c index 0536bfb892..333f7fd391 100644 --- a/contrib/pgcrypto/pgp-pgsql.c +++ b/contrib/pgcrypto/pgp-pgsql.c @@ -973,7 +973,7 @@ pgp_armor_headers(PG_FUNCTION_ARGS) /* build a tuple */ tuple = BuildTupleFromCStrings(funcctx->attinmeta, values); - SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funcctx, HeapTupleGetRawDatum(tuple)); } } diff --git a/contrib/pgstattuple/pgstatapprox.c b/contrib/pgstattuple/pgstatapprox.c index 1fe193bb25..147c8d22eb 100644 --- a/contrib/pgstattuple/pgstatapprox.c +++ b/contrib/pgstattuple/pgstatapprox.c @@ -314,5 +314,5 @@ pgstattuple_approx_internal(Oid relid, FunctionCallInfo fcinfo) values[i++] = Float8GetDatum(stat.free_percent); ret = heap_form_tuple(tupdesc, values, nulls); - return HeapTupleGetDatum(ret); + return HeapTupleGetRawDatum(ret); } diff --git a/contrib/pgstattuple/pgstatindex.c b/contrib/pgstattuple/pgstatindex.c index 5368bb30f0..7d74c316c7 100644 --- a/contrib/pgstattuple/pgstatindex.c +++ b/contrib/pgstattuple/pgstatindex.c @@ -362,7 +362,7 @@ pgstatindex_impl(Relation rel, FunctionCallInfo fcinfo) tuple = BuildTupleFromCStrings(TupleDescGetAttInMetadata(tupleDesc), values); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); } return result; @@ -571,7 +571,7 @@ pgstatginindex_internal(Oid relid, FunctionCallInfo fcinfo) * Build and return the tuple */ tuple = heap_form_tuple(tupleDesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); return result; } @@ -727,7 +727,7 @@ pgstathashindex(PG_FUNCTION_ARGS) values[7] = Float8GetDatum(free_percent); tuple = heap_form_tuple(tupleDesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(tuple->t_data); } /* ------------------------------------------------- diff --git a/contrib/pgstattuple/pgstattuple.c b/contrib/pgstattuple/pgstattuple.c index 21fdeff8af..70b8bf0855 100644 --- a/contrib/pgstattuple/pgstattuple.c +++ b/contrib/pgstattuple/pgstattuple.c @@ -147,7 +147,7 @@ build_pgstattuple_type(pgstattuple_type *stat, FunctionCallInfo fcinfo) tuple = BuildTupleFromCStrings(attinmeta, values); /* make the tuple into a datum */ - return HeapTupleGetDatum(tuple); + return HeapTupleGetRawDatum(tuple); } /* ---------- diff --git a/contrib/sslinfo/sslinfo.c b/contrib/sslinfo/sslinfo.c index 30cae0bb98..53801e92cd 100644 --- a/contrib/sslinfo/sslinfo.c +++ b/contrib/sslinfo/sslinfo.c @@ -460,7 +460,7 @@ ssl_extension_info(PG_FUNCTION_ARGS) /* Build tuple */ tuple = heap_form_tuple(fctx->tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); if (BIO_free(membuf) != 1) elog(ERROR, "could not free OpenSSL BIO structure"); diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c index 0b56b0fa5a..c36c283253 100644 --- a/src/backend/access/common/heaptuple.c +++ b/src/backend/access/common/heaptuple.c @@ -983,8 +983,6 @@ heap_expand_tuple(HeapTuple sourceTuple, TupleDesc tupleDesc) Datum heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc) { - HeapTupleHeader td; - /* * If the tuple contains any external TOAST pointers, we have to inline * those fields to meet the conventions for composite-type Datums. @@ -993,6 +991,21 @@ heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc) return toast_flatten_tuple_to_datum(tuple->t_data, tuple->t_len, tupleDesc); + else + return heap_copy_tuple_as_raw_datum(tuple, tupleDesc); +} + +/* ---------------- + * heap_copy_tuple_as_raw_datum + * + * copy a tuple as a composite-type Datum, but the input tuple should not + * contain any external data. + * ---------------- + */ +Datum +heap_copy_tuple_as_raw_datum(HeapTuple tuple, TupleDesc tupleDesc) +{ + HeapTupleHeader td; /* * Fast path for easy case: just make a palloc'd copy and insert the diff --git a/src/backend/access/transam/commit_ts.c b/src/backend/access/transam/commit_ts.c index 48e8d66286..015c84179e 100644 --- a/src/backend/access/transam/commit_ts.c +++ b/src/backend/access/transam/commit_ts.c @@ -469,7 +469,7 @@ pg_last_committed_xact(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } /* @@ -518,7 +518,7 @@ pg_xact_commit_timestamp_origin(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } /* diff --git a/src/backend/access/transam/multixact.c b/src/backend/access/transam/multixact.c index 1f9f1a1fa1..b44066a4eb 100644 --- a/src/backend/access/transam/multixact.c +++ b/src/backend/access/transam/multixact.c @@ -3399,7 +3399,7 @@ pg_get_multixact_members(PG_FUNCTION_ARGS) multi->iter++; pfree(values[0]); - SRF_RETURN_NEXT(funccxt, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funccxt, HeapTupleGetRawDatum(tuple)); } SRF_RETURN_DONE(funccxt); diff --git a/src/backend/access/transam/twophase.c b/src/backend/access/transam/twophase.c index 6023e7c16f..3f48597c2a 100644 --- a/src/backend/access/transam/twophase.c +++ b/src/backend/access/transam/twophase.c @@ -787,7 +787,7 @@ pg_prepared_xact(PG_FUNCTION_ARGS) values[4] = ObjectIdGetDatum(proc->databaseId); tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/src/backend/access/transam/xlogfuncs.c b/src/backend/access/transam/xlogfuncs.c index d8c5bf6dc2..53f84de41c 100644 --- a/src/backend/access/transam/xlogfuncs.c +++ b/src/backend/access/transam/xlogfuncs.c @@ -487,7 +487,7 @@ pg_walfile_name_offset(PG_FUNCTION_ARGS) */ resultHeapTuple = heap_form_tuple(resultTupleDesc, values, isnull); - result = HeapTupleGetDatum(resultHeapTuple); + result = HeapTupleGetRawDatum(resultHeapTuple); PG_RETURN_DATUM(result); } diff --git a/src/backend/catalog/objectaddress.c b/src/backend/catalog/objectaddress.c index 6d88b690d8..7cd62e70b8 100644 --- a/src/backend/catalog/objectaddress.c +++ b/src/backend/catalog/objectaddress.c @@ -2355,7 +2355,7 @@ pg_get_object_address(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } /* @@ -4233,7 +4233,7 @@ pg_identify_object(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } /* @@ -4304,7 +4304,7 @@ pg_identify_object_as_address(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } /* diff --git a/src/backend/commands/sequence.c b/src/backend/commands/sequence.c index 0415df9ccb..f566e1e0a5 100644 --- a/src/backend/commands/sequence.c +++ b/src/backend/commands/sequence.c @@ -1834,7 +1834,7 @@ pg_sequence_parameters(PG_FUNCTION_ARGS) ReleaseSysCache(pgstuple); - return HeapTupleGetDatum(heap_form_tuple(tupdesc, values, isnull)); + return HeapTupleGetRawDatum(heap_form_tuple(tupdesc, values, isnull)); } /* diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 6308286d8c..20949a5dec 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -3169,8 +3169,13 @@ ExecEvalConvertRowtype(ExprState *state, ExprEvalStep *op, ExprContext *econtext { /* Full conversion with attribute rearrangement needed */ result = execute_attr_map_tuple(&tmptup, op->d.convert_rowtype.map); - /* Result already has appropriate composite-datum header fields */ - *op->resvalue = HeapTupleGetDatum(result); + + /* + * Result already has appropriate composite-datum header fields. The + * input was a composite type so we aren't expecting to have to flatten + * any toasted fields so directly call HeapTupleGetRawDatum. + */ + *op->resvalue = HeapTupleGetRawDatum(result); } else { @@ -3181,10 +3186,10 @@ ExecEvalConvertRowtype(ExprState *state, ExprEvalStep *op, ExprContext *econtext * for this since it will both make the physical copy and insert the * correct composite header fields. Note that we aren't expecting to * have to flatten any toasted fields: the input was a composite - * datum, so it shouldn't contain any. So heap_copy_tuple_as_datum() - * is overkill here, but its check for external fields is cheap. + * datum, so it shouldn't contain any. So we can directly call + * heap_copy_tuple_as_raw_datum(). */ - *op->resvalue = heap_copy_tuple_as_datum(&tmptup, outdesc); + *op->resvalue = heap_copy_tuple_as_raw_datum(&tmptup, outdesc); } } diff --git a/src/backend/replication/slotfuncs.c b/src/backend/replication/slotfuncs.c index 9817b44113..f92628bc59 100644 --- a/src/backend/replication/slotfuncs.c +++ b/src/backend/replication/slotfuncs.c @@ -106,7 +106,7 @@ pg_create_physical_replication_slot(PG_FUNCTION_ARGS) nulls[1] = true; tuple = heap_form_tuple(tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); ReplicationSlotRelease(); @@ -205,7 +205,7 @@ pg_create_logical_replication_slot(PG_FUNCTION_ARGS) memset(nulls, 0, sizeof(nulls)); tuple = heap_form_tuple(tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); /* ok, slot is now fully created, mark it as persistent if needed */ if (!temporary) @@ -689,7 +689,7 @@ pg_replication_slot_advance(PG_FUNCTION_ARGS) nulls[1] = false; tuple = heap_form_tuple(tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); PG_RETURN_DATUM(result); } @@ -911,7 +911,7 @@ copy_replication_slot(FunctionCallInfo fcinfo, bool logical_slot) nulls[1] = true; tuple = heap_form_tuple(tupdesc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); ReplicationSlotRelease(); diff --git a/src/backend/replication/walreceiver.c b/src/backend/replication/walreceiver.c index e5f8a06fea..f973aea8e5 100644 --- a/src/backend/replication/walreceiver.c +++ b/src/backend/replication/walreceiver.c @@ -1420,5 +1420,6 @@ pg_stat_get_wal_receiver(PG_FUNCTION_ARGS) } /* Returns the record as Datum */ - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } diff --git a/src/backend/statistics/mcv.c b/src/backend/statistics/mcv.c index abbc1f1ba8..115131c707 100644 --- a/src/backend/statistics/mcv.c +++ b/src/backend/statistics/mcv.c @@ -1450,7 +1450,7 @@ pg_stats_ext_mcvlist_items(PG_FUNCTION_ARGS) tuple = heap_form_tuple(funcctx->attinmeta->tupdesc, values, nulls); /* make the tuple into a datum */ - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/src/backend/tsearch/wparser.c b/src/backend/tsearch/wparser.c index 71882dced9..633c90d08d 100644 --- a/src/backend/tsearch/wparser.c +++ b/src/backend/tsearch/wparser.c @@ -97,7 +97,7 @@ tt_process_call(FuncCallContext *funcctx) values[2] = st->list[st->cur].descr; tuple = BuildTupleFromCStrings(funcctx->attinmeta, values); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); pfree(values[1]); pfree(values[2]); @@ -236,7 +236,7 @@ prs_process_call(FuncCallContext *funcctx) sprintf(tid, "%d", st->list[st->cur].type); values[1] = st->list[st->cur].lexeme; tuple = BuildTupleFromCStrings(funcctx->attinmeta, values); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); pfree(values[1]); st->cur++; diff --git a/src/backend/utils/adt/acl.c b/src/backend/utils/adt/acl.c index c7f029e218..5871c0b404 100644 --- a/src/backend/utils/adt/acl.c +++ b/src/backend/utils/adt/acl.c @@ -1804,7 +1804,7 @@ aclexplode(PG_FUNCTION_ARGS) MemSet(nulls, 0, sizeof(nulls)); tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/src/backend/utils/adt/datetime.c b/src/backend/utils/adt/datetime.c index 350b0c55ea..ce11554b3b 100644 --- a/src/backend/utils/adt/datetime.c +++ b/src/backend/utils/adt/datetime.c @@ -4776,7 +4776,7 @@ pg_timezone_abbrevs(PG_FUNCTION_ARGS) (*pindex)++; tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/src/backend/utils/adt/genfile.c b/src/backend/utils/adt/genfile.c index 169ddf8d76..d3d386b748 100644 --- a/src/backend/utils/adt/genfile.c +++ b/src/backend/utils/adt/genfile.c @@ -454,7 +454,7 @@ pg_stat_file(PG_FUNCTION_ARGS) pfree(filename); - PG_RETURN_DATUM(HeapTupleGetDatum(tuple)); + PG_RETURN_HEAPTUPLEHEADER_RAW(tuple->t_data); } /* diff --git a/src/backend/utils/adt/lockfuncs.c b/src/backend/utils/adt/lockfuncs.c index 97f0265c12..081844939b 100644 --- a/src/backend/utils/adt/lockfuncs.c +++ b/src/backend/utils/adt/lockfuncs.c @@ -344,7 +344,7 @@ pg_lock_status(PG_FUNCTION_ARGS) nulls[15] = true; tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } @@ -415,7 +415,7 @@ pg_lock_status(PG_FUNCTION_ARGS) nulls[15] = true; tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/src/backend/utils/adt/misc.c b/src/backend/utils/adt/misc.c index 634f574d7e..57e0ea09a3 100644 --- a/src/backend/utils/adt/misc.c +++ b/src/backend/utils/adt/misc.c @@ -484,7 +484,7 @@ pg_get_keywords(PG_FUNCTION_ARGS) tuple = BuildTupleFromCStrings(funcctx->attinmeta, values); - SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funcctx, HeapTupleGetRawDatum(tuple)); } SRF_RETURN_DONE(funcctx); @@ -562,7 +562,7 @@ pg_get_catalog_foreign_keys(PG_FUNCTION_ARGS) tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funcctx, HeapTupleGetRawDatum(tuple)); } SRF_RETURN_DONE(funcctx); diff --git a/src/backend/utils/adt/partitionfuncs.c b/src/backend/utils/adt/partitionfuncs.c index 03660d5db6..6915939f43 100644 --- a/src/backend/utils/adt/partitionfuncs.c +++ b/src/backend/utils/adt/partitionfuncs.c @@ -159,7 +159,7 @@ pg_partition_tree(PG_FUNCTION_ARGS) values[3] = Int32GetDatum(level); tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/src/backend/utils/adt/pgstatfuncs.c b/src/backend/utils/adt/pgstatfuncs.c index 5102227a60..b93ad73241 100644 --- a/src/backend/utils/adt/pgstatfuncs.c +++ b/src/backend/utils/adt/pgstatfuncs.c @@ -1856,7 +1856,8 @@ pg_stat_get_wal(PG_FUNCTION_ARGS) values[8] = TimestampTzGetDatum(wal_stats->stat_reset_timestamp); /* Returns the record as Datum */ - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } /* @@ -2272,7 +2273,8 @@ pg_stat_get_archiver(PG_FUNCTION_ARGS) values[6] = TimestampTzGetDatum(archiver_stats->stat_reset_timestamp); /* Returns the record as Datum */ - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } /* Get the statistics for the replication slots */ diff --git a/src/backend/utils/adt/rowtypes.c b/src/backend/utils/adt/rowtypes.c index 23787a6ae7..079a5af5fb 100644 --- a/src/backend/utils/adt/rowtypes.c +++ b/src/backend/utils/adt/rowtypes.c @@ -293,7 +293,7 @@ record_in(PG_FUNCTION_ARGS) pfree(nulls); ReleaseTupleDesc(tupdesc); - PG_RETURN_HEAPTUPLEHEADER(result); + PG_RETURN_HEAPTUPLEHEADER_RAW(result); } /* @@ -660,7 +660,7 @@ record_recv(PG_FUNCTION_ARGS) pfree(nulls); ReleaseTupleDesc(tupdesc); - PG_RETURN_HEAPTUPLEHEADER(result); + PG_RETURN_HEAPTUPLEHEADER_RAW(result); } /* diff --git a/src/backend/utils/adt/tsvector_op.c b/src/backend/utils/adt/tsvector_op.c index 9236ebcc8f..6679493247 100644 --- a/src/backend/utils/adt/tsvector_op.c +++ b/src/backend/utils/adt/tsvector_op.c @@ -707,7 +707,7 @@ tsvector_unnest(PG_FUNCTION_ARGS) } tuple = heap_form_tuple(funcctx->tuple_desc, values, nulls); - SRF_RETURN_NEXT(funcctx, HeapTupleGetDatum(tuple)); + SRF_RETURN_NEXT(funcctx, HeapTupleGetRawDatum(tuple)); } else { @@ -2385,7 +2385,7 @@ ts_process_call(FuncCallContext *funcctx) values[2] = nentry; tuple = BuildTupleFromCStrings(funcctx->attinmeta, values); - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); pfree(values[0]); diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 855076b1fd..953837085f 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -9795,7 +9795,7 @@ show_all_settings(PG_FUNCTION_ARGS) tuple = BuildTupleFromCStrings(attinmeta, values); /* make the tuple into a datum */ - result = HeapTupleGetDatum(tuple); + result = HeapTupleGetRawDatum(tuple); SRF_RETURN_NEXT(funcctx, result); } diff --git a/src/backend/utils/misc/pg_controldata.c b/src/backend/utils/misc/pg_controldata.c index 209a20a882..195c127739 100644 --- a/src/backend/utils/misc/pg_controldata.c +++ b/src/backend/utils/misc/pg_controldata.c @@ -73,7 +73,7 @@ pg_control_system(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } Datum @@ -204,7 +204,7 @@ pg_control_checkpoint(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } Datum @@ -257,7 +257,7 @@ pg_control_recovery(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } Datum @@ -340,5 +340,5 @@ pg_control_init(PG_FUNCTION_ARGS) htup = heap_form_tuple(tupdesc, values, nulls); - PG_RETURN_DATUM(HeapTupleGetDatum(htup)); + PG_RETURN_HEAPTUPLEHEADER_RAW(htup->t_data); } diff --git a/src/include/access/htup_details.h b/src/include/access/htup_details.h index 7c62852e7f..7ac3c2382d 100644 --- a/src/include/access/htup_details.h +++ b/src/include/access/htup_details.h @@ -790,6 +790,7 @@ extern Datum getmissingattr(TupleDesc tupleDesc, extern HeapTuple heap_copytuple(HeapTuple tuple); extern void heap_copytuple_with_tuple(HeapTuple src, HeapTuple dest); extern Datum heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc); +extern Datum heap_copy_tuple_as_raw_datum(HeapTuple tuple, TupleDesc tupleDesc); extern HeapTuple heap_form_tuple(TupleDesc tupleDescriptor, Datum *values, bool *isnull); extern HeapTuple heap_modify_tuple(HeapTuple tuple, diff --git a/src/include/fmgr.h b/src/include/fmgr.h index ab7b85c86e..8771ccdac6 100644 --- a/src/include/fmgr.h +++ b/src/include/fmgr.h @@ -373,6 +373,7 @@ extern struct varlena *pg_detoast_datum_packed(struct varlena *datum); #define PG_RETURN_BPCHAR_P(x) PG_RETURN_POINTER(x) #define PG_RETURN_VARCHAR_P(x) PG_RETURN_POINTER(x) #define PG_RETURN_HEAPTUPLEHEADER(x) return HeapTupleHeaderGetDatum(x) +#define PG_RETURN_HEAPTUPLEHEADER_RAW(x) return HeapTupleHeaderGetRawDatum(x) /*------------------------------------------------------------------------- diff --git a/src/include/funcapi.h b/src/include/funcapi.h index 83e6bc2a1f..8ba7ae211f 100644 --- a/src/include/funcapi.h +++ b/src/include/funcapi.h @@ -205,8 +205,13 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple); * Datum HeapTupleHeaderGetDatum(HeapTupleHeader tuple) - convert a * HeapTupleHeader to a Datum. * - * Macro declarations: + * Macro declarations/inline functions: + * HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as + * HeapTupleHeaderGetDatum but the input tuple should not contain + * external varlena * HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum. + * HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum + * but the input tuple should not contain external varlena * * Obsolete routines and macros: * TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a @@ -218,7 +223,15 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple); *---------- */ +static inline Datum +HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) +{ + Assert(!HeapTupleHeaderHasExternal(tuple)); + return PointerGetDatum(tuple); +} + #define HeapTupleGetDatum(tuple) HeapTupleHeaderGetDatum((tuple)->t_data) +#define HeapTupleGetRawDatum(tuple) HeapTupleHeaderGetRawDatum((tuple)->t_data) /* obsolete version of above */ #define TupleGetDatum(_slot, _tuple) HeapTupleGetDatum(_tuple) diff --git a/src/pl/plperl/plperl.c b/src/pl/plperl/plperl.c index 6299adf71a..cdf79f78e8 100644 --- a/src/pl/plperl/plperl.c +++ b/src/pl/plperl/plperl.c @@ -1127,7 +1127,7 @@ plperl_hash_to_datum(SV *src, TupleDesc td) { HeapTuple tup = plperl_build_tuple_result((HV *) SvRV(src), td); - return HeapTupleGetDatum(tup); + return HeapTupleGetRawDatum(tup); } /* @@ -3334,7 +3334,7 @@ plperl_return_next_internal(SV *sv) current_call_data->ret_tdesc); if (OidIsValid(current_call_data->cdomain_oid)) - domain_check(HeapTupleGetDatum(tuple), false, + domain_check(HeapTupleGetRawDatum(tuple), false, current_call_data->cdomain_oid, ¤t_call_data->cdomain_info, rsi->econtext->ecxt_per_query_memory); diff --git a/src/pl/tcl/pltcl.c b/src/pl/tcl/pltcl.c index e11837559d..b4e710a22a 100644 --- a/src/pl/tcl/pltcl.c +++ b/src/pl/tcl/pltcl.c @@ -1024,7 +1024,7 @@ pltcl_func_handler(PG_FUNCTION_ARGS, pltcl_call_state *call_state, tup = pltcl_build_tuple_result(interp, resultObjv, resultObjc, call_state); - retval = HeapTupleGetDatum(tup); + retval = HeapTupleGetRawDatum(tup); } else retval = InputFunctionCall(&prodesc->result_in_func, @@ -3235,7 +3235,7 @@ pltcl_build_tuple_result(Tcl_Interp *interp, Tcl_Obj **kvObjv, int kvObjc, /* if result type is domain-over-composite, check domain constraints */ if (call_state->prodesc->fn_retisdomain) - domain_check(HeapTupleGetDatum(tuple), false, + domain_check(HeapTupleGetRawDatum(tuple), false, call_state->prodesc->result_typid, &call_state->prodesc->domain_info, call_state->prodesc->fn_cxt); diff --git a/src/test/modules/test_predtest/test_predtest.c b/src/test/modules/test_predtest/test_predtest.c index 9c0aadd61c..ec83645bbe 100644 --- a/src/test/modules/test_predtest/test_predtest.c +++ b/src/test/modules/test_predtest/test_predtest.c @@ -214,5 +214,6 @@ test_predtest(PG_FUNCTION_ARGS) values[6] = BoolGetDatum(s_r_holds); values[7] = BoolGetDatum(w_r_holds); - PG_RETURN_DATUM(HeapTupleGetDatum(heap_form_tuple(tupdesc, values, nulls))); + PG_RETURN_HEAPTUPLEHEADER_RAW( + heap_form_tuple(tupdesc, values, nulls)->t_data); } -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0002-Expand-the-external-data-before-forming-the-tuple.patch" From 64d02424b961406600d479a95a0d74fec5a385ce Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Thu, 4 Mar 2021 14:55:56 +0530 Subject: [PATCH 2/8] Expand the external data before forming the tuple All the callers of HeapTupleGetDatum and HeapTupleHeaderGetDatum, who might contain the external varlena in their tuple, try to flatten the external varlena before forming the tuple wherever it is possible. Dilip Kumar based on idea by Robert Haas --- src/backend/executor/execExprInterp.c | 29 +++++++++++++++++++++++++-- src/backend/utils/adt/jsonfuncs.c | 8 ++++++-- src/pl/plpgsql/src/pl_exec.c | 5 ++++- 3 files changed, 37 insertions(+), 5 deletions(-) diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index 20949a5dec..c3754acca4 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -2840,12 +2840,25 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op) { HeapTuple tuple; + /* Retrieve externally stored values and decompress. */ + for (int i = 0; i < op->d.row.tupdesc->natts; i++) + { + Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i); + + if (op->d.row.elemnulls[i] || attr->attlen != -1 || + !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i]))) + continue; + + op->d.row.elemvalues[i] = + PointerGetDatum(PG_DETOAST_DATUM_PACKED(op->d.row.elemvalues[i])); + } + /* build tuple from evaluated field values */ tuple = heap_form_tuple(op->d.row.tupdesc, op->d.row.elemvalues, op->d.row.elemnulls); - *op->resvalue = HeapTupleGetDatum(tuple); + *op->resvalue = HeapTupleGetRawDatum(tuple); *op->resnull = false; } @@ -3085,12 +3098,24 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext { HeapTuple tuple; + /* Retrieve externally stored values and decompress. */ + for (int i = 0; i < (*op->d.fieldstore.argdesc)->natts; i++) + { + Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i); + + if (op->d.fieldstore.nulls[i] || attr->attlen != -1 || + !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i]))) + continue; + op->d.fieldstore.values[i] = PointerGetDatum( + PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i])); + } + /* argdesc should already be valid from the DeForm step */ tuple = heap_form_tuple(*op->d.fieldstore.argdesc, op->d.fieldstore.values, op->d.fieldstore.nulls); - *op->resvalue = HeapTupleGetDatum(tuple); + *op->resvalue = HeapTupleGetRawDatum(tuple); *op->resnull = false; } diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c index 511467280f..c3d464f42b 100644 --- a/src/backend/utils/adt/jsonfuncs.c +++ b/src/backend/utils/adt/jsonfuncs.c @@ -2968,7 +2968,7 @@ populate_composite(CompositeIOData *io, /* populate resulting record tuple */ tuple = populate_record(io->tupdesc, &io->record_io, defaultval, mcxt, &jso); - result = HeapTupleHeaderGetDatum(tuple); + result = HeapTupleHeaderGetRawDatum(tuple); JsObjectFree(&jso); } @@ -3387,6 +3387,10 @@ populate_record(TupleDesc tupdesc, nulls[i] ? (Datum) 0 : values[i], &field, &nulls[i]); + + if (!nulls[i] && att->attlen == -1 && + VARATT_IS_EXTERNAL(DatumGetPointer(values[i]))) + values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i])); } res = heap_form_tuple(tupdesc, values, nulls); @@ -3765,7 +3769,7 @@ populate_recordset_record(PopulateRecordsetState *state, JsObject *obj) /* if it's domain over composite, check domain constraints */ if (cache->c.typcat == TYPECAT_COMPOSITE_DOMAIN) - domain_check(HeapTupleHeaderGetDatum(tuphead), false, + domain_check(HeapTupleHeaderGetRawDatum(tuphead), false, cache->argtype, &cache->c.io.composite.domain_info, cache->fn_mcxt); diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c index b4c70aaa7f..fd073767bc 100644 --- a/src/pl/plpgsql/src/pl_exec.c +++ b/src/pl/plpgsql/src/pl_exec.c @@ -5326,7 +5326,7 @@ exec_eval_datum(PLpgSQL_execstate *estate, elog(ERROR, "row not compatible with its own tupdesc"); *typeid = row->rowtupdesc->tdtypeid; *typetypmod = row->rowtupdesc->tdtypmod; - *value = HeapTupleGetDatum(tup); + *value = HeapTupleGetRawDatum(tup); *isnull = false; MemoryContextSwitchTo(oldcontext); break; @@ -7300,6 +7300,9 @@ make_tuple_from_row(PLpgSQL_execstate *estate, &dvalues[i], &nulls[i]); if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid) return NULL; + if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 && + VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i]))) + dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i])); /* XXX should we insist on typmod match, too? */ } -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0003-Disallow-compressed-data-inside-container-types.patch" From 4489b8e89557c92d932dd944db07c99eea8c088c Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Thu, 4 Mar 2021 16:33:10 +0530 Subject: [PATCH 3/8] Disallow compressed data inside container types Currently, we have a general rule that Datums of container types (rows, arrays, ranges, etc) must not contain any external TOAST pointers. But the rule for the compressed data is not defined and no specific rule is followed e.g. while constructing the array we decompress the compressed field but while constructing the row in some cases we don't decompress the compressed data whereas in the other cases we only decompress while flattening the external toast pointers. This patch make a general rule for the compressed data i.e. we don't allow the compressed data in the container type. Dilip Kumar based on idea from Robert Haas --- src/backend/access/common/heaptuple.c | 9 +-- src/backend/access/heap/heaptoast.c | 4 +- src/backend/executor/execExprInterp.c | 6 +- src/backend/executor/execTuples.c | 4 -- src/backend/utils/adt/expandedrecord.c | 76 +++++++++----------------- src/backend/utils/adt/jsonfuncs.c | 3 +- src/include/funcapi.h | 4 +- src/pl/plpgsql/src/pl_exec.c | 3 +- 8 files changed, 38 insertions(+), 71 deletions(-) diff --git a/src/backend/access/common/heaptuple.c b/src/backend/access/common/heaptuple.c index c36c283253..eb9f016dfa 100644 --- a/src/backend/access/common/heaptuple.c +++ b/src/backend/access/common/heaptuple.c @@ -984,15 +984,12 @@ Datum heap_copy_tuple_as_datum(HeapTuple tuple, TupleDesc tupleDesc) { /* - * If the tuple contains any external TOAST pointers, we have to inline - * those fields to meet the conventions for composite-type Datums. + * We have to inline any external/compressed data to meet the conventions + * for composite-type Datums. */ - if (HeapTupleHasExternal(tuple)) - return toast_flatten_tuple_to_datum(tuple->t_data, + return toast_flatten_tuple_to_datum(tuple->t_data, tuple->t_len, tupleDesc); - else - return heap_copy_tuple_as_raw_datum(tuple, tupleDesc); } /* ---------------- diff --git a/src/backend/access/heap/heaptoast.c b/src/backend/access/heap/heaptoast.c index 55bbe1d584..b09462348b 100644 --- a/src/backend/access/heap/heaptoast.c +++ b/src/backend/access/heap/heaptoast.c @@ -589,9 +589,9 @@ toast_build_flattened_tuple(TupleDesc tupleDesc, struct varlena *new_value; new_value = (struct varlena *) DatumGetPointer(new_values[i]); - if (VARATT_IS_EXTERNAL(new_value)) + if (VARATT_IS_EXTERNAL(new_value) || VARATT_IS_COMPRESSED(new_value)) { - new_value = detoast_external_attr(new_value); + new_value = detoast_attr(new_value); new_values[i] = PointerGetDatum(new_value); freeable_values[num_to_free++] = (Pointer) new_value; } diff --git a/src/backend/executor/execExprInterp.c b/src/backend/executor/execExprInterp.c index c3754acca4..71e6f41fee 100644 --- a/src/backend/executor/execExprInterp.c +++ b/src/backend/executor/execExprInterp.c @@ -2845,8 +2845,7 @@ ExecEvalRow(ExprState *state, ExprEvalStep *op) { Form_pg_attribute attr = TupleDescAttr(op->d.row.tupdesc, i); - if (op->d.row.elemnulls[i] || attr->attlen != -1 || - !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.row.elemvalues[i]))) + if (op->d.row.elemnulls[i] || attr->attlen != -1) continue; op->d.row.elemvalues[i] = @@ -3103,8 +3102,7 @@ ExecEvalFieldStoreForm(ExprState *state, ExprEvalStep *op, ExprContext *econtext { Form_pg_attribute attr = TupleDescAttr(*op->d.fieldstore.argdesc, i); - if (op->d.fieldstore.nulls[i] || attr->attlen != -1 || - !VARATT_IS_EXTERNAL(DatumGetPointer(op->d.fieldstore.values[i]))) + if (op->d.fieldstore.nulls[i] || attr->attlen != -1) continue; op->d.fieldstore.values[i] = PointerGetDatum( PG_DETOAST_DATUM_PACKED(op->d.fieldstore.values[i])); diff --git a/src/backend/executor/execTuples.c b/src/backend/executor/execTuples.c index 73c35df9c9..f11546468e 100644 --- a/src/backend/executor/execTuples.c +++ b/src/backend/executor/execTuples.c @@ -2207,10 +2207,6 @@ HeapTupleHeaderGetDatum(HeapTupleHeader tuple) Datum result; TupleDesc tupDesc; - /* No work if there are no external TOAST pointers in the tuple */ - if (!HeapTupleHeaderHasExternal(tuple)) - return PointerGetDatum(tuple); - /* Use the type data saved by heap_form_tuple to look up the rowtype */ tupDesc = lookup_rowtype_tupdesc(HeapTupleHeaderGetTypeId(tuple), HeapTupleHeaderGetTypMod(tuple)); diff --git a/src/backend/utils/adt/expandedrecord.c b/src/backend/utils/adt/expandedrecord.c index e19491ecf7..3cbc256671 100644 --- a/src/backend/utils/adt/expandedrecord.c +++ b/src/backend/utils/adt/expandedrecord.c @@ -673,14 +673,6 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr) erh->er_typmod = tupdesc->tdtypmod; } - /* - * If we have a valid flattened value without out-of-line fields, we can - * just use it as-is. - */ - if (erh->flags & ER_FLAG_FVALUE_VALID && - !(erh->flags & ER_FLAG_HAVE_EXTERNAL)) - return erh->fvalue->t_len; - /* If we have a cached size value, believe that */ if (erh->flat_size) return erh->flat_size; @@ -693,38 +685,36 @@ ER_get_flat_size(ExpandedObjectHeader *eohptr) tupdesc = erh->er_tupdesc; /* - * Composite datums mustn't contain any out-of-line values. + * Composite datums mustn't contain any out-of-line/compressed values. */ - if (erh->flags & ER_FLAG_HAVE_EXTERNAL) + for (i = 0; i < erh->nfields; i++) { - for (i = 0; i < erh->nfields; i++) - { - Form_pg_attribute attr = TupleDescAttr(tupdesc, i); + Form_pg_attribute attr = TupleDescAttr(tupdesc, i); - if (!erh->dnulls[i] && - !attr->attbyval && attr->attlen == -1 && - VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i]))) - { - /* - * expanded_record_set_field_internal can do the actual work - * of detoasting. It needn't recheck domain constraints. - */ - expanded_record_set_field_internal(erh, i + 1, - erh->dvalues[i], false, - true, - false); - } + if (!erh->dnulls[i] && + !attr->attbyval && attr->attlen == -1 && + (VARATT_IS_EXTERNAL(DatumGetPointer(erh->dvalues[i])) || + VARATT_IS_COMPRESSED(DatumGetPointer(erh->dvalues[i])))) + { + /* + * expanded_record_set_field_internal can do the actual work + * of detoasting. It needn't recheck domain constraints. + */ + expanded_record_set_field_internal(erh, i + 1, + erh->dvalues[i], false, + true, + false); } - - /* - * We have now removed all external field values, so we can clear the - * flag about them. This won't cause ER_flatten_into() to mistakenly - * take the fast path, since expanded_record_set_field() will have - * cleared ER_FLAG_FVALUE_VALID. - */ - erh->flags &= ~ER_FLAG_HAVE_EXTERNAL; } + /* + * We have now removed all external field values, so we can clear the + * flag about them. This won't cause ER_flatten_into() to mistakenly + * take the fast path, since expanded_record_set_field() will have + * cleared ER_FLAG_FVALUE_VALID. + */ + erh->flags &= ~ER_FLAG_HAVE_EXTERNAL; + /* Test if we currently have any null values */ hasnull = false; for (i = 0; i < erh->nfields; i++) @@ -770,19 +760,6 @@ ER_flatten_into(ExpandedObjectHeader *eohptr, Assert(erh->er_magic == ER_MAGIC); - /* Easy if we have a valid flattened value without out-of-line fields */ - if (erh->flags & ER_FLAG_FVALUE_VALID && - !(erh->flags & ER_FLAG_HAVE_EXTERNAL)) - { - Assert(allocated_size == erh->fvalue->t_len); - memcpy(tuphdr, erh->fvalue->t_data, allocated_size); - /* The original flattened value might not have datum header fields */ - HeapTupleHeaderSetDatumLength(tuphdr, allocated_size); - HeapTupleHeaderSetTypeId(tuphdr, erh->er_typeid); - HeapTupleHeaderSetTypMod(tuphdr, erh->er_typmod); - return; - } - /* Else allocation should match previous get_flat_size result */ Assert(allocated_size == erh->flat_size); @@ -1155,11 +1132,12 @@ expanded_record_set_field_internal(ExpandedRecordHeader *erh, int fnumber, if (expand_external) { if (attr->attlen == -1 && - VARATT_IS_EXTERNAL(DatumGetPointer(newValue))) + (VARATT_IS_EXTERNAL(DatumGetPointer(newValue)) || + VARATT_IS_COMPRESSED(DatumGetPointer(newValue)))) { /* Detoasting should be done in short-lived context. */ oldcxt = MemoryContextSwitchTo(get_short_term_cxt(erh)); - newValue = PointerGetDatum(detoast_external_attr((struct varlena *) DatumGetPointer(newValue))); + newValue = PointerGetDatum(detoast_attr((struct varlena *) DatumGetPointer(newValue))); MemoryContextSwitchTo(oldcxt); } else diff --git a/src/backend/utils/adt/jsonfuncs.c b/src/backend/utils/adt/jsonfuncs.c index c3d464f42b..821aa8fbdb 100644 --- a/src/backend/utils/adt/jsonfuncs.c +++ b/src/backend/utils/adt/jsonfuncs.c @@ -3388,8 +3388,7 @@ populate_record(TupleDesc tupdesc, &field, &nulls[i]); - if (!nulls[i] && att->attlen == -1 && - VARATT_IS_EXTERNAL(DatumGetPointer(values[i]))) + if (!nulls[i] && att->attlen == -1) values[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(values[i])); } diff --git a/src/include/funcapi.h b/src/include/funcapi.h index 8ba7ae211f..c869012873 100644 --- a/src/include/funcapi.h +++ b/src/include/funcapi.h @@ -208,10 +208,10 @@ extern TupleDesc build_function_result_tupdesc_t(HeapTuple procTuple); * Macro declarations/inline functions: * HeapTupleHeaderGetRawDatum(HeapTupleHeader tuple) - same as * HeapTupleHeaderGetDatum but the input tuple should not contain - * external varlena + * external/compressed varlena * HeapTupleGetDatum(HeapTuple tuple) - convert a HeapTuple to a Datum. * HeapTupleGetRawDatum(HeapTuple tuple) - same as HeapTupleGetDatum - * but the input tuple should not contain external varlena + * but the input tuple should not contain external/compressed varlena * * Obsolete routines and macros: * TupleDesc RelationNameGetTupleDesc(const char *relname) - Use to get a diff --git a/src/pl/plpgsql/src/pl_exec.c b/src/pl/plpgsql/src/pl_exec.c index fd073767bc..0519253cbe 100644 --- a/src/pl/plpgsql/src/pl_exec.c +++ b/src/pl/plpgsql/src/pl_exec.c @@ -7300,8 +7300,7 @@ make_tuple_from_row(PLpgSQL_execstate *estate, &dvalues[i], &nulls[i]); if (fieldtypeid != TupleDescAttr(tupdesc, i)->atttypid) return NULL; - if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1 && - VARATT_IS_EXTERNAL(DatumGetPointer(dvalues[i]))) + if (!nulls[i] && TupleDescAttr(tupdesc, i)->attlen == -1) dvalues[i] = PointerGetDatum(PG_DETOAST_DATUM_PACKED(dvalues[i])); /* XXX should we insist on typmod match, too? */ } -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0004-Built-in-compression-method.patch" From d0d7896a88f009181f9540eaea8cf2a89a552637 Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Fri, 5 Mar 2021 09:33:08 +0530 Subject: [PATCH 4/8] Built-in compression method Add syntax allowing a compression method to be specified. As of now there is only 2 option for build-in compression method (pglz, lz4) which can be set while creating a table or adding a new column. No option for altering the compression method for an existing column. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/20171213151818.75a20259@postgrespro.ru https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40mail.gmail.com https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail.gmail.com https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail.gmail.com --- configure | 215 ++++++++++++ configure.ac | 19 ++ doc/src/sgml/catalogs.sgml | 10 + doc/src/sgml/ref/create_table.sgml | 33 +- doc/src/sgml/ref/psql-ref.sgml | 11 + src/backend/access/brin/brin_tuple.c | 5 +- src/backend/access/common/Makefile | 1 + src/backend/access/common/detoast.c | 71 ++-- src/backend/access/common/indextuple.c | 3 +- src/backend/access/common/toast_compression.c | 308 ++++++++++++++++++ src/backend/access/common/toast_internals.c | 54 ++- src/backend/access/common/tupdesc.c | 8 + src/backend/access/heap/heapam_handler.c | 22 ++ src/backend/access/table/toast_helper.c | 5 +- src/backend/bootstrap/bootstrap.c | 5 + src/backend/catalog/genbki.pl | 3 + src/backend/catalog/heap.c | 4 + src/backend/catalog/index.c | 1 + src/backend/catalog/toasting.c | 6 + src/backend/commands/tablecmds.c | 110 +++++++ src/backend/executor/nodeModifyTable.c | 128 ++++++++ src/backend/nodes/copyfuncs.c | 1 + src/backend/nodes/equalfuncs.c | 1 + src/backend/nodes/nodeFuncs.c | 2 + src/backend/nodes/outfuncs.c | 1 + src/backend/parser/gram.y | 26 +- src/backend/parser/parse_utilcmd.c | 9 + src/backend/utils/adt/varlena.c | 41 +++ src/bin/pg_dump/pg_backup.h | 1 + src/bin/pg_dump/pg_dump.c | 39 +++ src/bin/pg_dump/pg_dump.h | 1 + src/bin/pg_dump/t/002_pg_dump.pl | 12 +- src/bin/psql/describe.c | 31 +- src/bin/psql/help.c | 2 + src/bin/psql/settings.h | 1 + src/bin/psql/startup.c | 10 + src/include/access/detoast.h | 8 + src/include/access/toast_compression.h | 119 +++++++ src/include/access/toast_helper.h | 1 + src/include/access/toast_internals.h | 20 +- src/include/catalog/pg_attribute.h | 8 +- src/include/catalog/pg_proc.dat | 4 + src/include/executor/executor.h | 4 +- src/include/nodes/execnodes.h | 6 + src/include/nodes/parsenodes.h | 2 + src/include/parser/kwlist.h | 1 + src/include/pg_config.h.in | 3 + src/include/postgres.h | 14 +- src/test/regress/expected/compression.out | 267 +++++++++++++++ src/test/regress/expected/compression_1.out | 269 +++++++++++++++ src/test/regress/parallel_schedule | 2 +- src/test/regress/pg_regress_main.c | 4 +- src/test/regress/serial_schedule | 1 + src/test/regress/sql/compression.sql | 113 +++++++ src/tools/msvc/Solution.pm | 1 + src/tools/pgindent/typedefs.list | 1 + 56 files changed, 1962 insertions(+), 86 deletions(-) create mode 100644 src/backend/access/common/toast_compression.c create mode 100644 src/include/access/toast_compression.h create mode 100644 src/test/regress/expected/compression.out create mode 100644 src/test/regress/expected/compression_1.out create mode 100644 src/test/regress/sql/compression.sql diff --git a/configure b/configure index fad817bb38..87ed16060e 100755 --- a/configure +++ b/configure @@ -654,6 +654,8 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl +LZ4_LIBS +LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -699,6 +701,7 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +with_lz4 with_zlib with_system_tzdata with_libxslt @@ -864,6 +867,7 @@ with_libxml with_libxslt with_system_tzdata with_zlib +with_lz4 with_gnu_ld with_ssl with_openssl @@ -895,6 +899,8 @@ LDFLAGS_EX LDFLAGS_SL PERL PYTHON +LZ4_CFLAGS +LZ4_LIBS MSGFMT TCLSH' @@ -1569,6 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib + --with-lz4 build with LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -1600,6 +1607,8 @@ Some influential environment variables: LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8563,6 +8572,41 @@ fi +# +# LZ4 +# +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking whether to build with LZ4 support" >&5 +$as_echo_n "checking whether to build with LZ4 support... " >&6; } + + + +# Check whether --with-lz4 was given. +if test "${with_lz4+set}" = set; then : + withval=$with_lz4; + case $withval in + yes) + +$as_echo "#define USE_LZ4 1" >>confdefs.h + + ;; + no) + : + ;; + *) + as_fn_error $? "no argument expected for --with-lz4 option" "$LINENO" 5 + ;; + esac + +else + with_lz4=no + +fi + + +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $with_lz4" >&5 +$as_echo "$with_lz4" >&6; } + + # # Assignments # @@ -12110,6 +12154,147 @@ fi fi +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see . +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 +$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } +if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : + $as_echo_n "(cached) " >&6 +else + ac_check_lib_save_LIBS=$LIBS +LIBS="-llz4 $LIBS" +cat confdefs.h - <<_ACEOF >conftest.$ac_ext +/* end confdefs.h. */ + +/* Override any GCC internal prototype to avoid an error. + Use char because int might match the return type of a GCC + builtin and then its argument prototype would still apply. */ +#ifdef __cplusplus +extern "C" +#endif +char LZ4_compress (); +int +main () +{ +return LZ4_compress (); + ; + return 0; +} +_ACEOF +if ac_fn_c_try_link "$LINENO"; then : + ac_cv_lib_lz4_LZ4_compress=yes +else + ac_cv_lib_lz4_LZ4_compress=no +fi +rm -f core conftest.err conftest.$ac_objext \ + conftest$ac_exeext conftest.$ac_ext +LIBS=$ac_check_lib_save_LIBS +fi +{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 +$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } +if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LIBLZ4 1 +_ACEOF + + LIBS="-llz4 $LIBS" + +else + as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 +fi + +fi + if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13376,6 +13561,36 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi +fi + +if test "$with_lz4" = yes; then + for ac_header in lz4/lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_LZ4_H 1 +_ACEOF + +else + for ac_header in lz4.h +do : + ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" +if test "x$ac_cv_header_lz4_h" = xyes; then : + cat >>confdefs.h <<_ACEOF +#define HAVE_LZ4_H 1 +_ACEOF + +else + as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 +fi + +done + +fi + +done + fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index 0ed53571dd..c8f199dc5e 100644 --- a/configure.ac +++ b/configure.ac @@ -986,6 +986,15 @@ PGAC_ARG_BOOL(with, zlib, yes, [do not use Zlib]) AC_SUBST(with_zlib) +# +# LZ4 +# +AC_MSG_CHECKING([whether to build with LZ4 support]) +PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +AC_MSG_RESULT([$with_lz4]) +AC_SUBST(with_lz4) + # # Assignments # @@ -1174,6 +1183,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) +fi + if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1407,6 +1421,11 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi +if test "$with_lz4" = yes; then + AC_CHECK_HEADERS(lz4/lz4.h, [], + [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) +fi + if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) diff --git a/doc/src/sgml/catalogs.sgml b/doc/src/sgml/catalogs.sgml index b1de6d0674..5fdc80ff3d 100644 --- a/doc/src/sgml/catalogs.sgml +++ b/doc/src/sgml/catalogs.sgml @@ -1355,6 +1355,16 @@ + + + attcompression char + + + The current compression method of the column. Must be InvalidCompressionMethod + if and only if typstorage is 'plain' or 'external'. + + + attacl aclitem[] diff --git a/doc/src/sgml/ref/create_table.sgml b/doc/src/sgml/ref/create_table.sgml index 3b2b227683..7c8fc77983 100644 --- a/doc/src/sgml/ref/create_table.sgml +++ b/doc/src/sgml/ref/create_table.sgml @@ -22,7 +22,7 @@ PostgreSQL documentation CREATE [ [ GLOBAL | LOCAL ] { TEMPORARY | TEMP } | UNLOGGED ] TABLE [ IF NOT EXISTS ] table_name ( [ - { column_name data_type [ COLLATE collation ] [ column_constraint [ ... ] ] + { column_name data_type [ COLLATE collation ] [ COMPRESSION compression_method ] [ column_constraint [ ... ] ] | table_constraint | LIKE source_table [ like_option ... ] } [, ... ] @@ -288,6 +288,26 @@ WITH ( MODULUS numeric_literal, REM + + COMPRESSION compression_method + + + The COMPRESSION clause sets the compression method + for a column. Compression is supported only for variable-width data + types, and is used only for columns whose storage type is main or + extended. (See for information on + column storage types.) Setting this property for a partitioned table + has no direct effect, because such tables have no storage of their own, + but the configured value is inherited by newly-created partitions. + The supported compression methods are pglz and + lz4. lz4 is available only if + --with-lz4 was used when building + PostgreSQL. The default + is pglz. + + + + INHERITS ( parent_table [, ... ] ) @@ -605,6 +625,17 @@ WITH ( MODULUS numeric_literal, REM + + INCLUDING COMPRESSION + + + Compression method of the columns will be copied. The default + behavior is to exclude compression methods, resulting in columns + having the default compression method. + + + + INCLUDING CONSTRAINTS diff --git a/doc/src/sgml/ref/psql-ref.sgml b/doc/src/sgml/ref/psql-ref.sgml index 13c1edfa4d..01ec9b8b0a 100644 --- a/doc/src/sgml/ref/psql-ref.sgml +++ b/doc/src/sgml/ref/psql-ref.sgml @@ -3863,6 +3863,17 @@ bar + + HIDE_TOAST_COMPRESSION + + + If this variable is set to true, column + compression method details are not displayed. This is mainly + useful for regression tests. + + + + HIDE_TABLEAM diff --git a/src/backend/access/brin/brin_tuple.c b/src/backend/access/brin/brin_tuple.c index a7eb1c9473..0ab5712c71 100644 --- a/src/backend/access/brin/brin_tuple.c +++ b/src/backend/access/brin/brin_tuple.c @@ -213,7 +213,10 @@ brin_form_tuple(BrinDesc *brdesc, BlockNumber blkno, BrinMemTuple *tuple, (atttype->typstorage == TYPSTORAGE_EXTENDED || atttype->typstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(value); + Form_pg_attribute att = TupleDescAttr(brdesc->bd_tupdesc, + keyno); + Datum cvalue = toast_compress_datum(value, + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/Makefile b/src/backend/access/common/Makefile index 5a007d63f1..b9aff0ccfd 100644 --- a/src/backend/access/common/Makefile +++ b/src/backend/access/common/Makefile @@ -25,6 +25,7 @@ OBJS = \ scankey.o \ session.o \ syncscan.o \ + toast_compression.o \ toast_internals.o \ tupconvert.o \ tupdesc.o diff --git a/src/backend/access/common/detoast.c b/src/backend/access/common/detoast.c index d1cdbaf648..9f05a4502b 100644 --- a/src/backend/access/common/detoast.c +++ b/src/backend/access/common/detoast.c @@ -16,6 +16,7 @@ #include "access/detoast.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/toast_internals.h" #include "common/int.h" #include "common/pg_lzcompress.h" @@ -456,6 +457,37 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, return result; } +/* ---------- + * toast_get_compression_method + * + * Returns compression method for the compressed varlena. If it is not + * compressed then returns InvalidCompressionMethod. + */ +char +toast_get_compression_method(struct varlena *attr) +{ + if (VARATT_IS_EXTERNAL_ONDISK(attr)) + { + struct varatt_external toast_pointer; + + VARATT_EXTERNAL_GET_POINTER(toast_pointer, attr); + + /* fast path for non-compressed external datums */ + if (!VARATT_EXTERNAL_IS_COMPRESSED(toast_pointer)) + return InvalidCompressionMethod; + + /* + * Fetch just enough of the value to examine the compression header, + * so that we can find out the compression method. + */ + attr = toast_fetch_datum_slice(attr, 0, VARHDRSZ); + } + else if (!VARATT_IS_COMPRESSED(attr)) + return InvalidCompressionMethod; + + return CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr)); +} + /* ---------- * toast_decompress_datum - * @@ -464,21 +496,18 @@ toast_fetch_datum_slice(struct varlena *attr, int32 sliceoffset, static struct varlena * toast_decompress_datum(struct varlena *attr) { - struct varlena *result; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) - palloc(TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - SET_VARSIZE(result, TOAST_COMPRESS_RAWSIZE(attr) + VARHDRSZ); - - if (pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - TOAST_COMPRESS_SIZE(attr), - VARDATA(result), - TOAST_COMPRESS_RAWSIZE(attr), true) < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - return result; + return cmroutine->datum_decompress(attr); } @@ -492,22 +521,18 @@ toast_decompress_datum(struct varlena *attr) static struct varlena * toast_decompress_datum_slice(struct varlena *attr, int32 slicelength) { - struct varlena *result; - int32 rawsize; + const CompressionRoutine *cmroutine; Assert(VARATT_IS_COMPRESSED(attr)); - result = (struct varlena *) palloc(slicelength + VARHDRSZ); - - rawsize = pglz_decompress(TOAST_COMPRESS_RAWDATA(attr), - VARSIZE(attr) - TOAST_COMPRESS_HDRSZ, - VARDATA(result), - slicelength, false); - if (rawsize < 0) - elog(ERROR, "compressed data is corrupted"); + /* + * Get compression handler routines, using the compression id stored in the + * toast header. + */ + cmroutine = GetCompressionRoutines( + CompressionIdToMethod(TOAST_COMPRESS_METHOD(attr))); - SET_VARSIZE(result, rawsize + VARHDRSZ); - return result; + return cmroutine->datum_decompress_slice(attr, slicelength); } /* ---------- diff --git a/src/backend/access/common/indextuple.c b/src/backend/access/common/indextuple.c index b72a138497..1f6b7b77d4 100644 --- a/src/backend/access/common/indextuple.c +++ b/src/backend/access/common/indextuple.c @@ -103,7 +103,8 @@ index_form_tuple(TupleDesc tupleDescriptor, (att->attstorage == TYPSTORAGE_EXTENDED || att->attstorage == TYPSTORAGE_MAIN)) { - Datum cvalue = toast_compress_datum(untoasted_values[i]); + Datum cvalue = toast_compress_datum(untoasted_values[i], + att->attcompression); if (DatumGetPointer(cvalue) != NULL) { diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c new file mode 100644 index 0000000000..3463b42438 --- /dev/null +++ b/src/backend/access/common/toast_compression.c @@ -0,0 +1,308 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.c + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * + * IDENTIFICATION + * src/backend/access/common/toast_compression.c + * + *------------------------------------------------------------------------- + */ +#include "postgres.h" + +#ifdef HAVE_LIBLZ4 +#include +#endif + +#include "access/toast_compression.h" +#include "common/pg_lzcompress.h" +#include "fmgr.h" +#include "utils/builtins.h" + +static struct varlena *pglz_cmcompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress(const struct varlena *value); +static struct varlena *pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength); +static struct varlena *lz4_cmcompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress(const struct varlena *value); +static struct varlena *lz4_cmdecompress_slice(const struct varlena *value, + int32 slicelength); + +#define NO_LZ4_SUPPORT() \ + ereport(ERROR, \ + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), \ + errmsg("unsupported LZ4 compression method"), \ + errdetail("This functionality requires the server to be built with lz4 support."), \ + errhint("You need to rebuild PostgreSQL using --with-lz4."))) + +/* handler routines for pglz and lz4 built-in compression methods */ +const CompressionRoutine toast_compression[] = +{ + { + .cmname = "pglz", + .datum_compress = pglz_cmcompress, + .datum_decompress = pglz_cmdecompress, + .datum_decompress_slice = pglz_cmdecompress_slice + }, + { + .cmname = "lz4", + .datum_compress = lz4_cmcompress, + .datum_decompress = lz4_cmdecompress, + .datum_decompress_slice = lz4_cmdecompress_slice + } +}; + +/* + * pglz_cmcompress - compression routine for pglz compression method + * + * Compresses source into dest using the default strategy. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +pglz_cmcompress(const struct varlena *value) +{ + int32 valsize, + len; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + /* + * No point in wasting a palloc cycle if value size is outside the allowed + * range for compression. + */ + if (valsize < PGLZ_strategy_default->min_input_size || + valsize > PGLZ_strategy_default->max_input_size) + return NULL; + + /* + * Figure out the maximum possible size of the pglz output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + + VARHDRSZ_COMPRESS); + + len = pglz_compress(VARDATA_ANY(value), + valsize, + (char *) tmp + VARHDRSZ_COMPRESS, + NULL); + if (len < 0) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +} + +/* + * pglz_cmdecompress - decompression routine for pglz compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress(const struct varlena *value) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + VARRAWSIZE_4B_C(value), true); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * pglz_decompress - slice decompression routine for pglz compression method + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +pglz_cmdecompress_slice(const struct varlena *value, + int32 slicelength) +{ + struct varlena *result; + int32 rawsize; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = pglz_decompress((char *) value + VARHDRSZ_COMPRESS, + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARDATA(result), + slicelength, false); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed pglz data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +} + +/* + * lz4_cmcompress - compression routine for lz4 compression method + * + * Compresses source into dest using the LZ4 defaults. Returns the + * compressed varlena, or NULL if compression fails. + */ +static struct varlena * +lz4_cmcompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 valsize; + int32 len; + int32 max_size; + struct varlena *tmp = NULL; + + valsize = VARSIZE_ANY_EXHDR(value); + + /* + * Figure out the maximum possible size of the LZ4 output, add the bytes + * that will be needed for varlena overhead, and allocate that amount. + */ + max_size = LZ4_compressBound(valsize); + tmp = (struct varlena *) palloc(max_size + VARHDRSZ_COMPRESS); + + len = LZ4_compress_default(VARDATA_ANY(value), + (char *) tmp + VARHDRSZ_COMPRESS, + valsize, max_size); + if (len <= 0) + elog(ERROR, "could not compress data with lz4"); + + /* data is incompressible so just free the memory and return NULL */ + if (len > valsize) + { + pfree(tmp); + return NULL; + } + + SET_VARSIZE_COMPRESSED(tmp, len + VARHDRSZ_COMPRESS); + + return tmp; +#endif +} + +/* + * lz4_cmdecompress - decompression routine for lz4 compression method + * + * Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress(const struct varlena *value) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(VARRAWSIZE_4B_C(value) + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + VARRAWSIZE_4B_C(value)); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * lz4_cmdecompress_slice - slice decompression routine for lz4 compression + * + * Decompresses part of the data. Returns the decompressed varlena. + */ +static struct varlena * +lz4_cmdecompress_slice(const struct varlena *value, int32 slicelength) +{ +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#elif LZ4_VERSION_NUMBER < 10803 + return lz4_cmdecompress(value); +#else + int32 rawsize; + struct varlena *result; + + /* allocate memory for the uncompressed data */ + result = (struct varlena *) palloc(slicelength + VARHDRSZ); + + /* decompress the data */ + rawsize = LZ4_decompress_safe_partial((char *) value + VARHDRSZ_COMPRESS, + VARDATA(result), + VARSIZE(value) - VARHDRSZ_COMPRESS, + slicelength, + slicelength); + if (rawsize < 0) + ereport(ERROR, + (errcode(ERRCODE_DATA_CORRUPTED), + errmsg_internal("compressed lz4 data is corrupt"))); + + SET_VARSIZE(result, rawsize + VARHDRSZ); + + return result; +#endif +} + +/* + * CompressionNameToMethod - Get compression method from compression name + * + * Search in the available built-in methods. If the compression not found + * in the built-in methods then return InvalidCompressionMethod. + */ +char +CompressionNameToMethod(char *compression) +{ + if (strcmp(toast_compression[PGLZ_COMPRESSION_ID].cmname, + compression) == 0) + return PGLZ_COMPRESSION; + else if (strcmp(toast_compression[LZ4_COMPRESSION_ID].cmname, + compression) == 0) + { +#ifndef HAVE_LIBLZ4 + NO_LZ4_SUPPORT(); +#endif + return LZ4_COMPRESSION; + } + + return InvalidCompressionMethod; +} + +/* + * GetCompressionRoutines - Get compression handler routines + */ +const CompressionRoutine* +GetCompressionRoutines(char method) +{ + return &toast_compression[CompressionMethodToId(method)]; +} diff --git a/src/backend/access/common/toast_internals.c b/src/backend/access/common/toast_internals.c index 9b9da0f41b..69dd9492f6 100644 --- a/src/backend/access/common/toast_internals.c +++ b/src/backend/access/common/toast_internals.c @@ -44,46 +44,42 @@ static bool toastid_valueid_exists(Oid toastrelid, Oid valueid); * ---------- */ Datum -toast_compress_datum(Datum value) +toast_compress_datum(Datum value, char cmethod) { - struct varlena *tmp; - int32 valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); - int32 len; + struct varlena *tmp = NULL; + int32 valsize; + const CompressionRoutine *cmroutine = NULL; Assert(!VARATT_IS_EXTERNAL(DatumGetPointer(value))); Assert(!VARATT_IS_COMPRESSED(DatumGetPointer(value))); - /* - * No point in wasting a palloc cycle if value size is out of the allowed - * range for compression - */ - if (valsize < PGLZ_strategy_default->min_input_size || - valsize > PGLZ_strategy_default->max_input_size) - return PointerGetDatum(NULL); + Assert(CompressionMethodIsValid(cmethod)); - tmp = (struct varlena *) palloc(PGLZ_MAX_OUTPUT(valsize) + - TOAST_COMPRESS_HDRSZ); + /* get the handler routines for the compression method */ + cmroutine = GetCompressionRoutines(cmethod); + + /* call the actual compression function */ + tmp = cmroutine->datum_compress((const struct varlena *) value); + if (!tmp) + return PointerGetDatum(NULL); /* - * We recheck the actual size even if pglz_compress() reports success, - * because it might be satisfied with having saved as little as one byte - * in the compressed data --- which could turn into a net loss once you - * consider header and alignment padding. Worst case, the compressed - * format might require three padding bytes (plus header, which is - * included in VARSIZE(tmp)), whereas the uncompressed format would take - * only one header byte and no padding if the value is short enough. So - * we insist on a savings of more than 2 bytes to ensure we have a gain. + * We recheck the actual size even if compression reports success, because + * it might be satisfied with having saved as little as one byte in the + * compressed data --- which could turn into a net loss once you consider + * header and alignment padding. Worst case, the compressed format might + * require three padding bytes (plus header, which is included in + * VARSIZE(tmp)), whereas the uncompressed format would take only one + * header byte and no padding if the value is short enough. So we insist + * on a savings of more than 2 bytes to ensure we have a gain. */ - len = pglz_compress(VARDATA_ANY(DatumGetPointer(value)), - valsize, - TOAST_COMPRESS_RAWDATA(tmp), - PGLZ_strategy_default); - if (len >= 0 && - len + TOAST_COMPRESS_HDRSZ < valsize - 2) + valsize = VARSIZE_ANY_EXHDR(DatumGetPointer(value)); + + if (VARSIZE(tmp) < valsize - 2) { - TOAST_COMPRESS_SET_RAWSIZE(tmp, valsize); - SET_VARSIZE_COMPRESSED(tmp, len + TOAST_COMPRESS_HDRSZ); /* successful compression */ + TOAST_COMPRESS_SET_SIZE_AND_METHOD(tmp, valsize, + CompressionMethodToId(cmethod)); return PointerGetDatum(tmp); } else diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index 902f59440c..a66d1113db 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -20,6 +20,7 @@ #include "postgres.h" #include "access/htup_details.h" +#include "access/toast_compression.h" #include "access/tupdesc_details.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" @@ -470,6 +471,8 @@ equalTupleDescs(TupleDesc tupdesc1, TupleDesc tupdesc2) return false; if (attr1->attcollation != attr2->attcollation) return false; + if (attr1->attcompression != attr2->attcompression) + return false; /* attacl, attoptions and attfdwoptions are not even present... */ } @@ -664,6 +667,11 @@ TupleDescInitEntry(TupleDesc desc, att->attstorage = typeForm->typstorage; att->attcollation = typeForm->typcollation; + if (IsStorageCompressible(typeForm->typstorage)) + att->attcompression = DefaultCompressionMethod; + else + att->attcompression = InvalidCompressionMethod; + ReleaseSysCache(tuple); } diff --git a/src/backend/access/heap/heapam_handler.c b/src/backend/access/heap/heapam_handler.c index bd5faf0c1f..99f5f63ea7 100644 --- a/src/backend/access/heap/heapam_handler.c +++ b/src/backend/access/heap/heapam_handler.c @@ -19,6 +19,7 @@ */ #include "postgres.h" +#include "access/detoast.h" #include "access/genam.h" #include "access/heapam.h" #include "access/heaptoast.h" @@ -26,6 +27,7 @@ #include "access/rewriteheap.h" #include "access/syncscan.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/tsmapi.h" #include "access/xact.h" #include "catalog/catalog.h" @@ -2469,6 +2471,26 @@ reform_and_rewrite_tuple(HeapTuple tuple, { if (TupleDescAttr(newTupDesc, i)->attisdropped) isnull[i] = true; + /* + * Since we are rewriting the table, use this opportunity to recompress + * any compressed attribute with current compression method of the + * attribute. Basically, if the compression method of the compressed + * varlena is not same as current compression method of the attribute + * then decompress it so that if it need to be compressed then it will + * be compressed with the current compression method of the attribute. + */ + else if (!isnull[i] && TupleDescAttr(newTupDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + new_value = (struct varlena *) DatumGetPointer(values[i]); + cmethod = toast_get_compression_method(new_value); + + if (IsValidCompression(cmethod) && + TupleDescAttr(newTupDesc, i)->attcompression != cmethod) + values[i] = PointerGetDatum(detoast_attr(new_value)); + } } copiedTuple = heap_form_tuple(newTupDesc, values, isnull); diff --git a/src/backend/access/table/toast_helper.c b/src/backend/access/table/toast_helper.c index fb36151ce5..53f78f9c3e 100644 --- a/src/backend/access/table/toast_helper.c +++ b/src/backend/access/table/toast_helper.c @@ -54,6 +54,7 @@ toast_tuple_init(ToastTupleContext *ttc) ttc->ttc_attr[i].tai_colflags = 0; ttc->ttc_attr[i].tai_oldexternal = NULL; + ttc->ttc_attr[i].tai_compression = att->attcompression; if (ttc->ttc_oldvalues != NULL) { @@ -226,9 +227,11 @@ void toast_tuple_try_compression(ToastTupleContext *ttc, int attribute) { Datum *value = &ttc->ttc_values[attribute]; - Datum new_value = toast_compress_datum(*value); + Datum new_value; ToastAttrInfo *attr = &ttc->ttc_attr[attribute]; + new_value = toast_compress_datum(*value, attr->tai_compression); + if (DatumGetPointer(new_value) != NULL) { /* successful compression */ diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index 6f615e6622..af0c0aa77e 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -21,6 +21,7 @@ #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog_internal.h" #include "bootstrap/bootstrap.h" @@ -731,6 +732,10 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->attcacheoff = -1; attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; + if (IsStorageCompressible(attrtypes[attnum]->attstorage)) + attrtypes[attnum]->attcompression = DefaultCompressionMethod; + else + attrtypes[attnum]->attcompression = InvalidCompressionMethod; if (nullness == BOOTCOL_NULL_FORCE_NOT_NULL) { diff --git a/src/backend/catalog/genbki.pl b/src/backend/catalog/genbki.pl index b159958112..9586c29ad0 100644 --- a/src/backend/catalog/genbki.pl +++ b/src/backend/catalog/genbki.pl @@ -906,6 +906,9 @@ sub morph_row_for_pgattr $row->{attcollation} = $type->{typcollation} ne '0' ? $C_COLLATION_OID : 0; + $row->{attcompression} = + $type->{typstorage} ne 'p' && $type->{typstorage} ne 'e' ? 'p' : '\0'; + if (defined $attr->{forcenotnull}) { $row->{attnotnull} = 't'; diff --git a/src/backend/catalog/heap.c b/src/backend/catalog/heap.c index 693a94107d..7033ba9bed 100644 --- a/src/backend/catalog/heap.c +++ b/src/backend/catalog/heap.c @@ -36,6 +36,7 @@ #include "access/sysattr.h" #include "access/table.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/xact.h" #include "access/xlog.h" @@ -789,6 +790,7 @@ InsertPgAttributeTuples(Relation pg_attribute_rel, slot[slotCount]->tts_values[Anum_pg_attribute_attislocal - 1] = BoolGetDatum(attrs->attislocal); slot[slotCount]->tts_values[Anum_pg_attribute_attinhcount - 1] = Int32GetDatum(attrs->attinhcount); slot[slotCount]->tts_values[Anum_pg_attribute_attcollation - 1] = ObjectIdGetDatum(attrs->attcollation); + slot[slotCount]->tts_values[Anum_pg_attribute_attcompression - 1] = CharGetDatum(attrs->attcompression); if (attoptions && attoptions[natts] != (Datum) 0) slot[slotCount]->tts_values[Anum_pg_attribute_attoptions - 1] = attoptions[natts]; else @@ -1716,6 +1718,8 @@ RemoveAttributeById(Oid relid, AttrNumber attnum) /* Unset this so no one tries to look up the generation expression */ attStruct->attgenerated = '\0'; + attStruct->attcompression = InvalidCompressionMethod; + /* * Change the column name to something that isn't likely to conflict */ diff --git a/src/backend/catalog/index.c b/src/backend/catalog/index.c index 4ef61b5efd..397d70d226 100644 --- a/src/backend/catalog/index.c +++ b/src/backend/catalog/index.c @@ -348,6 +348,7 @@ ConstructTupleDescriptor(Relation heapRelation, to->attbyval = from->attbyval; to->attstorage = from->attstorage; to->attalign = from->attalign; + to->attcompression = from->attcompression; } else { diff --git a/src/backend/catalog/toasting.c b/src/backend/catalog/toasting.c index d7b806020d..933a0734d1 100644 --- a/src/backend/catalog/toasting.c +++ b/src/backend/catalog/toasting.c @@ -15,6 +15,7 @@ #include "postgres.h" #include "access/heapam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/binary_upgrade.h" #include "catalog/catalog.h" @@ -220,6 +221,11 @@ create_toast_table(Relation rel, Oid toastOid, Oid toastIndexOid, TupleDescAttr(tupdesc, 1)->attstorage = TYPSTORAGE_PLAIN; TupleDescAttr(tupdesc, 2)->attstorage = TYPSTORAGE_PLAIN; + /* Toast field should not be compressed */ + TupleDescAttr(tupdesc, 0)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 1)->attcompression = InvalidCompressionMethod; + TupleDescAttr(tupdesc, 2)->attcompression = InvalidCompressionMethod; + /* * Toast tables for regular relations go in pg_toast; those for temp * relations go into the per-backend temp-toast-table namespace. diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index 8745c1e722..e20dd5f63c 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -24,6 +24,7 @@ #include "access/relscan.h" #include "access/sysattr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "access/xlog.h" #include "catalog/catalog.h" @@ -562,6 +563,7 @@ static void refuseDupeIndexAttach(Relation parentIdx, Relation partIdx, static List *GetParentedForeignKeyRefs(Relation partition); static void ATDetachCheckNoForeignKeyRefs(Relation partition); static void ATExecAlterCollationRefreshVersion(Relation rel, List *coll); +static char GetAttributeCompression(Form_pg_attribute att, char *compression); /* ---------------------------------------------------------------- @@ -855,6 +857,18 @@ DefineRelation(CreateStmt *stmt, char relkind, Oid ownerId, if (colDef->generated) attr->attgenerated = colDef->generated; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (relkind == RELKIND_RELATION || + relkind == RELKIND_PARTITIONED_TABLE || + relkind == RELKIND_MATVIEW) + attr->attcompression = + GetAttributeCompression(attr, colDef->compression); + else + attr->attcompression = InvalidCompressionMethod; } /* @@ -2399,6 +2413,22 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(attribute->attstorage)))); + /* Copy/check compression parameter */ + if (CompressionMethodIsValid(attribute->attcompression)) + { + const char *compression = GetCompressionMethodName( + attribute->attcompression); + + if (def->compression == NULL) + def->compression = pstrdup(compression); + else if (strcmp(def->compression, compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, compression))); + } + def->inhcount++; /* Merge of NOT NULL constraints = OR 'em together */ def->is_not_null |= attribute->attnotnull; @@ -2433,6 +2463,11 @@ MergeAttributes(List *schema, List *supers, char relpersistence, def->collOid = attribute->attcollation; def->constraints = NIL; def->location = -1; + if (CompressionMethodIsValid(attribute->attcompression)) + def->compression = pstrdup(GetCompressionMethodName( + attribute->attcompression)); + else + def->compression = NULL; inhSchema = lappend(inhSchema, def); newattmap->attnums[parent_attno - 1] = ++child_attno; } @@ -2678,6 +2713,19 @@ MergeAttributes(List *schema, List *supers, char relpersistence, storage_name(def->storage), storage_name(newdef->storage)))); + /* Copy compression parameter */ + if (def->compression == NULL) + def->compression = newdef->compression; + else if (newdef->compression != NULL) + { + if (strcmp(def->compression, newdef->compression) != 0) + ereport(ERROR, + (errcode(ERRCODE_DATATYPE_MISMATCH), + errmsg("column \"%s\" has a compression method conflict", + attributeName), + errdetail("%s versus %s", def->compression, newdef->compression))); + } + /* Mark the column as locally defined */ def->is_local = true; /* Merge of NOT NULL constraints = OR 'em together */ @@ -6380,6 +6428,18 @@ ATExecAddColumn(List **wqueue, AlteredTableInfo *tab, Relation rel, attribute.attislocal = colDef->is_local; attribute.attinhcount = colDef->inhcount; attribute.attcollation = collOid; + + /* + * lookup attribute's compression method and store it in the + * attr->attcompression. + */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + attribute.attcompression = GetAttributeCompression(&attribute, + colDef->compression); + else + attribute.attcompression = InvalidCompressionMethod; + /* attribute.attacl is handled by InsertPgAttributeTuples() */ ReleaseSysCache(typeTuple); @@ -11899,6 +11959,23 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, ReleaseSysCache(typeTuple); + /* Setup attribute compression */ + if (rel->rd_rel->relkind == RELKIND_RELATION || + rel->rd_rel->relkind == RELKIND_PARTITIONED_TABLE) + { + /* + * No compression for plain/external storage, otherwise, default + * compression method if it is not already set, refer comments atop + * attcompression parameter in pg_attribute.h. + */ + if (!IsStorageCompressible(tform->typstorage)) + attTup->attcompression = InvalidCompressionMethod; + else if (!CompressionMethodIsValid(attTup->attcompression)) + attTup->attcompression = DefaultCompressionMethod; + } + else + attTup->attcompression = InvalidCompressionMethod; + CatalogTupleUpdate(attrelation, &heapTup->t_self, heapTup); table_close(attrelation, RowExclusiveLock); @@ -17743,3 +17820,36 @@ ATExecAlterCollationRefreshVersion(Relation rel, List *coll) index_update_collation_versions(rel->rd_id, get_collation_oid(coll, false)); CacheInvalidateRelcache(rel); } + +/* + * resolve column compression specification to compression method. + */ +static char +GetAttributeCompression(Form_pg_attribute att, char *compression) +{ + char typstorage = get_typstorage(att->atttypid); + char cmethod; + + /* + * No compression for plain/external storage, refer comments atop + * attcompression parameter in pg_attribute.h + */ + if (!IsStorageCompressible(typstorage)) + { + if (compression == NULL) + return InvalidCompressionMethod; + + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(att->atttypid)))); + } + + /* fallback to default compression if it's not specified */ + if (compression == NULL) + return DefaultCompressionMethod; + + cmethod = CompressionNameToMethod(compression); + + return cmethod; +} diff --git a/src/backend/executor/nodeModifyTable.c b/src/backend/executor/nodeModifyTable.c index 2993ba43e3..631ba1f193 100644 --- a/src/backend/executor/nodeModifyTable.c +++ b/src/backend/executor/nodeModifyTable.c @@ -37,9 +37,11 @@ #include "postgres.h" +#include "access/detoast.h" #include "access/heapam.h" #include "access/htup_details.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/xact.h" #include "catalog/catalog.h" #include "commands/trigger.h" @@ -2036,6 +2038,119 @@ ExecPrepareTupleRouting(ModifyTableState *mtstate, return slot; } +/* + * Compare the compression method of each compressed value with the + * compression method of the target attribute. If the compression method + * of the compressed value is not same as the target attribute then decompress + * the value. If any of the value need to be decompressed then we need to + * store that into a new slot. + */ +TupleTableSlot * +CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc) +{ + int i; + int attnum; + int natts = slot->tts_tupleDescriptor->natts; + bool decompressed_any = false; + bool slot_tup_deformed = false; + TupleDesc tupleDesc = slot->tts_tupleDescriptor; + TupleTableSlot *newslot = *outslot; + + if (natts == 0) + return slot; + + /* + * Loop over all the attributes in the tuple and check if any attribute is + * compressed and its compression method is not same as the target + * atrribute's compression method then decompress it. + */ + for (i = 0; i < natts; i++) + { + attnum = tupleDesc->attrs[i].attnum; + + if (TupleDescAttr(tupleDesc, i)->attlen == -1) + { + struct varlena *new_value; + char cmethod; + + /* fetch the values of all the attribute, if not already done */ + if (!slot_tup_deformed) + { + slot_getallattrs(slot); + slot_tup_deformed = true; + } + + /* nothing to be done, if the value is null */ + if (slot->tts_isnull[attnum - 1]) + continue; + + new_value = (struct varlena *) + DatumGetPointer(slot->tts_values[attnum - 1]); + + /* + * Get the compression method stored in the toast header and + * compare it with the compression method of the target. + */ + cmethod = toast_get_compression_method(new_value); + if (IsValidCompression(cmethod) && + targetTupDesc->attrs[i].attcompression != cmethod) + { + if (!decompressed_any) + { + /* + * If the caller has passed an invalid slot then create a + * new slot. Otherwise, just clear the existing tuple from + * the slot. This slot should be stored by the caller so + * that it can be reused for decompressing the subsequent + * tuples. + */ + if (newslot == NULL) + newslot = MakeSingleTupleTableSlot(tupleDesc, + slot->tts_ops); + else + ExecClearTuple(newslot); + + /* copy the value/null array to the new slot */ + memcpy(newslot->tts_values, slot->tts_values, + natts * sizeof(Datum)); + memcpy(newslot->tts_isnull, slot->tts_isnull, + natts * sizeof(bool)); + + } + + /* detoast the value and store into the new slot */ + new_value = detoast_attr(new_value); + newslot->tts_values[attnum - 1] = PointerGetDatum(new_value); + decompressed_any = true; + } + } + } + + /* + * If we have decompressed any of the fields then return the new slot + * otherwise return the original slot. + */ + if (decompressed_any) + { + ExecStoreVirtualTuple(newslot); + + /* + * If the original slot was materialized then materialize the new slot + * as well. + */ + if (TTS_SHOULDFREE(slot)) + ExecMaterializeSlot(newslot); + + *outslot = newslot; + + return newslot; + } + else + return slot; +} + /* ---------------------------------------------------------------- * ExecModifyTable * @@ -2244,6 +2359,15 @@ ExecModifyTable(PlanState *pstate) slot = ExecFilterJunk(junkfilter, slot); } + /* + * Compare the compression method of the compressed attribute in the + * source tuple with target attribute and if those are different then + * decompress those attributes. + */ + slot = CompareCompressionMethodAndDecompress(slot, + &node->mt_decompress_tuple_slot, + resultRelInfo->ri_RelationDesc->rd_att); + switch (operation) { case CMD_INSERT: @@ -2876,6 +3000,10 @@ ExecEndModifyTable(ModifyTableState *node) ExecDropSingleTupleTableSlot(node->mt_root_tuple_slot); } + /* release the slot used for decompressing the tuple */ + if (node->mt_decompress_tuple_slot) + ExecDropSingleTupleTableSlot(node->mt_decompress_tuple_slot); + /* * Free the exprcontext */ diff --git a/src/backend/nodes/copyfuncs.c b/src/backend/nodes/copyfuncs.c index da91cbd2b1..0d3b923a38 100644 --- a/src/backend/nodes/copyfuncs.c +++ b/src/backend/nodes/copyfuncs.c @@ -2988,6 +2988,7 @@ _copyColumnDef(const ColumnDef *from) COPY_STRING_FIELD(colname); COPY_NODE_FIELD(typeName); + COPY_STRING_FIELD(compression); COPY_SCALAR_FIELD(inhcount); COPY_SCALAR_FIELD(is_local); COPY_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/equalfuncs.c b/src/backend/nodes/equalfuncs.c index c2d73626fc..f3592003da 100644 --- a/src/backend/nodes/equalfuncs.c +++ b/src/backend/nodes/equalfuncs.c @@ -2599,6 +2599,7 @@ _equalColumnDef(const ColumnDef *a, const ColumnDef *b) { COMPARE_STRING_FIELD(colname); COMPARE_NODE_FIELD(typeName); + COMPARE_STRING_FIELD(compression); COMPARE_SCALAR_FIELD(inhcount); COMPARE_SCALAR_FIELD(is_local); COMPARE_SCALAR_FIELD(is_not_null); diff --git a/src/backend/nodes/nodeFuncs.c b/src/backend/nodes/nodeFuncs.c index 49357ac5c2..38226530c6 100644 --- a/src/backend/nodes/nodeFuncs.c +++ b/src/backend/nodes/nodeFuncs.c @@ -3897,6 +3897,8 @@ raw_expression_tree_walker(Node *node, if (walker(coldef->typeName, context)) return true; + if (walker(coldef->compression, context)) + return true; if (walker(coldef->raw_default, context)) return true; if (walker(coldef->collClause, context)) diff --git a/src/backend/nodes/outfuncs.c b/src/backend/nodes/outfuncs.c index 6493a03ff8..75343b8aaf 100644 --- a/src/backend/nodes/outfuncs.c +++ b/src/backend/nodes/outfuncs.c @@ -2876,6 +2876,7 @@ _outColumnDef(StringInfo str, const ColumnDef *node) WRITE_STRING_FIELD(colname); WRITE_NODE_FIELD(typeName); + WRITE_STRING_FIELD(compression); WRITE_INT_FIELD(inhcount); WRITE_BOOL_FIELD(is_local); WRITE_BOOL_FIELD(is_not_null); diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index 2ae41f8f4d..f8bc11d28b 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -596,6 +596,8 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); %type hash_partbound %type hash_partbound_elem +%type optColumnCompression + /* * Non-keyword token types. These are hard-wired into the "flex" lexer. * They must be listed first so that their numeric codes do not depend on @@ -631,9 +633,9 @@ static Node *makeRecursiveViewSelect(char *relname, List *aliases, Node *query); CACHE CALL CALLED CASCADE CASCADED CASE CAST CATALOG_P CHAIN CHAR_P CHARACTER CHARACTERISTICS CHECK CHECKPOINT CLASS CLOSE CLUSTER COALESCE COLLATE COLLATION COLUMN COLUMNS COMMENT COMMENTS COMMIT - COMMITTED CONCURRENTLY CONFIGURATION CONFLICT CONNECTION CONSTRAINT - CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY COST CREATE - CROSS CSV CUBE CURRENT_P + COMMITTED COMPRESSION CONCURRENTLY CONFIGURATION CONFLICT + CONNECTION CONSTRAINT CONSTRAINTS CONTENT_P CONTINUE_P CONVERSION_P COPY + COST CREATE CROSS CSV CUBE CURRENT_P CURRENT_CATALOG CURRENT_DATE CURRENT_ROLE CURRENT_SCHEMA CURRENT_TIME CURRENT_TIMESTAMP CURRENT_USER CURSOR CYCLE @@ -3429,11 +3431,12 @@ TypedTableElement: | TableConstraint { $$ = $1; } ; -columnDef: ColId Typename create_generic_options ColQualList +columnDef: ColId Typename optColumnCompression create_generic_options ColQualList { ColumnDef *n = makeNode(ColumnDef); n->colname = $1; n->typeName = $2; + n->compression = $3; n->inhcount = 0; n->is_local = true; n->is_not_null = false; @@ -3442,8 +3445,8 @@ columnDef: ColId Typename create_generic_options ColQualList n->raw_default = NULL; n->cooked_default = NULL; n->collOid = InvalidOid; - n->fdwoptions = $3; - SplitColQualList($4, &n->constraints, &n->collClause, + n->fdwoptions = $4; + SplitColQualList($5, &n->constraints, &n->collClause, yyscanner); n->location = @1; $$ = (Node *)n; @@ -3488,6 +3491,14 @@ columnOptions: ColId ColQualList } ; +optColumnCompression: + COMPRESSION name + { + $$ = $2; + } + | /*EMPTY*/ { $$ = NULL; } + ; + ColQualList: ColQualList ColConstraint { $$ = lappend($1, $2); } | /*EMPTY*/ { $$ = NIL; } @@ -3718,6 +3729,7 @@ TableLikeOption: | INDEXES { $$ = CREATE_TABLE_LIKE_INDEXES; } | STATISTICS { $$ = CREATE_TABLE_LIKE_STATISTICS; } | STORAGE { $$ = CREATE_TABLE_LIKE_STORAGE; } + | COMPRESSION { $$ = CREATE_TABLE_LIKE_COMPRESSION; } | ALL { $$ = CREATE_TABLE_LIKE_ALL; } ; @@ -15304,6 +15316,7 @@ unreserved_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONFIGURATION | CONFLICT | CONNECTION @@ -15824,6 +15837,7 @@ bare_label_keyword: | COMMENTS | COMMIT | COMMITTED + | COMPRESSION | CONCURRENTLY | CONFIGURATION | CONFLICT diff --git a/src/backend/parser/parse_utilcmd.c b/src/backend/parser/parse_utilcmd.c index 75266caeb4..681151e32c 100644 --- a/src/backend/parser/parse_utilcmd.c +++ b/src/backend/parser/parse_utilcmd.c @@ -31,6 +31,7 @@ #include "access/relation.h" #include "access/reloptions.h" #include "access/table.h" +#include "access/toast_compression.h" #include "catalog/dependency.h" #include "catalog/heap.h" #include "catalog/index.h" @@ -1082,6 +1083,14 @@ transformTableLikeClause(CreateStmtContext *cxt, TableLikeClause *table_like_cla else def->storage = 0; + /* Likewise, copy compression if requested */ + if ((table_like_clause->options & CREATE_TABLE_LIKE_COMPRESSION) != 0 + && CompressionMethodIsValid(attribute->attcompression)) + def->compression = + pstrdup(GetCompressionMethodName(attribute->attcompression)); + else + def->compression = NULL; + /* Likewise, copy comment if requested */ if ((table_like_clause->options & CREATE_TABLE_LIKE_COMMENTS) && (comment = GetComment(attribute->attrelid, diff --git a/src/backend/utils/adt/varlena.c b/src/backend/utils/adt/varlena.c index 479ed9ae54..187e55bd4e 100644 --- a/src/backend/utils/adt/varlena.c +++ b/src/backend/utils/adt/varlena.c @@ -18,6 +18,7 @@ #include #include "access/detoast.h" +#include "access/toast_compression.h" #include "catalog/pg_collation.h" #include "catalog/pg_type.h" #include "common/hashfn.h" @@ -5299,6 +5300,46 @@ pg_column_size(PG_FUNCTION_ARGS) PG_RETURN_INT32(result); } +/* + * Return the compression method stored in the compressed attribute. Return + * NULL for non varlena type or uncompressed data. + */ +Datum +pg_column_compression(PG_FUNCTION_ARGS) +{ + Datum value = PG_GETARG_DATUM(0); + int typlen; + char method; + + /* On first call, get the input type's typlen, and save at *fn_extra */ + if (fcinfo->flinfo->fn_extra == NULL) + { + /* Lookup the datatype of the supplied argument */ + Oid argtypeid = get_fn_expr_argtype(fcinfo->flinfo, 0); + + typlen = get_typlen(argtypeid); + if (typlen == 0) /* should not happen */ + elog(ERROR, "cache lookup failed for type %u", argtypeid); + + fcinfo->flinfo->fn_extra = MemoryContextAlloc(fcinfo->flinfo->fn_mcxt, + sizeof(int)); + *((int *) fcinfo->flinfo->fn_extra) = typlen; + } + else + typlen = *((int *) fcinfo->flinfo->fn_extra); + + if (typlen != -1) + PG_RETURN_NULL(); + + method = toast_get_compression_method( + (struct varlena *) DatumGetPointer(value)); + + if (!CompressionMethodIsValid(method)) + PG_RETURN_NULL(); + else + PG_RETURN_TEXT_P(cstring_to_text(GetCompressionMethodName(method))); +} + /* * string_agg - Concatenates values and returns string. * diff --git a/src/bin/pg_dump/pg_backup.h b/src/bin/pg_dump/pg_backup.h index 99ace71a2a..f54f285f38 100644 --- a/src/bin/pg_dump/pg_backup.h +++ b/src/bin/pg_dump/pg_backup.h @@ -161,6 +161,7 @@ typedef struct _dumpOptions int no_subscriptions; int no_synchronized_snapshots; int no_unlogged_table_data; + int no_toast_compression; int serializable_deferrable; int disable_triggers; int outputNoTableam; diff --git a/src/bin/pg_dump/pg_dump.c b/src/bin/pg_dump/pg_dump.c index 3fd7f48605..be5d369efa 100644 --- a/src/bin/pg_dump/pg_dump.c +++ b/src/bin/pg_dump/pg_dump.c @@ -387,6 +387,7 @@ main(int argc, char **argv) {"no-synchronized-snapshots", no_argument, &dopt.no_synchronized_snapshots, 1}, {"no-unlogged-table-data", no_argument, &dopt.no_unlogged_table_data, 1}, {"no-subscriptions", no_argument, &dopt.no_subscriptions, 1}, + {"no-toast-compression", no_argument, &dopt.no_toast_compression, 1}, {"no-sync", no_argument, NULL, 7}, {"on-conflict-do-nothing", no_argument, &dopt.do_nothing, 1}, {"rows-per-insert", required_argument, NULL, 10}, @@ -1048,6 +1049,7 @@ help(const char *progname) printf(_(" --no-publications do not dump publications\n")); printf(_(" --no-security-labels do not dump security label assignments\n")); printf(_(" --no-subscriptions do not dump subscriptions\n")); + printf(_(" --no-toast-compression do not dump toast compression methods\n")); printf(_(" --no-synchronized-snapshots do not use synchronized snapshots in parallel jobs\n")); printf(_(" --no-tablespaces do not dump tablespace assignments\n")); printf(_(" --no-unlogged-table-data do not dump unlogged table data\n")); @@ -8626,6 +8628,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) { DumpOptions *dopt = fout->dopt; PQExpBuffer q = createPQExpBuffer(); + bool createWithCompression; for (int i = 0; i < numTables; i++) { @@ -8711,6 +8714,15 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) appendPQExpBufferStr(q, "'' AS attidentity,\n"); + createWithCompression = (fout->remoteVersion >= 140000); + + if (createWithCompression) + appendPQExpBuffer(q, + "a.attcompression AS attcompression,\n"); + else + appendPQExpBuffer(q, + "NULL AS attcmname,\n"); + if (fout->remoteVersion >= 110000) appendPQExpBufferStr(q, "CASE WHEN a.atthasmissing AND NOT a.attisdropped " @@ -8756,6 +8768,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation = (Oid *) pg_malloc(ntups * sizeof(Oid)); tbinfo->attfdwoptions = (char **) pg_malloc(ntups * sizeof(char *)); tbinfo->attmissingval = (char **) pg_malloc(ntups * sizeof(char *)); + tbinfo->attcompression = (char *) pg_malloc(ntups * sizeof(char *)); tbinfo->notnull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->inhNotNull = (bool *) pg_malloc(ntups * sizeof(bool)); tbinfo->attrdefs = (AttrDefInfo **) pg_malloc(ntups * sizeof(AttrDefInfo *)); @@ -8784,6 +8797,7 @@ getTableAttrs(Archive *fout, TableInfo *tblinfo, int numTables) tbinfo->attcollation[j] = atooid(PQgetvalue(res, j, PQfnumber(res, "attcollation"))); tbinfo->attfdwoptions[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attfdwoptions"))); tbinfo->attmissingval[j] = pg_strdup(PQgetvalue(res, j, PQfnumber(res, "attmissingval"))); + tbinfo->attcompression[j] = *(PQgetvalue(res, j, PQfnumber(res, "attcompression"))); tbinfo->attrdefs[j] = NULL; /* fix below */ if (PQgetvalue(res, j, PQfnumber(res, "atthasdef"))[0] == 't') hasdefaults = true; @@ -15900,6 +15914,31 @@ dumpTableSchema(Archive *fout, const TableInfo *tbinfo) tbinfo->atttypnames[j]); } + /* + * Attribute compression + */ + if (!dopt->no_toast_compression && + tbinfo->attcompression[j] && + tbinfo->attcompression[j] != '\0') + { + char *cmname; + + switch (tbinfo->attcompression[j]) + { + case 'p': + cmname = "pglz"; + break; + case 'l': + cmname = "lz4"; + break; + default: + cmname = NULL; + } + + if (cmname != NULL) + appendPQExpBuffer(q, " COMPRESSION %s", cmname); + } + if (print_default) { if (tbinfo->attgenerated[j] == ATTRIBUTE_GENERATED_STORED) diff --git a/src/bin/pg_dump/pg_dump.h b/src/bin/pg_dump/pg_dump.h index 0a2213fb06..d956b035f3 100644 --- a/src/bin/pg_dump/pg_dump.h +++ b/src/bin/pg_dump/pg_dump.h @@ -326,6 +326,7 @@ typedef struct _tableInfo char *partbound; /* partition bound definition */ bool needs_override; /* has GENERATED ALWAYS AS IDENTITY */ char *amname; /* relation access method */ + char *attcompression; /* per-attribute current compression method */ /* * Stuff computed only for dumpable tables. diff --git a/src/bin/pg_dump/t/002_pg_dump.pl b/src/bin/pg_dump/t/002_pg_dump.pl index 737e46464a..bc91bb12ac 100644 --- a/src/bin/pg_dump/t/002_pg_dump.pl +++ b/src/bin/pg_dump/t/002_pg_dump.pl @@ -2284,9 +2284,9 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text,\E\n - \s+\Qcol3 text,\E\n - \s+\Qcol4 text,\E\n + \s+\Qcol2 text COMPRESSION\E\D*,\n + \s+\Qcol3 text COMPRESSION\E\D*,\n + \s+\Qcol4 text COMPRESSION\E\D*,\n \s+\QCONSTRAINT test_table_col1_check CHECK ((col1 <= 1000))\E\n \Q)\E\n \QWITH (autovacuum_enabled='false', fillfactor='80');\E\n/xm, @@ -2326,7 +2326,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_second_table (\E \n\s+\Qcol1 integer,\E - \n\s+\Qcol2 text\E + \n\s+\Qcol2 text COMPRESSION\E\D* \n\); /xm, like => @@ -2441,7 +2441,7 @@ my %tests = ( \n\s+\Qcol1 integer,\E \n\s+\Qcol2 boolean,\E \n\s+\Qcol3 boolean,\E - \n\s+\Qcol4 bit(5),\E + \n\s+\Qcol4 bit(5) COMPRESSION\E\D*, \n\s+\Qcol5 double precision\E \n\); /xm, @@ -2459,7 +2459,7 @@ my %tests = ( regexp => qr/^ \QCREATE TABLE dump_test.test_table_identity (\E\n \s+\Qcol1 integer NOT NULL,\E\n - \s+\Qcol2 text\E\n + \s+\Qcol2 text COMPRESSION\E\D*\n \); .* \QALTER TABLE dump_test.test_table_identity ALTER COLUMN col1 ADD GENERATED ALWAYS AS IDENTITY (\E\n diff --git a/src/bin/psql/describe.c b/src/bin/psql/describe.c index 20af5a92b4..b284113d55 100644 --- a/src/bin/psql/describe.c +++ b/src/bin/psql/describe.c @@ -1459,7 +1459,7 @@ describeOneTableDetails(const char *schemaname, bool printTableInitialized = false; int i; char *view_def = NULL; - char *headers[11]; + char *headers[12]; PQExpBufferData title; PQExpBufferData tmpbuf; int cols; @@ -1475,7 +1475,8 @@ describeOneTableDetails(const char *schemaname, fdwopts_col = -1, attstorage_col = -1, attstattarget_col = -1, - attdescr_col = -1; + attdescr_col = -1, + attcompression_col = -1; int numrows; struct { @@ -1892,6 +1893,17 @@ describeOneTableDetails(const char *schemaname, appendPQExpBufferStr(&buf, ",\n a.attstorage"); attstorage_col = cols++; + /* compresssion info */ + if (pset.sversion >= 140000 && + !pset.hide_compression && + (tableinfo.relkind == RELKIND_RELATION || + tableinfo.relkind == RELKIND_PARTITIONED_TABLE || + tableinfo.relkind == RELKIND_MATVIEW)) + { + appendPQExpBufferStr(&buf, ",\n a.attcompression AS attcompression"); + attcompression_col = cols++; + } + /* stats target, if relevant to relkind */ if (tableinfo.relkind == RELKIND_RELATION || tableinfo.relkind == RELKIND_INDEX || @@ -2018,6 +2030,8 @@ describeOneTableDetails(const char *schemaname, headers[cols++] = gettext_noop("FDW options"); if (attstorage_col >= 0) headers[cols++] = gettext_noop("Storage"); + if (attcompression_col >= 0) + headers[cols++] = gettext_noop("Compression"); if (attstattarget_col >= 0) headers[cols++] = gettext_noop("Stats target"); if (attdescr_col >= 0) @@ -2097,6 +2111,19 @@ describeOneTableDetails(const char *schemaname, false, false); } + /* Column compression. */ + if (attcompression_col >= 0) + { + char *compression = PQgetvalue(res, i, attcompression_col); + + /* these strings are literal in our syntax, so not translated. */ + printTableAddCell(&cont, (compression[0] == 'p' ? "pglz" : + (compression[0] == 'l' ? "lz4" : + (compression[0] == '\0' ? "" : + "???"))), + false, false); + } + /* Statistics target, if the relkind supports this feature */ if (attstattarget_col >= 0) printTableAddCell(&cont, PQgetvalue(res, i, attstattarget_col), diff --git a/src/bin/psql/help.c b/src/bin/psql/help.c index daa5081eac..99a59470c5 100644 --- a/src/bin/psql/help.c +++ b/src/bin/psql/help.c @@ -372,6 +372,8 @@ helpVariables(unsigned short int pager) " true if last query failed, else false\n")); fprintf(output, _(" FETCH_COUNT\n" " the number of result rows to fetch and display at a time (0 = unlimited)\n")); + fprintf(output, _(" HIDE_TOAST_COMPRESSION\n" + " if set, compression methods are not displayed\n")); fprintf(output, _(" HIDE_TABLEAM\n" " if set, table access methods are not displayed\n")); fprintf(output, _(" HISTCONTROL\n" diff --git a/src/bin/psql/settings.h b/src/bin/psql/settings.h index d65990059d..83f2e6f254 100644 --- a/src/bin/psql/settings.h +++ b/src/bin/psql/settings.h @@ -134,6 +134,7 @@ typedef struct _psqlSettings bool quiet; bool singleline; bool singlestep; + bool hide_compression; bool hide_tableam; int fetch_count; int histsize; diff --git a/src/bin/psql/startup.c b/src/bin/psql/startup.c index 780479c8d7..110906a4e9 100644 --- a/src/bin/psql/startup.c +++ b/src/bin/psql/startup.c @@ -1159,6 +1159,13 @@ show_context_hook(const char *newval) return true; } +static bool +hide_compression_hook(const char *newval) +{ + return ParseVariableBool(newval, "HIDE_TOAST_COMPRESSION", + &pset.hide_compression); +} + static bool hide_tableam_hook(const char *newval) { @@ -1227,6 +1234,9 @@ EstablishVariableSpace(void) SetVariableHooks(pset.vars, "SHOW_CONTEXT", show_context_substitute_hook, show_context_hook); + SetVariableHooks(pset.vars, "HIDE_TOAST_COMPRESSION", + bool_substitute_hook, + hide_compression_hook); SetVariableHooks(pset.vars, "HIDE_TABLEAM", bool_substitute_hook, hide_tableam_hook); diff --git a/src/include/access/detoast.h b/src/include/access/detoast.h index 0adf53c77b..9a428aae2d 100644 --- a/src/include/access/detoast.h +++ b/src/include/access/detoast.h @@ -89,4 +89,12 @@ extern Size toast_raw_datum_size(Datum value); */ extern Size toast_datum_size(Datum value); +/* ---------- + * toast_get_compression_method - + * + * Return the compression method from the compressed value + * ---------- + */ +extern char toast_get_compression_method(struct varlena *attr); + #endif /* DETOAST_H */ diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h new file mode 100644 index 0000000000..38800cb97a --- /dev/null +++ b/src/include/access/toast_compression.h @@ -0,0 +1,119 @@ +/*------------------------------------------------------------------------- + * + * toast_compression.h + * Functions for toast compression. + * + * Copyright (c) 2021, PostgreSQL Global Development Group + * + * src/include/access/toast_compression.h + * + *------------------------------------------------------------------------- + */ + +#ifndef TOAST_COMPRESSION_H +#define TOAST_COMPRESSION_H + +#include "postgres.h" + +/* + * Built-in compression methods. pg_attribute will store this in the + * attcompression column. + */ +#define PGLZ_COMPRESSION 'p' +#define LZ4_COMPRESSION 'l' + +#define InvalidCompressionMethod '\0' +#define CompressionMethodIsValid(cm) ((bool) ((cm) != InvalidCompressionMethod)) + +/* + * Built-in compression method-id. The toast compression header will store + * this in the first 2 bits of the raw length. These built-in compression + * method-id are directly mapped to the built-in compression methods. + */ +typedef enum CompressionId +{ + PGLZ_COMPRESSION_ID = 0, + LZ4_COMPRESSION_ID = 1 +} CompressionId; + +/* use default compression method if it is not specified. */ +#define DefaultCompressionMethod PGLZ_COMPRESSION +#define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) + +#define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ + (storage) != TYPSTORAGE_EXTERNAL) + +/* compression handler routines */ +typedef struct varlena *(*cmcompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_function) (const struct varlena *value); +typedef struct varlena *(*cmdecompress_slice_function) + (const struct varlena *value, int32 slicelength); + +/* + * API struct for a compression routines. + * + * 'cmname' - name of the compression method + * 'datum_compress' - varlena compression function. + * 'datum_decompress' - varlena decompression function. + * 'datum_decompress_slice' - varlena slice decompression functions. + */ +typedef struct CompressionRoutine +{ + char cmname[64]; + cmcompress_function datum_compress; + cmdecompress_function datum_decompress; + cmdecompress_slice_function datum_decompress_slice; +} CompressionRoutine; + +extern char CompressionNameToMethod(char *compression); +extern const CompressionRoutine *GetCompressionRoutines(char method); + +/* + * CompressionMethodToId - Convert compression method to compression id. + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline CompressionId +CompressionMethodToId(char method) +{ + switch (method) + { + case PGLZ_COMPRESSION: + return PGLZ_COMPRESSION_ID; + case LZ4_COMPRESSION: + return LZ4_COMPRESSION_ID; + default: + elog(ERROR, "invalid compression method %c", method); + } +} + +/* + * CompressionIdToMethod - Convert compression id to compression method + * + * For more details refer comment atop CompressionId in toast_compression.h + */ +static inline Oid +CompressionIdToMethod(CompressionId cmid) +{ + switch (cmid) + { + case PGLZ_COMPRESSION_ID: + return PGLZ_COMPRESSION; + case LZ4_COMPRESSION_ID: + return LZ4_COMPRESSION; + default: + elog(ERROR, "invalid compression method id %d", cmid); + } +} + +/* + * GetCompressionMethodName - Get compression method name + */ +static inline const char* +GetCompressionMethodName(char method) +{ + return GetCompressionRoutines(method)->cmname; +} + + +#endif /* TOAST_COMPRESSION_H */ diff --git a/src/include/access/toast_helper.h b/src/include/access/toast_helper.h index a9a6d644bc..05104ce237 100644 --- a/src/include/access/toast_helper.h +++ b/src/include/access/toast_helper.h @@ -32,6 +32,7 @@ typedef struct struct varlena *tai_oldexternal; int32 tai_size; uint8 tai_colflags; + char tai_compression; } ToastAttrInfo; /* diff --git a/src/include/access/toast_internals.h b/src/include/access/toast_internals.h index cedfb890d8..284a15761a 100644 --- a/src/include/access/toast_internals.h +++ b/src/include/access/toast_internals.h @@ -12,6 +12,7 @@ #ifndef TOAST_INTERNALS_H #define TOAST_INTERNALS_H +#include "access/toast_compression.h" #include "storage/lockdefs.h" #include "utils/relcache.h" #include "utils/snapshot.h" @@ -22,22 +23,23 @@ typedef struct toast_compress_header { int32 vl_len_; /* varlena header (do not touch directly!) */ - int32 rawsize; + uint32 tcinfo; /* 2 bits for compression method and 30 bits + * rawsize */ } toast_compress_header; /* * Utilities for manipulation of header information for compressed * toast entries. */ -#define TOAST_COMPRESS_HDRSZ ((int32) sizeof(toast_compress_header)) -#define TOAST_COMPRESS_RAWSIZE(ptr) (((toast_compress_header *) (ptr))->rawsize) -#define TOAST_COMPRESS_SIZE(ptr) ((int32) VARSIZE_ANY(ptr) - TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_RAWDATA(ptr) \ - (((char *) (ptr)) + TOAST_COMPRESS_HDRSZ) -#define TOAST_COMPRESS_SET_RAWSIZE(ptr, len) \ - (((toast_compress_header *) (ptr))->rawsize = (len)) +#define TOAST_COMPRESS_METHOD(ptr) (((toast_compress_header *) (ptr))->tcinfo >> VARLENA_RAWSIZE_BITS) +#define TOAST_COMPRESS_SET_SIZE_AND_METHOD(ptr, len, cm_method) \ + do { \ + Assert((len) > 0 && (len) <= VARLENA_RAWSIZE_MASK); \ + Assert((cm_method) >= PGLZ_COMPRESSION_ID && (cm_method) <= LZ4_COMPRESSION_ID); \ + ((toast_compress_header *) (ptr))->tcinfo = ((len) | (cm_method) << VARLENA_RAWSIZE_BITS); \ + } while (0) -extern Datum toast_compress_datum(Datum value); +extern Datum toast_compress_datum(Datum value, char cmethod); extern Oid toast_get_valid_index(Oid toastoid, LOCKMODE lock); extern void toast_delete_datum(Relation rel, Datum value, bool is_speculative); diff --git a/src/include/catalog/pg_attribute.h b/src/include/catalog/pg_attribute.h index 3db42abf08..560f8f00bb 100644 --- a/src/include/catalog/pg_attribute.h +++ b/src/include/catalog/pg_attribute.h @@ -160,6 +160,12 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, /* attribute's collation, if any */ Oid attcollation BKI_LOOKUP_OPT(pg_collation); + /* + * compression method. Must be InvalidCompressionMethod if and only if + * typstorage is 'plain' or 'external'. + */ + char attcompression BKI_DEFAULT('\0'); + #ifdef CATALOG_VARLEN /* variable-length fields start here */ /* NOTE: The following fields are not present in tuple descriptors. */ @@ -187,7 +193,7 @@ CATALOG(pg_attribute,1249,AttributeRelationId) BKI_BOOTSTRAP BKI_ROWTYPE_OID(75, * can access fields beyond attcollation except in a real tuple! */ #define ATTRIBUTE_FIXED_PART_SIZE \ - (offsetof(FormData_pg_attribute,attcollation) + sizeof(Oid)) + (offsetof(FormData_pg_attribute,attcompression) + sizeof(char)) /* ---------------- * Form_pg_attribute corresponds to a pointer to a tuple with diff --git a/src/include/catalog/pg_proc.dat b/src/include/catalog/pg_proc.dat index c7619f8cd3..91c1c5b720 100644 --- a/src/include/catalog/pg_proc.dat +++ b/src/include/catalog/pg_proc.dat @@ -7099,6 +7099,10 @@ descr => 'bytes required to store the value, perhaps with compression', proname => 'pg_column_size', provolatile => 's', prorettype => 'int4', proargtypes => 'any', prosrc => 'pg_column_size' }, +{ oid => '2121', + descr => 'compression method for the compressed datum', + proname => 'pg_column_compression', provolatile => 's', prorettype => 'text', + proargtypes => 'any', prosrc => 'pg_column_compression' }, { oid => '2322', descr => 'total disk space usage for the specified tablespace', proname => 'pg_tablespace_size', provolatile => 'v', prorettype => 'int8', diff --git a/src/include/executor/executor.h b/src/include/executor/executor.h index 071e363d54..6495162a33 100644 --- a/src/include/executor/executor.h +++ b/src/include/executor/executor.h @@ -621,5 +621,7 @@ extern void CheckCmdReplicaIdentity(Relation rel, CmdType cmd); extern void CheckSubscriptionRelkind(char relkind, const char *nspname, const char *relname); - +extern TupleTableSlot *CompareCompressionMethodAndDecompress(TupleTableSlot *slot, + TupleTableSlot **outslot, + TupleDesc targetTupDesc); #endif /* EXECUTOR_H */ diff --git a/src/include/nodes/execnodes.h b/src/include/nodes/execnodes.h index e31ad6204e..e388336e02 100644 --- a/src/include/nodes/execnodes.h +++ b/src/include/nodes/execnodes.h @@ -1185,6 +1185,12 @@ typedef struct ModifyTableState */ TupleTableSlot *mt_root_tuple_slot; + /* + * Slot for storing the modified tuple, in case the target attribute's + * compression method doesn't match that of the source table. + */ + TupleTableSlot *mt_decompress_tuple_slot; + /* Tuple-routing support info */ struct PartitionTupleRouting *mt_partition_tuple_routing; diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index ce54f76610..56e24529f1 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -646,6 +646,7 @@ typedef struct ColumnDef NodeTag type; char *colname; /* name of column */ TypeName *typeName; /* type of column */ + char *compression; /* compression method for column */ int inhcount; /* number of times column is inherited */ bool is_local; /* column has local (non-inherited) def'n */ bool is_not_null; /* NOT NULL constraint specified? */ @@ -685,6 +686,7 @@ typedef enum TableLikeOption CREATE_TABLE_LIKE_INDEXES = 1 << 5, CREATE_TABLE_LIKE_STATISTICS = 1 << 6, CREATE_TABLE_LIKE_STORAGE = 1 << 7, + CREATE_TABLE_LIKE_COMPRESSION = 1 << 8, CREATE_TABLE_LIKE_ALL = PG_INT32_MAX } TableLikeOption; diff --git a/src/include/parser/kwlist.h b/src/include/parser/kwlist.h index 28083aaac9..ca1f950cbe 100644 --- a/src/include/parser/kwlist.h +++ b/src/include/parser/kwlist.h @@ -88,6 +88,7 @@ PG_KEYWORD("comment", COMMENT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("comments", COMMENTS, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("commit", COMMIT, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("committed", COMMITTED, UNRESERVED_KEYWORD, BARE_LABEL) +PG_KEYWORD("compression", COMPRESSION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("concurrently", CONCURRENTLY, TYPE_FUNC_NAME_KEYWORD, BARE_LABEL) PG_KEYWORD("configuration", CONFIGURATION, UNRESERVED_KEYWORD, BARE_LABEL) PG_KEYWORD("conflict", CONFLICT, UNRESERVED_KEYWORD, BARE_LABEL) diff --git a/src/include/pg_config.h.in b/src/include/pg_config.h.in index 7a7cc21d8d..6007d72a73 100644 --- a/src/include/pg_config.h.in +++ b/src/include/pg_config.h.in @@ -346,6 +346,9 @@ /* Define to 1 if you have the `z' library (-lz). */ #undef HAVE_LIBZ +/* Define to 1 if you have the `lz4' library (-llz4). */ +#undef HAVE_LIBLZ4 + /* Define to 1 if you have the `link' function. */ #undef HAVE_LINK diff --git a/src/include/postgres.h b/src/include/postgres.h index 2ed572004d..e98af05b30 100644 --- a/src/include/postgres.h +++ b/src/include/postgres.h @@ -145,7 +145,8 @@ typedef union struct /* Compressed-in-line format */ { uint32 va_header; - uint32 va_rawsize; /* Original data size (excludes header) */ + uint32 va_tcinfo; /* Original data size (excludes header) and + * compression method */ char va_data[FLEXIBLE_ARRAY_MEMBER]; /* Compressed data */ } va_compressed; } varattrib_4b; @@ -274,14 +275,23 @@ typedef struct (VARSIZE(PTR) - VARHDRSZ + VARHDRSZ_SHORT) #define VARHDRSZ_EXTERNAL offsetof(varattrib_1b_e, va_data) +#define VARHDRSZ_COMPRESS offsetof(varattrib_4b, va_compressed.va_data) #define VARDATA_4B(PTR) (((varattrib_4b *) (PTR))->va_4byte.va_data) #define VARDATA_4B_C(PTR) (((varattrib_4b *) (PTR))->va_compressed.va_data) #define VARDATA_1B(PTR) (((varattrib_1b *) (PTR))->va_data) #define VARDATA_1B_E(PTR) (((varattrib_1b_e *) (PTR))->va_data) +#define VARLENA_RAWSIZE_BITS 30 +#define VARLENA_RAWSIZE_MASK ((1U << VARLENA_RAWSIZE_BITS) - 1) + +/* + * va_tcinfo in va_compress contains raw size of datum and compression method. + */ #define VARRAWSIZE_4B_C(PTR) \ - (((varattrib_4b *) (PTR))->va_compressed.va_rawsize) + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo & VARLENA_RAWSIZE_MASK) +#define VARCOMPRESS_4B_C(PTR) \ + (((varattrib_4b *) (PTR))->va_compressed.va_tcinfo >> VARLENA_RAWSIZE_BITS) /* Externally visible macros */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out new file mode 100644 index 0000000000..6981d580dc --- /dev/null +++ b/src/test/regress/expected/compression.out @@ -0,0 +1,267 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + Table "public.cmdata1" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 +(1 row) + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + substr +---------------------------------------------------- + 01234567890123456789012345678901234567890123456789 +(1 row) + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +DROP TABLE cmdata3; +DROP TABLE cmdata2; +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + +DROP TABLE cmdata2; +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | pglz | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +SELECT pg_column_compression(f1) FROM cmdata1; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +SELECT pg_column_compression(x) FROM mv; + pg_column_compression +----------------------- + lz4 + lz4 +(2 rows) + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz +(2 rows) + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +NOTICE: merging multiple inherited definitions of column "f1" +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; + length +-------- + 10040 + 12449 +(2 rows) + +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10040 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 + 10040 +(2 rows) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out new file mode 100644 index 0000000000..893c18c7fa --- /dev/null +++ b/src/test/regress/expected/compression_1.out @@ -0,0 +1,269 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); + ^ +\d+ cmdata1 +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); +ERROR: column data type integer does not support compression +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz +(1 row) + +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; + substr +-------- + 01234 +(1 row) + +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + ^ +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + pg_column_compression +----------------------- + pglz +(1 row) + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove2; + pg_column_compression +----------------------- + pglz +(1 row) + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmmove3 SELECT * FROM cmdata1; + ^ +SELECT pg_column_compression(f1) FROM cmmove3; + pg_column_compression +----------------------- + pglz +(1 row) + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +ERROR: type "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (x cmdata1 compression pglz); + ^ +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +ERROR: relation "cmdata2" does not exist +LINE 1: INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +ERROR: type "cmdata2" does not exist +LINE 1: CREATE TABLE cmdata3 (x cmdata2 compression pglz); + ^ +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +ERROR: relation "cmdata3" does not exist +LINE 1: INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata... + ^ +SELECT pg_column_compression((x).f1) FROM cmdata2; +ERROR: relation "cmdata2" does not exist +LINE 1: SELECT pg_column_compression((x).f1) FROM cmdata2; + ^ +DROP TABLE cmdata3; +ERROR: table "cmdata3" does not exist +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + pglz +(1 row) + +INSERT INTO cmdata1 SELECT * FROM cmdata2; +ERROR: relation "cmdata1" does not exist +LINE 1: INSERT INTO cmdata1 SELECT * FROM cmdata2; + ^ +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +DROP TABLE cmdata2; +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); + ^ +\d+ cmdata2 +DROP TABLE cmdata2; +ERROR: table "cmdata2" does not exist +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+---------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | integer | | | | plain | | | + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | character varying | | | | extended | pglz | | + +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+-------------------+-----------+----------+---------+---------+-------------+--------------+------------- + f1 | character varying | | | | plain | pglz | | + +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + pg_column_compression +----------------------- + +(1 row) + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; + ^ +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmdata1; + ^ +SELECT pg_column_compression(x) FROM mv; +ERROR: relation "mv" does not exist +LINE 1: SELECT pg_column_compression(x) FROM mv; + ^ +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +ERROR: relation "cmpart" does not exist +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +ERROR: relation "cmpart" does not exist +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +ERROR: relation "cmdata1" does not exist +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +NOTICE: merging column "f1" with inherited definition +ERROR: column "f1" has a compression method conflict +DETAIL: pglz versus lz4 +-- check data is ok +SELECT length(f1) FROM cmdata; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmdata1; +ERROR: relation "cmdata1" does not exist +LINE 1: SELECT length(f1) FROM cmdata1; + ^ +SELECT length(f1) FROM cmmove1; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove2; + length +-------- + 10000 +(1 row) + +SELECT length(f1) FROM cmmove3; + length +-------- + 10000 +(1 row) + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/test/regress/parallel_schedule b/src/test/regress/parallel_schedule index e280198b17..70c38309d7 100644 --- a/src/test/regress/parallel_schedule +++ b/src/test/regress/parallel_schedule @@ -115,7 +115,7 @@ test: plancache limit plpgsql copy2 temp domain rangefuncs prepare conversion tr # ---------- # Another group of parallel tests # ---------- -test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain +test: partition_join partition_prune reloptions hash_part indexing partition_aggregate partition_info tuplesort explain compression # event triggers cannot run concurrently with any test that runs DDL # oidjoins is read-only, though, and should run late for best coverage diff --git a/src/test/regress/pg_regress_main.c b/src/test/regress/pg_regress_main.c index 8dc4941c24..1524676f3b 100644 --- a/src/test/regress/pg_regress_main.c +++ b/src/test/regress/pg_regress_main.c @@ -78,11 +78,11 @@ psql_start_test(const char *testname, * against different AMs without unnecessary differences. */ offset += snprintf(psql_cmd + offset, sizeof(psql_cmd) - offset, - "\"%s%spsql\" -X -a -q -d \"%s\" -v %s < \"%s\" > \"%s\" 2>&1", + "\"%s%spsql\" -X -a -q -d \"%s\" %s < \"%s\" > \"%s\" 2>&1", bindir ? bindir : "", bindir ? "/" : "", dblist->str, - "HIDE_TABLEAM=\"on\"", + "-v HIDE_TABLEAM=on -v HIDE_TOAST_COMPRESSION=on", infile, outfile); if (offset >= sizeof(psql_cmd)) diff --git a/src/test/regress/serial_schedule b/src/test/regress/serial_schedule index 6a57e889a1..d81d04136c 100644 --- a/src/test/regress/serial_schedule +++ b/src/test/regress/serial_schedule @@ -201,6 +201,7 @@ test: partition_aggregate test: partition_info test: tuplesort test: explain +test: compression test: event_trigger test: oidjoins test: fast_default diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql new file mode 100644 index 0000000000..b085c9fbd9 --- /dev/null +++ b/src/test/regress/sql/compression.sql @@ -0,0 +1,113 @@ +\set HIDE_TOAST_COMPRESSION false +-- test creating table with compression method +CREATE TABLE cmdata(f1 text COMPRESSION pglz); +CREATE INDEX idx ON cmdata(f1); +INSERT INTO cmdata VALUES(repeat('1234567890',1000)); +\d+ cmdata + +CREATE TABLE cmdata1(f1 TEXT COMPRESSION lz4); +INSERT INTO cmdata1 VALUES(repeat('1234567890',1004)); +\d+ cmdata1 + +-- try setting compression for incompressible data type +CREATE TABLE cmdata2 (f1 int COMPRESSION pglz); + +-- verify stored compression method +SELECT pg_column_compression(f1) FROM cmdata; +SELECT pg_column_compression(f1) FROM cmdata1; + +-- decompress data slice +SELECT SUBSTR(f1, 200, 5) FROM cmdata; +SELECT SUBSTR(f1, 2000, 50) FROM cmdata1; + +-- copy with table creation +SELECT * INTO cmmove1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove1; + +-- update using datum from different table +CREATE TABLE cmmove2(f1 text COMPRESSION pglz); +INSERT INTO cmmove2 VALUES (repeat('1234567890',1004)); +SELECT pg_column_compression(f1) FROM cmmove2; + +UPDATE cmmove2 SET f1 = cmdata.f1 FROM cmdata; +SELECT pg_column_compression(f1) FROM cmmove2; +UPDATE cmmove2 SET f1 = cmdata1.f1 FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove2; + +-- copy to existing table +CREATE TABLE cmmove3(f1 text COMPRESSION pglz); +INSERT INTO cmmove3 SELECT * FROM cmdata; +INSERT INTO cmmove3 SELECT * FROM cmdata1; +SELECT pg_column_compression(f1) FROM cmmove3; + +--test composite type +CREATE TABLE cmdata2 (x cmdata1 compression pglz); +INSERT INTO cmdata2 SELECT cmdata1 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; + +CREATE TABLE cmdata3 (x cmdata2 compression pglz); +INSERT INTO cmdata3 SELECT row(cmdata1)::cmdata2 FROM cmdata1; +SELECT pg_column_compression((x).f1) FROM cmdata2; +DROP TABLE cmdata3; +DROP TABLE cmdata2; + +-- test external compressed data +CREATE OR REPLACE FUNCTION large_val() RETURNS TEXT LANGUAGE SQL AS +'select array_agg(md5(g::text))::text from generate_series(1, 256) g'; +CREATE TABLE cmdata2 (f1 text COMPRESSION pglz); +INSERT INTO cmdata2 SELECT large_val() || repeat('a', 4000); +SELECT pg_column_compression(f1) FROM cmdata2; +INSERT INTO cmdata1 SELECT * FROM cmdata2; +SELECT pg_column_compression(f1) FROM cmdata1; +DROP TABLE cmdata2; + +-- test LIKE INCLUDING COMPRESSION +CREATE TABLE cmdata2 (LIKE cmdata1 INCLUDING COMPRESSION); +\d+ cmdata2 +DROP TABLE cmdata2; + +--test column type update varlena/non-varlena +CREATE TABLE cmdata2 (f1 int); +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE int USING f1::integer; +\d+ cmdata2 + +--changing column storage should not impact the compression method +--but the data should not be compressed +ALTER TABLE cmdata2 ALTER COLUMN f1 TYPE varchar; +\d+ cmdata2 +ALTER TABLE cmdata2 ALTER COLUMN f1 SET STORAGE plain; +\d+ cmdata2 +INSERT INTO cmdata2 VALUES (repeat('123456789', 800)); +SELECT pg_column_compression(f1) FROM cmdata2; + +-- test compression with materialized view +CREATE MATERIALIZED VIEW mv(x) AS SELECT * FROM cmdata1; +\d+ mv +SELECT pg_column_compression(f1) FROM cmdata1; +SELECT pg_column_compression(x) FROM mv; + +-- test compression with partition +CREATE TABLE cmpart(f1 text COMPRESSION lz4) PARTITION BY HASH(f1); +CREATE TABLE cmpart1 PARTITION OF cmpart FOR VALUES WITH (MODULUS 2, REMAINDER 0); +CREATE TABLE cmpart2(f1 text COMPRESSION pglz); + +ALTER TABLE cmpart ATTACH PARTITION cmpart2 FOR VALUES WITH (MODULUS 2, REMAINDER 1); +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + +-- test compression with inheritence, error +CREATE TABLE cminh() INHERITS(cmdata, cmdata1); +CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); + +-- check data is ok +SELECT length(f1) FROM cmdata; +SELECT length(f1) FROM cmdata1; +SELECT length(f1) FROM cmmove1; +SELECT length(f1) FROM cmmove2; +SELECT length(f1) FROM cmmove3; + +\set HIDE_TOAST_COMPRESSION true diff --git a/src/tools/msvc/Solution.pm b/src/tools/msvc/Solution.pm index a4f5cc4bdb..5f39a92111 100644 --- a/src/tools/msvc/Solution.pm +++ b/src/tools/msvc/Solution.pm @@ -307,6 +307,7 @@ sub GenerateFiles HAVE_LIBXML2 => undef, HAVE_LIBXSLT => undef, HAVE_LIBZ => $self->{options}->{zlib} ? 1 : undef, + HAVE_LIBLZ4 => undef, HAVE_LINK => undef, HAVE_LOCALE_T => 1, HAVE_LONG_INT_64 => undef, diff --git a/src/tools/pgindent/typedefs.list b/src/tools/pgindent/typedefs.list index 574a8a94fa..e6cb35a16f 100644 --- a/src/tools/pgindent/typedefs.list +++ b/src/tools/pgindent/typedefs.list @@ -395,6 +395,7 @@ CompositeIOData CompositeTypeStmt CompoundAffixFlag CompressionAlgorithm +CompressionRoutine CompressorState ComputeXidHorizonsResult ConditionVariable -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0005-f-2nd-attempt-to-use-pkgconfig-to-allow-compiling-on.patch" From ef577dfd2bcae3594d7239d4714f1dad8c8e9d79 Mon Sep 17 00:00:00 2001 From: Justin Pryzby Date: Thu, 11 Mar 2021 09:18:46 -0600 Subject: [PATCH 5/8] f!2nd attempt to use pkgconfig to allow compiling on OSX --- configure | 279 +++++++++++++++++++-------------------------------- configure.ac | 16 ++- 2 files changed, 108 insertions(+), 187 deletions(-) diff --git a/configure b/configure index 87ed16060e..387c553d5a 100755 --- a/configure +++ b/configure @@ -654,8 +654,6 @@ UUID_LIBS LDAP_LIBS_BE LDAP_LIBS_FE with_ssl -LZ4_LIBS -LZ4_CFLAGS PTHREAD_CFLAGS PTHREAD_LIBS PTHREAD_CC @@ -701,6 +699,8 @@ with_gnu_ld LD LDFLAGS_SL LDFLAGS_EX +LZ4_LIBS +LZ4_CFLAGS with_lz4 with_zlib with_system_tzdata @@ -895,12 +895,12 @@ ICU_LIBS XML2_CONFIG XML2_CFLAGS XML2_LIBS +LZ4_CFLAGS +LZ4_LIBS LDFLAGS_EX LDFLAGS_SL PERL PYTHON -LZ4_CFLAGS -LZ4_LIBS MSGFMT TCLSH' @@ -1603,12 +1603,12 @@ Some influential environment variables: XML2_CONFIG path to xml2-config utility XML2_CFLAGS C compiler flags for XML2, overriding pkg-config XML2_LIBS linker flags for XML2, overriding pkg-config + LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config + LZ4_LIBS linker flags for LZ4, overriding pkg-config LDFLAGS_EX extra linker flags for linking executables only LDFLAGS_SL extra linker flags for linking shared libraries only PERL Perl program PYTHON Python program - LZ4_CFLAGS C compiler flags for LZ4, overriding pkg-config - LZ4_LIBS linker flags for LZ4, overriding pkg-config MSGFMT msgfmt program for NLS TCLSH Tcl interpreter program (tclsh) @@ -8607,6 +8607,102 @@ fi $as_echo "$with_lz4" >&6; } +if test "$with_lz4" = yes; then + +pkg_failed=no +{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 +$as_echo_n "checking for liblz4... " >&6; } + +if test -n "$LZ4_CFLAGS"; then + pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi +if test -n "$LZ4_LIBS"; then + pkg_cv_LZ4_LIBS="$LZ4_LIBS" + elif test -n "$PKG_CONFIG"; then + if test -n "$PKG_CONFIG" && \ + { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 + ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 + ac_status=$? + $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 + test $ac_status = 0; }; then + pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` + test "x$?" != "x0" && pkg_failed=yes +else + pkg_failed=yes +fi + else + pkg_failed=untried +fi + + + +if test $pkg_failed = yes; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + +if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then + _pkg_short_errors_supported=yes +else + _pkg_short_errors_supported=no +fi + if test $_pkg_short_errors_supported = yes; then + LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` + else + LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` + fi + # Put the nasty error message in config.log where it belongs + echo "$LZ4_PKG_ERRORS" >&5 + + as_fn_error $? "Package requirements (liblz4) were not met: + +$LZ4_PKG_ERRORS + +Consider adjusting the PKG_CONFIG_PATH environment variable if you +installed software in a non-standard prefix. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details." "$LINENO" 5 +elif test $pkg_failed = untried; then + { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 +$as_echo "no" >&6; } + { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 +$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} +as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it +is in your PATH or set the PKG_CONFIG environment variable to the full +path to pkg-config. + +Alternatively, you may set the environment variables LZ4_CFLAGS +and LZ4_LIBS to avoid the need to call pkg-config. +See the pkg-config man page for more details. + +To get pkg-config, see . +See \`config.log' for more details" "$LINENO" 5; } +else + LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS + LZ4_LIBS=$pkg_cv_LZ4_LIBS + { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 +$as_echo "yes" >&6; } + +fi + LIBS="$LZ4_LIBS $LIBS" + CFLAGS="$LZ4_CFLAGS $CFLAGS" +fi + # # Assignments # @@ -12154,147 +12250,6 @@ fi fi -if test "$with_lz4" = yes; then - -pkg_failed=no -{ $as_echo "$as_me:${as_lineno-$LINENO}: checking for liblz4" >&5 -$as_echo_n "checking for liblz4... " >&6; } - -if test -n "$LZ4_CFLAGS"; then - pkg_cv_LZ4_CFLAGS="$LZ4_CFLAGS" - elif test -n "$PKG_CONFIG"; then - if test -n "$PKG_CONFIG" && \ - { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 - ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 - ac_status=$? - $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 - test $ac_status = 0; }; then - pkg_cv_LZ4_CFLAGS=`$PKG_CONFIG --cflags "liblz4" 2>/dev/null` - test "x$?" != "x0" && pkg_failed=yes -else - pkg_failed=yes -fi - else - pkg_failed=untried -fi -if test -n "$LZ4_LIBS"; then - pkg_cv_LZ4_LIBS="$LZ4_LIBS" - elif test -n "$PKG_CONFIG"; then - if test -n "$PKG_CONFIG" && \ - { { $as_echo "$as_me:${as_lineno-$LINENO}: \$PKG_CONFIG --exists --print-errors \"liblz4\""; } >&5 - ($PKG_CONFIG --exists --print-errors "liblz4") 2>&5 - ac_status=$? - $as_echo "$as_me:${as_lineno-$LINENO}: \$? = $ac_status" >&5 - test $ac_status = 0; }; then - pkg_cv_LZ4_LIBS=`$PKG_CONFIG --libs "liblz4" 2>/dev/null` - test "x$?" != "x0" && pkg_failed=yes -else - pkg_failed=yes -fi - else - pkg_failed=untried -fi - - - -if test $pkg_failed = yes; then - { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 -$as_echo "no" >&6; } - -if $PKG_CONFIG --atleast-pkgconfig-version 0.20; then - _pkg_short_errors_supported=yes -else - _pkg_short_errors_supported=no -fi - if test $_pkg_short_errors_supported = yes; then - LZ4_PKG_ERRORS=`$PKG_CONFIG --short-errors --print-errors --cflags --libs "liblz4" 2>&1` - else - LZ4_PKG_ERRORS=`$PKG_CONFIG --print-errors --cflags --libs "liblz4" 2>&1` - fi - # Put the nasty error message in config.log where it belongs - echo "$LZ4_PKG_ERRORS" >&5 - - as_fn_error $? "Package requirements (liblz4) were not met: - -$LZ4_PKG_ERRORS - -Consider adjusting the PKG_CONFIG_PATH environment variable if you -installed software in a non-standard prefix. - -Alternatively, you may set the environment variables LZ4_CFLAGS -and LZ4_LIBS to avoid the need to call pkg-config. -See the pkg-config man page for more details." "$LINENO" 5 -elif test $pkg_failed = untried; then - { $as_echo "$as_me:${as_lineno-$LINENO}: result: no" >&5 -$as_echo "no" >&6; } - { { $as_echo "$as_me:${as_lineno-$LINENO}: error: in \`$ac_pwd':" >&5 -$as_echo "$as_me: error: in \`$ac_pwd':" >&2;} -as_fn_error $? "The pkg-config script could not be found or is too old. Make sure it -is in your PATH or set the PKG_CONFIG environment variable to the full -path to pkg-config. - -Alternatively, you may set the environment variables LZ4_CFLAGS -and LZ4_LIBS to avoid the need to call pkg-config. -See the pkg-config man page for more details. - -To get pkg-config, see . -See \`config.log' for more details" "$LINENO" 5; } -else - LZ4_CFLAGS=$pkg_cv_LZ4_CFLAGS - LZ4_LIBS=$pkg_cv_LZ4_LIBS - { $as_echo "$as_me:${as_lineno-$LINENO}: result: yes" >&5 -$as_echo "yes" >&6; } - -fi - { $as_echo "$as_me:${as_lineno-$LINENO}: checking for LZ4_compress in -llz4" >&5 -$as_echo_n "checking for LZ4_compress in -llz4... " >&6; } -if ${ac_cv_lib_lz4_LZ4_compress+:} false; then : - $as_echo_n "(cached) " >&6 -else - ac_check_lib_save_LIBS=$LIBS -LIBS="-llz4 $LIBS" -cat confdefs.h - <<_ACEOF >conftest.$ac_ext -/* end confdefs.h. */ - -/* Override any GCC internal prototype to avoid an error. - Use char because int might match the return type of a GCC - builtin and then its argument prototype would still apply. */ -#ifdef __cplusplus -extern "C" -#endif -char LZ4_compress (); -int -main () -{ -return LZ4_compress (); - ; - return 0; -} -_ACEOF -if ac_fn_c_try_link "$LINENO"; then : - ac_cv_lib_lz4_LZ4_compress=yes -else - ac_cv_lib_lz4_LZ4_compress=no -fi -rm -f core conftest.err conftest.$ac_objext \ - conftest$ac_exeext conftest.$ac_ext -LIBS=$ac_check_lib_save_LIBS -fi -{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_lib_lz4_LZ4_compress" >&5 -$as_echo "$ac_cv_lib_lz4_LZ4_compress" >&6; } -if test "x$ac_cv_lib_lz4_LZ4_compress" = xyes; then : - cat >>confdefs.h <<_ACEOF -#define HAVE_LIBLZ4 1 -_ACEOF - - LIBS="-llz4 $LIBS" - -else - as_fn_error $? "library 'lz4' is required for LZ4 support" "$LINENO" 5 -fi - -fi - if test "$enable_spinlocks" = yes; then $as_echo "#define HAVE_SPINLOCKS 1" >>confdefs.h @@ -13561,36 +13516,6 @@ Use --without-zlib to disable zlib support." "$LINENO" 5 fi -fi - -if test "$with_lz4" = yes; then - for ac_header in lz4/lz4.h -do : - ac_fn_c_check_header_mongrel "$LINENO" "lz4/lz4.h" "ac_cv_header_lz4_lz4_h" "$ac_includes_default" -if test "x$ac_cv_header_lz4_lz4_h" = xyes; then : - cat >>confdefs.h <<_ACEOF -#define HAVE_LZ4_LZ4_H 1 -_ACEOF - -else - for ac_header in lz4.h -do : - ac_fn_c_check_header_mongrel "$LINENO" "lz4.h" "ac_cv_header_lz4_h" "$ac_includes_default" -if test "x$ac_cv_header_lz4_h" = xyes; then : - cat >>confdefs.h <<_ACEOF -#define HAVE_LZ4_H 1 -_ACEOF - -else - as_fn_error $? "lz4.h header file is required for LZ4" "$LINENO" 5 -fi - -done - -fi - -done - fi if test "$with_gssapi" = yes ; then diff --git a/configure.ac b/configure.ac index c8f199dc5e..4cd683afb3 100644 --- a/configure.ac +++ b/configure.ac @@ -995,6 +995,12 @@ PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], AC_MSG_RESULT([$with_lz4]) AC_SUBST(with_lz4) +if test "$with_lz4" = yes; then + PKG_CHECK_MODULES(LZ4, liblz4) + LIBS="$LZ4_LIBS $LIBS" + CFLAGS="$LZ4_CFLAGS $CFLAGS" +fi + # # Assignments # @@ -1183,11 +1189,6 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi -if test "$with_lz4" = yes; then - PKG_CHECK_MODULES(LZ4, liblz4) - AC_CHECK_LIB(lz4, LZ4_compress, [], [AC_MSG_ERROR([library 'lz4' is required for LZ4 support])]) -fi - if test "$enable_spinlocks" = yes; then AC_DEFINE(HAVE_SPINLOCKS, 1, [Define to 1 if you have spinlocks.]) else @@ -1421,11 +1422,6 @@ failure. It is possible the compiler isn't looking in the proper directory. Use --without-zlib to disable zlib support.])]) fi -if test "$with_lz4" = yes; then - AC_CHECK_HEADERS(lz4/lz4.h, [], - [AC_CHECK_HEADERS(lz4.h, [], [AC_MSG_ERROR([lz4.h header file is required for LZ4])])]) -fi - if test "$with_gssapi" = yes ; then AC_CHECK_HEADERS(gssapi/gssapi.h, [], [AC_CHECK_HEADERS(gssapi.h, [], [AC_MSG_ERROR([gssapi.h header file is required for GSSAPI])])]) -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0006-Add-default_toast_compression-GUC.patch" From 685d3e6178ce1443220940052cac9f5d8a08a504 Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Wed, 10 Mar 2021 14:05:05 +0530 Subject: [PATCH 6/8] Add default_toast_compression GUC Justin Pryzby and Dilip Kumar --- src/backend/access/common/toast_compression.c | 46 +++++++++++++++++++ src/backend/access/common/tupdesc.c | 2 +- src/backend/bootstrap/bootstrap.c | 2 +- src/backend/commands/tablecmds.c | 8 ++-- src/backend/utils/misc/guc.c | 12 +++++ src/backend/utils/misc/postgresql.conf.sample | 1 + src/include/access/toast_compression.h | 22 ++++++++- src/test/regress/expected/compression.out | 16 +++++++ src/test/regress/expected/compression_1.out | 19 ++++++++ src/test/regress/sql/compression.sql | 8 ++++ 10 files changed, 128 insertions(+), 8 deletions(-) diff --git a/src/backend/access/common/toast_compression.c b/src/backend/access/common/toast_compression.c index 3463b42438..e33f92687c 100644 --- a/src/backend/access/common/toast_compression.c +++ b/src/backend/access/common/toast_compression.c @@ -55,6 +55,9 @@ const CompressionRoutine toast_compression[] = } }; +/* Compile-time default */ +char *default_toast_compression = DEFAULT_TOAST_COMPRESSION; + /* * pglz_cmcompress - compression routine for pglz compression method * @@ -306,3 +309,46 @@ GetCompressionRoutines(char method) { return &toast_compression[CompressionMethodToId(method)]; } + +/* check_hook: validate new default_toast_compression */ +bool +check_default_toast_compression(char **newval, void **extra, GucSource source) +{ + if (**newval == '\0') + { + GUC_check_errdetail("%s cannot be empty.", + "default_toast_compression"); + return false; + } + + if (strlen(*newval) >= NAMEDATALEN) + { + GUC_check_errdetail("%s is too long (maximum %d characters).", + "default_toast_compression", NAMEDATALEN - 1); + return false; + } + + if (!CompressionMethodIsValid(CompressionNameToMethod(*newval))) + { + /* + * When source == PGC_S_TEST, don't throw a hard error for a + * nonexistent compression method, only a NOTICE. See comments in + * guc.h. + */ + if (source == PGC_S_TEST) + { + ereport(NOTICE, + (errcode(ERRCODE_UNDEFINED_OBJECT), + errmsg("compression method \"%s\" does not exist", + *newval))); + } + else + { + GUC_check_errdetail("Compression method \"%s\" does not exist.", + *newval); + return false; + } + } + + return true; +} diff --git a/src/backend/access/common/tupdesc.c b/src/backend/access/common/tupdesc.c index a66d1113db..362a371a6c 100644 --- a/src/backend/access/common/tupdesc.c +++ b/src/backend/access/common/tupdesc.c @@ -668,7 +668,7 @@ TupleDescInitEntry(TupleDesc desc, att->attcollation = typeForm->typcollation; if (IsStorageCompressible(typeForm->typstorage)) - att->attcompression = DefaultCompressionMethod; + att->attcompression = GetDefaultToastCompression(); else att->attcompression = InvalidCompressionMethod; diff --git a/src/backend/bootstrap/bootstrap.c b/src/backend/bootstrap/bootstrap.c index af0c0aa77e..9be0841d08 100644 --- a/src/backend/bootstrap/bootstrap.c +++ b/src/backend/bootstrap/bootstrap.c @@ -733,7 +733,7 @@ DefineAttr(char *name, char *type, int attnum, int nullness) attrtypes[attnum]->atttypmod = -1; attrtypes[attnum]->attislocal = true; if (IsStorageCompressible(attrtypes[attnum]->attstorage)) - attrtypes[attnum]->attcompression = DefaultCompressionMethod; + attrtypes[attnum]->attcompression = GetDefaultToastCompression(); else attrtypes[attnum]->attcompression = InvalidCompressionMethod; diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index e20dd5f63c..a0d2af0cde 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -11971,7 +11971,7 @@ ATExecAlterColumnType(AlteredTableInfo *tab, Relation rel, if (!IsStorageCompressible(tform->typstorage)) attTup->attcompression = InvalidCompressionMethod; else if (!CompressionMethodIsValid(attTup->attcompression)) - attTup->attcompression = DefaultCompressionMethod; + attTup->attcompression = GetDefaultToastCompression(); } else attTup->attcompression = InvalidCompressionMethod; @@ -17847,9 +17847,9 @@ GetAttributeCompression(Form_pg_attribute att, char *compression) /* fallback to default compression if it's not specified */ if (compression == NULL) - return DefaultCompressionMethod; - - cmethod = CompressionNameToMethod(compression); + cmethod = GetDefaultToastCompression(); + else + cmethod = CompressionNameToMethod(compression); return cmethod; } diff --git a/src/backend/utils/misc/guc.c b/src/backend/utils/misc/guc.c index 953837085f..e59ed5b214 100644 --- a/src/backend/utils/misc/guc.c +++ b/src/backend/utils/misc/guc.c @@ -33,6 +33,7 @@ #include "access/gin.h" #include "access/rmgr.h" #include "access/tableam.h" +#include "access/toast_compression.h" #include "access/transam.h" #include "access/twophase.h" #include "access/xact.h" @@ -3915,6 +3916,17 @@ static struct config_string ConfigureNamesString[] = check_default_table_access_method, NULL, NULL }, + { + {"default_toast_compression", PGC_USERSET, CLIENT_CONN_STATEMENT, + gettext_noop("Sets the default compression for new columns."), + NULL, + GUC_IS_NAME + }, + &default_toast_compression, + DEFAULT_TOAST_COMPRESSION, + check_default_toast_compression, NULL, NULL + }, + { {"default_tablespace", PGC_USERSET, CLIENT_CONN_STATEMENT, gettext_noop("Sets the default tablespace to create tables and indexes in."), diff --git a/src/backend/utils/misc/postgresql.conf.sample b/src/backend/utils/misc/postgresql.conf.sample index f46c2dd7a8..7d7a433dcc 100644 --- a/src/backend/utils/misc/postgresql.conf.sample +++ b/src/backend/utils/misc/postgresql.conf.sample @@ -659,6 +659,7 @@ #temp_tablespaces = '' # a list of tablespace names, '' uses # only default tablespace #default_table_access_method = 'heap' +#default_toast_compression = 'pglz' # 'pglz' or 'lz4' #check_function_bodies = on #default_transaction_isolation = 'read committed' #default_transaction_read_only = off diff --git a/src/include/access/toast_compression.h b/src/include/access/toast_compression.h index 38800cb97a..e9d9ce5634 100644 --- a/src/include/access/toast_compression.h +++ b/src/include/access/toast_compression.h @@ -15,6 +15,14 @@ #include "postgres.h" +#include "utils/guc.h" + +/* default compression method if not specified. */ +#define DEFAULT_TOAST_COMPRESSION "pglz" + +/* GUCs */ +extern char *default_toast_compression; + /* * Built-in compression methods. pg_attribute will store this in the * attcompression column. @@ -36,8 +44,6 @@ typedef enum CompressionId LZ4_COMPRESSION_ID = 1 } CompressionId; -/* use default compression method if it is not specified. */ -#define DefaultCompressionMethod PGLZ_COMPRESSION #define IsValidCompression(cm) ((cm) != InvalidCompressionMethod) #define IsStorageCompressible(storage) ((storage) != TYPSTORAGE_PLAIN && \ @@ -67,6 +73,8 @@ typedef struct CompressionRoutine extern char CompressionNameToMethod(char *compression); extern const CompressionRoutine *GetCompressionRoutines(char method); +extern bool check_default_toast_compression(char **newval, void **extra, + GucSource source); /* * CompressionMethodToId - Convert compression method to compression id. @@ -115,5 +123,15 @@ GetCompressionMethodName(char method) return GetCompressionRoutines(method)->cmname; } +/* + * GetDefaultToastCompression -- get the current toast compression + * + * This exists to hide the use of the default_toast_compression GUC variable. + */ +static inline char +GetDefaultToastCompression(void) +{ + return CompressionNameToMethod(default_toast_compression); +} #endif /* TOAST_COMPRESSION_H */ diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index 6981d580dc..9ddffc8b18 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -231,6 +231,22 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); NOTICE: merging column "f1" with inherited definition ERROR: column "f1" has a compression method conflict DETAIL: pglz versus lz4 +-- test default_toast_compression GUC +SET default_toast_compression = ''; +ERROR: invalid value for parameter "default_toast_compression": "" +DETAIL: default_toast_compression cannot be empty. +SET default_toast_compression = 'I do not exist compression'; +ERROR: invalid value for parameter "default_toast_compression": "I do not exist compression" +DETAIL: Compression method "I do not exist compression" does not exist. +SET default_toast_compression = 'lz4'; +DROP TABLE cmdata2; +CREATE TABLE cmdata2 (f1 text); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | + -- check data is ok SELECT length(f1) FROM cmdata; length diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index 893c18c7fa..03a7f46554 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -237,6 +237,25 @@ CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); NOTICE: merging column "f1" with inherited definition ERROR: column "f1" has a compression method conflict DETAIL: pglz versus lz4 +-- test default_toast_compression GUC +SET default_toast_compression = ''; +ERROR: invalid value for parameter "default_toast_compression": "" +DETAIL: default_toast_compression cannot be empty. +SET default_toast_compression = 'I do not exist compression'; +ERROR: invalid value for parameter "default_toast_compression": "I do not exist compression" +DETAIL: Compression method "I do not exist compression" does not exist. +SET default_toast_compression = 'lz4'; +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +DROP TABLE cmdata2; +CREATE TABLE cmdata2 (f1 text); +\d+ cmdata2 + Table "public.cmdata2" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | + -- check data is ok SELECT length(f1) FROM cmdata; length diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index b085c9fbd9..6c9b24f1f7 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -103,6 +103,14 @@ SELECT pg_column_compression(f1) FROM cmpart; CREATE TABLE cminh() INHERITS(cmdata, cmdata1); CREATE TABLE cminh(f1 TEXT COMPRESSION lz4) INHERITS(cmdata); +-- test default_toast_compression GUC +SET default_toast_compression = ''; +SET default_toast_compression = 'I do not exist compression'; +SET default_toast_compression = 'lz4'; +DROP TABLE cmdata2; +CREATE TABLE cmdata2 (f1 text); +\d+ cmdata2 + -- check data is ok SELECT length(f1) FROM cmdata; SELECT length(f1) FROM cmdata1; -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0007-Alter-table-set-compression.patch" From 99876667a05a13d751c3e5fb8f4a7cb3b81a986c Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Wed, 10 Mar 2021 14:34:08 +0530 Subject: [PATCH 7/8] Alter table set compression Add support for changing the compression method associated with a column. There are only built-in methods so we don't need to rewrite the table, only the new tuples will be compressed with the new compression method. Dilip Kumar based on the patches from Ildus Kurbangaliev. Design input from Robert Haas and Tomas Vondra. Reviewed by Robert Haas, Tomas Vondra, Alexander Korotkov and Justin Pryzby Discussions: https://www.postgresql.org/message-id/20171213151818.75a20259@postgrespro.ru https://www.postgresql.org/message-id/CA%2BTgmoaKDW1Oi9V%3Djc9hOGyf77NbkNEABuqgHD1Cq%3D%3D1QsOcxg%40mail.gmail.com https://www.postgresql.org/message-id/CA%2BTgmobSDVgUage9qQ5P_%3DF_9jaMkCgyKxUQGtFQU7oN4kX-AA%40mail.gmail.com https://www.postgresql.org/message-id/20201005160355.byp74sh3ejsv7wrj%40development https://www.postgresql.org/message-id/CAFiTN-tzTTT2oqWdRGLv1dvvS5MC1W%2BLE%2B3bqWPJUZj4GnHOJg%40mail.gmail.com --- doc/src/sgml/ref/alter_table.sgml | 16 ++ src/backend/commands/tablecmds.c | 198 +++++++++++++++----- src/backend/parser/gram.y | 9 + src/bin/psql/tab-complete.c | 2 +- src/include/nodes/parsenodes.h | 3 +- src/test/regress/expected/compression.out | 47 ++++- src/test/regress/expected/compression_1.out | 48 ++++- src/test/regress/sql/compression.sql | 20 ++ 8 files changed, 296 insertions(+), 47 deletions(-) diff --git a/doc/src/sgml/ref/alter_table.sgml b/doc/src/sgml/ref/alter_table.sgml index 38e416d183..e147831640 100644 --- a/doc/src/sgml/ref/alter_table.sgml +++ b/doc/src/sgml/ref/alter_table.sgml @@ -54,6 +54,7 @@ ALTER TABLE [ IF EXISTS ] name ALTER [ COLUMN ] column_name SET ( attribute_option = value [, ... ] ) ALTER [ COLUMN ] column_name RESET ( attribute_option [, ... ] ) ALTER [ COLUMN ] column_name SET STORAGE { PLAIN | EXTERNAL | EXTENDED | MAIN } + ALTER [ COLUMN ] column_name SET COMPRESSION compression_method ADD table_constraint [ NOT VALID ] ADD table_constraint_using_index ALTER CONSTRAINT constraint_name [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -104,6 +105,7 @@ WITH ( MODULUS numeric_literal, REM GENERATED { ALWAYS | BY DEFAULT } AS IDENTITY [ ( sequence_options ) ] | UNIQUE index_parameters | PRIMARY KEY index_parameters | + COMPRESSION compression_method | REFERENCES reftable [ ( refcolumn ) ] [ MATCH FULL | MATCH PARTIAL | MATCH SIMPLE ] [ ON DELETE referential_action ] [ ON UPDATE referential_action ] } [ DEFERRABLE | NOT DEFERRABLE ] [ INITIALLY DEFERRED | INITIALLY IMMEDIATE ] @@ -384,6 +386,20 @@ WITH ( MODULUS numeric_literal, REM + + + SET COMPRESSION compression_method + + + + This sets the compression method for a column. The supported compression + methods are pglz and lz4. + lz4 is available only if --with-lz4 + was used when building PostgreSQL. + + + + ADD table_constraint [ NOT VALID ] diff --git a/src/backend/commands/tablecmds.c b/src/backend/commands/tablecmds.c index a0d2af0cde..bb284741ce 100644 --- a/src/backend/commands/tablecmds.c +++ b/src/backend/commands/tablecmds.c @@ -532,6 +532,8 @@ static void ATExecReplicaIdentity(Relation rel, ReplicaIdentityStmt *stmt, LOCKM static void ATExecGenericOptions(Relation rel, List *options); static void ATExecSetRowSecurity(Relation rel, bool rls); static void ATExecForceNoForceRowSecurity(Relation rel, bool force_rls); +static ObjectAddress ATExecSetCompression(AlteredTableInfo *tab, Relation rel, + const char *column, Node *newValue, LOCKMODE lockmode); static void index_copy_data(Relation rel, RelFileNode newrnode); static const char *storage_name(char c); @@ -3977,6 +3979,7 @@ AlterTableGetLockLevel(List *cmds) */ case AT_GenericOptions: case AT_AlterColumnGenericOptions: + case AT_SetCompression: cmd_lockmode = AccessExclusiveLock; break; @@ -4511,7 +4514,8 @@ ATPrepCmd(List **wqueue, Relation rel, AlterTableCmd *cmd, case AT_DisableRowSecurity: case AT_ForceRowSecurity: case AT_NoForceRowSecurity: - ATSimplePermissions(rel, ATT_TABLE); + case AT_SetCompression: + ATSimplePermissions(rel, ATT_TABLE | ATT_MATVIEW); /* These commands never recurse */ /* No command-specific prep needed */ pass = AT_PASS_MISC; @@ -4924,6 +4928,10 @@ ATExecCmd(List **wqueue, AlteredTableInfo *tab, Relation rel, Assert(rel->rd_rel->relkind == RELKIND_INDEX); ATExecAlterCollationRefreshVersion(rel, cmd->object); break; + case AT_SetCompression: + address = ATExecSetCompression(tab, rel, cmd->name, cmd->def, + lockmode); + break; default: /* oops */ elog(ERROR, "unrecognized alter table type: %d", (int) cmd->subtype); @@ -7812,6 +7820,67 @@ ATExecSetOptions(Relation rel, const char *colName, Node *options, return address; } +/* + * Helper function for ATExecSetStorage and ATExecSetCompression + * + * Set the attcompression and/or attstorage for the respective index attribute + * if the respective input values are valid. + */ +static void +ApplyChangesToIndexes(Relation rel, Relation attrelation, AttrNumber attnum, + char newcompression, char newstorage, LOCKMODE lockmode) +{ + HeapTuple tuple; + ListCell *lc; + Form_pg_attribute attrtuple; + + foreach(lc, RelationGetIndexList(rel)) + { + Oid indexoid = lfirst_oid(lc); + Relation indrel; + AttrNumber indattnum = 0; + + indrel = index_open(indexoid, lockmode); + + for (int i = 0; i < indrel->rd_index->indnatts; i++) + { + if (indrel->rd_index->indkey.values[i] == attnum) + { + indattnum = i + 1; + break; + } + } + + if (indattnum == 0) + { + index_close(indrel, lockmode); + continue; + } + + tuple = SearchSysCacheCopyAttNum(RelationGetRelid(indrel), indattnum); + + if (HeapTupleIsValid(tuple)) + { + attrtuple = (Form_pg_attribute) GETSTRUCT(tuple); + + if (CompressionMethodIsValid(newcompression)) + attrtuple->attcompression = newcompression; + + if (newstorage != '\0') + attrtuple->attstorage = newstorage; + + CatalogTupleUpdate(attrelation, &tuple->t_self, tuple); + + InvokeObjectPostAlterHook(RelationRelationId, + RelationGetRelid(rel), + attrtuple->attnum); + + heap_freetuple(tuple); + } + + index_close(indrel, lockmode); + } +} /* * ALTER TABLE ALTER COLUMN SET STORAGE * @@ -7827,7 +7896,6 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc Form_pg_attribute attrtuple; AttrNumber attnum; ObjectAddress address; - ListCell *lc; Assert(IsA(newValue, String)); storagemode = strVal(newValue); @@ -7891,47 +7959,8 @@ ATExecSetStorage(Relation rel, const char *colName, Node *newValue, LOCKMODE loc * Apply the change to indexes as well (only for simple index columns, * matching behavior of index.c ConstructTupleDescriptor()). */ - foreach(lc, RelationGetIndexList(rel)) - { - Oid indexoid = lfirst_oid(lc); - Relation indrel; - AttrNumber indattnum = 0; - - indrel = index_open(indexoid, lockmode); - - for (int i = 0; i < indrel->rd_index->indnatts; i++) - { - if (indrel->rd_index->indkey.values[i] == attnum) - { - indattnum = i + 1; - break; - } - } - - if (indattnum == 0) - { - index_close(indrel, lockmode); - continue; - } - - tuple = SearchSysCacheCopyAttNum(RelationGetRelid(indrel), indattnum); - - if (HeapTupleIsValid(tuple)) - { - attrtuple = (Form_pg_attribute) GETSTRUCT(tuple); - attrtuple->attstorage = newstorage; - - CatalogTupleUpdate(attrelation, &tuple->t_self, tuple); - - InvokeObjectPostAlterHook(RelationRelationId, - RelationGetRelid(rel), - attrtuple->attnum); - - heap_freetuple(tuple); - } - - index_close(indrel, lockmode); - } + ApplyChangesToIndexes(rel, attrelation, attnum, InvalidCompressionMethod, + newstorage, lockmode); table_close(attrelation, RowExclusiveLock); @@ -15118,6 +15147,89 @@ ATExecGenericOptions(Relation rel, List *options) heap_freetuple(tuple); } +/* + * ALTER TABLE ALTER COLUMN SET COMPRESSION + * + * Return value is the address of the modified column + */ +static ObjectAddress +ATExecSetCompression(AlteredTableInfo *tab, + Relation rel, + const char *column, + Node *newValue, + LOCKMODE lockmode) +{ + Relation attrel; + HeapTuple tuple; + Form_pg_attribute atttableform; + AttrNumber attnum; + char *compression; + char typstorage; + Oid cmoid; + Datum values[Natts_pg_attribute]; + bool nulls[Natts_pg_attribute]; + bool replace[Natts_pg_attribute]; + ObjectAddress address; + + Assert(IsA(newValue, String)); + compression = strVal(newValue); + + attrel = table_open(AttributeRelationId, RowExclusiveLock); + + tuple = SearchSysCacheAttName(RelationGetRelid(rel), column); + if (!HeapTupleIsValid(tuple)) + ereport(ERROR, + (errcode(ERRCODE_UNDEFINED_COLUMN), + errmsg("column \"%s\" of relation \"%s\" does not exist", + column, RelationGetRelationName(rel)))); + + /* prevent them from altering a system attribute */ + atttableform = (Form_pg_attribute) GETSTRUCT(tuple); + attnum = atttableform->attnum; + if (attnum <= 0) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("cannot alter system column \"%s\"", column))); + + typstorage = get_typstorage(atttableform->atttypid); + + /* prevent from setting compression methods for uncompressible type */ + if (!IsStorageCompressible(typstorage)) + ereport(ERROR, + (errcode(ERRCODE_FEATURE_NOT_SUPPORTED), + errmsg("column data type %s does not support compression", + format_type_be(atttableform->atttypid)))); + + /* initialize buffers for new tuple values */ + memset(values, 0, sizeof(values)); + memset(nulls, false, sizeof(nulls)); + memset(replace, false, sizeof(replace)); + + /* get the attribute compression method. */ + cmoid = GetAttributeCompression(atttableform, compression); + + atttableform->attcompression = cmoid; + CatalogTupleUpdate(attrel, &tuple->t_self, tuple); + + InvokeObjectPostAlterHook(RelationRelationId, + RelationGetRelid(rel), + atttableform->attnum); + + ReleaseSysCache(tuple); + + /* apply changes to the index column as well */ + ApplyChangesToIndexes(rel, attrel, attnum, cmoid, '\0', lockmode); + table_close(attrel, RowExclusiveLock); + + /* make changes visible */ + CommandCounterIncrement(); + + ObjectAddressSubSet(address, RelationRelationId, + RelationGetRelid(rel), atttableform->attnum); + return address; +} + + /* * Preparation phase for SET LOGGED/UNLOGGED * diff --git a/src/backend/parser/gram.y b/src/backend/parser/gram.y index f8bc11d28b..07aa3c9330 100644 --- a/src/backend/parser/gram.y +++ b/src/backend/parser/gram.y @@ -2308,6 +2308,15 @@ alter_table_cmd: n->missing_ok = true; $$ = (Node *)n; } + /* ALTER TABLE ALTER [COLUMN] SET (COMPRESSION ) */ + | ALTER opt_column ColId SET optColumnCompression + { + AlterTableCmd *n = makeNode(AlterTableCmd); + n->subtype = AT_SetCompression; + n->name = $3; + n->def = (Node *) makeString($5); + $$ = (Node *)n; + } /* ALTER TABLE DROP [COLUMN] IF EXISTS [RESTRICT|CASCADE] */ | DROP opt_column IF_P EXISTS ColId opt_drop_behavior { diff --git a/src/bin/psql/tab-complete.c b/src/bin/psql/tab-complete.c index fdd1bdd92a..ebe07a17cd 100644 --- a/src/bin/psql/tab-complete.c +++ b/src/bin/psql/tab-complete.c @@ -2115,7 +2115,7 @@ psql_completion(const char *text, int start, int end) /* ALTER TABLE ALTER [COLUMN] SET */ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET")) - COMPLETE_WITH("(", "DEFAULT", "NOT NULL", "STATISTICS", "STORAGE"); + COMPLETE_WITH("(", "COMPRESSION", "DEFAULT", "NOT NULL", "STATISTICS", "STORAGE"); /* ALTER TABLE ALTER [COLUMN] SET ( */ else if (Matches("ALTER", "TABLE", MatchAny, "ALTER", "COLUMN", MatchAny, "SET", "(") || Matches("ALTER", "TABLE", MatchAny, "ALTER", MatchAny, "SET", "(")) diff --git a/src/include/nodes/parsenodes.h b/src/include/nodes/parsenodes.h index 56e24529f1..b20c03f691 100644 --- a/src/include/nodes/parsenodes.h +++ b/src/include/nodes/parsenodes.h @@ -1902,7 +1902,8 @@ typedef enum AlterTableType AT_AddIdentity, /* ADD IDENTITY */ AT_SetIdentity, /* SET identity column options */ AT_DropIdentity, /* DROP IDENTITY */ - AT_AlterCollationRefreshVersion /* ALTER COLLATION ... REFRESH VERSION */ + AT_AlterCollationRefreshVersion, /* ALTER COLLATION ... REFRESH VERSION */ + AT_SetCompression /* SET COMPRESSION */ } AlterTableType; typedef struct ReplicaIdentityStmt diff --git a/src/test/regress/expected/compression.out b/src/test/regress/expected/compression.out index 9ddffc8b18..19625708a9 100644 --- a/src/test/regress/expected/compression.out +++ b/src/test/regress/expected/compression.out @@ -247,12 +247,57 @@ CREATE TABLE cmdata2 (f1 text); --------+------+-----------+----------+---------+----------+-------------+--------------+------------- f1 | text | | | | extended | lz4 | | +-- test alter compression method +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4; +INSERT INTO cmdata VALUES (repeat('123456789',4004)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | lz4 | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz + lz4 +(2 rows) + +-- test alter compression method for the materialized view +ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4; +\d+ mv + Materialized view "public.mv" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + x | text | | | | extended | lz4 | | +View definition: + SELECT cmdata1.f1 AS x + FROM cmdata1; + +-- test alter compression method for the partitioned table +ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4; +-- new data should be compressed with the current compression method +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +SELECT pg_column_compression(f1) FROM cmpart; + pg_column_compression +----------------------- + lz4 + pglz + pglz + lz4 +(4 rows) + -- check data is ok SELECT length(f1) FROM cmdata; length -------- 10000 -(1 row) + 36036 +(2 rows) SELECT length(f1) FROM cmdata1; length diff --git a/src/test/regress/expected/compression_1.out b/src/test/regress/expected/compression_1.out index 03a7f46554..bd642634a6 100644 --- a/src/test/regress/expected/compression_1.out +++ b/src/test/regress/expected/compression_1.out @@ -256,12 +256,58 @@ CREATE TABLE cmdata2 (f1 text); --------+------+-----------+----------+---------+----------+-------------+--------------+------------- f1 | text | | | | extended | pglz | | +-- test alter compression method +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4; +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +INSERT INTO cmdata VALUES (repeat('123456789',4004)); +\d+ cmdata + Table "public.cmdata" + Column | Type | Collation | Nullable | Default | Storage | Compression | Stats target | Description +--------+------+-----------+----------+---------+----------+-------------+--------------+------------- + f1 | text | | | | extended | pglz | | +Indexes: + "idx" btree (f1) + +SELECT pg_column_compression(f1) FROM cmdata; + pg_column_compression +----------------------- + pglz + pglz +(2 rows) + +-- test alter compression method for the materialized view +ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4; +ERROR: relation "mv" does not exist +\d+ mv +-- test alter compression method for the partitioned table +ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz; +ERROR: relation "cmpart1" does not exist +ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4; +ERROR: unsupported LZ4 compression method +DETAIL: This functionality requires the server to be built with lz4 support. +HINT: You need to rebuild PostgreSQL using --with-lz4. +-- new data should be compressed with the current compression method +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',1004)); + ^ +INSERT INTO cmpart VALUES (repeat('123456789',4004)); +ERROR: relation "cmpart" does not exist +LINE 1: INSERT INTO cmpart VALUES (repeat('123456789',4004)); + ^ +SELECT pg_column_compression(f1) FROM cmpart; +ERROR: relation "cmpart" does not exist +LINE 1: SELECT pg_column_compression(f1) FROM cmpart; + ^ -- check data is ok SELECT length(f1) FROM cmdata; length -------- 10000 -(1 row) + 36036 +(2 rows) SELECT length(f1) FROM cmdata1; ERROR: relation "cmdata1" does not exist diff --git a/src/test/regress/sql/compression.sql b/src/test/regress/sql/compression.sql index 6c9b24f1f7..00b1ff1137 100644 --- a/src/test/regress/sql/compression.sql +++ b/src/test/regress/sql/compression.sql @@ -111,6 +111,26 @@ DROP TABLE cmdata2; CREATE TABLE cmdata2 (f1 text); \d+ cmdata2 +-- test alter compression method +ALTER TABLE cmdata ALTER COLUMN f1 SET COMPRESSION lz4; +INSERT INTO cmdata VALUES (repeat('123456789',4004)); +\d+ cmdata +SELECT pg_column_compression(f1) FROM cmdata; + +-- test alter compression method for the materialized view +ALTER MATERIALIZED VIEW mv ALTER COLUMN x SET COMPRESSION lz4; +\d+ mv + +-- test alter compression method for the partitioned table +ALTER TABLE cmpart1 ALTER COLUMN f1 SET COMPRESSION pglz; +ALTER TABLE cmpart2 ALTER COLUMN f1 SET COMPRESSION lz4; + +-- new data should be compressed with the current compression method +INSERT INTO cmpart VALUES (repeat('123456789',1004)); +INSERT INTO cmpart VALUES (repeat('123456789',4004)); + +SELECT pg_column_compression(f1) FROM cmpart; + -- check data is ok SELECT length(f1) FROM cmdata; SELECT length(f1) FROM cmdata1; -- 2.17.0 --cvVnyQ+4j833TQvp Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="0008-default-to-with-lz4.patch" From a1cc5febe7065f777b66470162cea5eb5899b7cc Mon Sep 17 00:00:00 2001 From: Dilip Kumar Date: Wed, 10 Mar 2021 14:44:42 +0530 Subject: [PATCH 8/8] default to --with-lz4 this is meant to excercize the new feature in the CIs, and not meant to be merged --- configure | 6 ++++-- configure.ac | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/configure b/configure index 387c553d5a..66b9edcc7b 100755 --- a/configure +++ b/configure @@ -1575,7 +1575,7 @@ Optional Packages: --with-system-tzdata=DIR use system time zone data in DIR --without-zlib do not use Zlib - --with-lz4 build with LZ4 support + --without-lz4 build without LZ4 support --with-gnu-ld assume the C compiler uses GNU ld [default=no] --with-ssl=LIB use LIB for SSL/TLS support (openssl) --with-openssl obsolete spelling of --with-ssl=openssl @@ -8598,7 +8598,9 @@ $as_echo "#define USE_LZ4 1" >>confdefs.h esac else - with_lz4=no + with_lz4=yes + +$as_echo "#define USE_LZ4 1" >>confdefs.h fi diff --git a/configure.ac b/configure.ac index 4cd683afb3..7907aded9a 100644 --- a/configure.ac +++ b/configure.ac @@ -990,8 +990,8 @@ AC_SUBST(with_zlib) # LZ4 # AC_MSG_CHECKING([whether to build with LZ4 support]) -PGAC_ARG_BOOL(with, lz4, no, [build with LZ4 support], - [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build with LZ4 support. (--with-lz4)])]) +PGAC_ARG_BOOL(with, lz4, yes, [build without LZ4 support], + [AC_DEFINE([USE_LZ4], 1, [Define to 1 to build without LZ4 support. (--without-lz4)])]) AC_MSG_RESULT([$with_lz4]) AC_SUBST(with_lz4) -- 2.17.0 --cvVnyQ+4j833TQvp--