Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sOB4B-00CpBG-CM for pgsql-announce@arkaria.postgresql.org; Mon, 01 Jul 2024 07:03:11 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sOB49-00Fney-Mt for pgsql-announce@arkaria.postgresql.org; Mon, 01 Jul 2024 07:03:10 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sOB48-00Fne2-KW for pgsql-announce@lists.postgresql.org; Mon, 01 Jul 2024 07:03:09 +0000 Received: from mahout.postgresql.org ([2001:4800:3e1:1::227]) by makus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sOB45-003zHj-Er for pgsql-announce@lists.postgresql.org; Mon, 01 Jul 2024 07:03:07 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=postgresql.org; s=20171124; h=Message-ID:Date:Reply-To:From:To:Subject: MIME-Version:Content-Type:Sender:Cc:Content-Transfer-Encoding:Content-ID: Content-Description:In-Reply-To:References; bh=WT1C8E3HDcZ0aNJyrYdQuQWtMKID9jDfBna/Q8Ap2cA=; b=M5k1EAy+JXcZplXQVnk9Uz7y5w QoCwWEcCXy7q3kwO1t2emvhHvImcuA5glp5HL0bsq6vTSk822OcKCvPZ4w6on0Cbe8fK2RA6N7tDj GSemj82d4WLtIYGorQFqPs63D6+9V4iD7kLqLc6vxN9kPsGlPjKNowdO1DPtBTtEzyEG+C/y05JMy /G0rM9xq8IQo7mCCrrxmCJBYzLu18EDfawiL4rwVodpS+wBIQfenSJSdFDtZW/rV9IBDtZRkX//FH VZDdJP6nWDj5GPwZnfmwn+im9efOAzd69iSJobMIuQnRtloxMZ2ikbIU3RaT1DimuPBvUDvEljUHy daleXt8w==; Received: from wrigleys.postgresql.org ([2a02:16a8:dc51::60]) by mahout.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1sOB44-00CcJU-Bs for pgsql-announce@lists.postgresql.org; Mon, 01 Jul 2024 07:03:04 +0000 Received: from localhost ([127.0.0.1] helo=wrigleys.postgresql.org) by wrigleys.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1sOB43-007PcA-3E for pgsql-announce@lists.postgresql.org; Mon, 01 Jul 2024 07:03:02 +0000 Content-Type: multipart/mixed; boundary="===============3563550150148469430==" MIME-Version: 1.0 Subject: Greenmask v0.2.0b1 Release To: PostgreSQL Announce From: "Greenmask.io via PostgreSQL Announce" Reply-To: info@greenmask.io Date: Mon, 01 Jul 2024 07:02:12 +0000 Message-ID: <171981733221.699.12562411608736373211@wrigleys.postgresql.org> X-Auto-Response-Suppress: All Auto-Submitted: auto-generated X-pglister-tags: related,security X-pglister-tagsig: 01457491605f48304d7d9d11a1bacec5cecf7d7a4f29483693688b0465be33f7 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --===============3563550150148469430== Content-Type: multipart/alternative; boundary="===============0612736970004896121==" MIME-Version: 1.0 --===============0612736970004896121== Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable # Greenmask v0.2.0b1 Release ## PostgreSQL Logical Dump and Anonymization Tool This major **beta release** introduces new features and refactored transfor= mers, significantly enhancing Greenmask's flexibility to better meet busine= ss needs. Help us improve GreenMask and tailor it to meet community needs. = We welcome your feedback in the [release discussion on GitHub](https://gith= ub.com/GreenmaskIO/greenmask/discussions/129). ## Greenmask Overview Greenmask is a versatile open-source tool for database backup, anonymizatio= n, and restoration. Written in pure Go with ported PostgreSQL libraries, it= is platform-independent and stateless, requiring no schema modifications. = It is customizable and compatible with existing PostgreSQL utilities. ### Greenmask is ideally suited for: * Routine backup and restoration tasks, ensuring data integrity and availab= ility. * Anonymization and data masking for staging environments and analytics, **= protecting sensitive** information while maintaining data utility. ### Key features * **Deterministic transformers** =E2=80=94 deterministic approach to data t= ransformation based on the hash functions. This ensures that the same input data will always produce the = same output data. Almost each transformer supports either `random` or `hash` engine making it universal for any use= case. * **Dynamic parameters** =E2=80=94 almost each transformer supports dynamic= parameters, allowing to parametrize the transformer dynamically from the table column value. This is helpful for = resolving the functional dependencies=20 between columns and satisfying the constraints. * **Database type safe** - Ensures data integrity by validating data and ut= ilizing the database driver for encoding and decoding operations. This approach guarantees the preservati= on of data formats. * **Transformation validation and easy maintainable** - During obfuscation = development, Greenmask provides validation warnings and a transformation diff feature, allowing you to monitor and m= aintain transformations effectively throughout the software lifecycle. * **Partitioned tables transformation inheritance** - Define transformation= configurations once and apply them to all partitions within partitioned tables, simplifying the obfuscation process. * **Stateless** - Greenmask operates as a logical dump and does not impact = your existing database schema. * **Backward compatible** - It fully supports the same features and protoco= ls as existing vanilla PostgreSQL utilities. Dumps created by Greenmask can be successfully restored using the pg_rest= ore utility. * **Extensible** - Users have the flexibility to implement domain-based tra= nsformations in any programming language or use predefined templates. * **Provide a variety of storage** - Greenmask offers a variety of storage = options for local and remote data storage, including directories and S3-like storage solutions. ## Playground usage for the beta version If you want to run a Greenmask [playground](https://greenmask.io/v0.2.0b1/p= layground/) for the beta version execute: ``` git checkout tags/v0.2.0b1 -b v0.2.0b1 docker-compose run greenmask-from-source ``` ## Changes overview * [Introduced dynamic parameters in the transformers](https://greenmask.io/= v0.2.0b1/built_in_transformers/dynamic_parameters/) * Most transformers now support dynamic parameters where applicable. * Dynamic parameters are strictly enforced. If you need to cast values = to another type, Greenmask provides templates and predefined cast functions= accessible via `cast_to`. These functions cover frequent operations such a= s `UnixTimestampToDate` and `IntToBool`. * The transformation logic has been significantly refactored, making transf= ormers more customizable and flexible than before. * [Introduced transformation engines](https://greenmask.io/v0.2.0b1/built_i= n_transformers/transformation_engines/) * `random` - generates transformer values based on pseudo-random algori= thms. * `hash` - generates transformer values using hash functions. Currently= , it utilizes `sha3` hash functions, which are secure but perform slowly. I= n the stable release, there will be an option to choose between `sha3` and = `SipHash`. * [Introduced static parameters value template](https://greenmask.io/v0.2.0= b1/built_in_transformers/parameters_templating/) ## Notable changes ### Core * Introduced the `Parametrizer` interface, now implemented for both dynamic= and static parameters. * Renamed most of the toolkit types for enhanced clarity and comprehensive = documentation coverage. * Refactored the `Driver` initialization logic. * Added validation warnings for overridden types in the `Driver`. * Migrated existing built-in transformers to utilize the new `Parametrizer`= interface. * Implemented a new abstraction, `TransformationContext`, as the first step= towards enabling new feature transformation conditions (#34). * Optimized most transformers for performance in both dynamic and static mo= des. While dynamic mode offers flexibility, static mode ensures performance= remains high. Using only the necessary transformation features helps keep = transformation time predictable. ### Documentation Documentation has been significantly refactored. New information about feat= ures and updates to transformer descriptions have been added. ### Transformers * [RandomEmail](https://greenmask.io/v0.2.0b1/built_in_transformers/standar= d_transformers/random_email/) - Introduces a new transformer that supports = both random and deterministic engines. It allows for flexible email value g= eneration; you can use column values in the template and choose to keep the= original domain or select any from the `domains` parameter. * [NoiseDate](https://greenmask.io/v0.2.0b1/built_in_transformers/standard_= transformers/noise_date/), [NoiseFloat](https://greenmask.io/v0.2.0b1/built= _in_transformers/standard_transformers/noise_float/), [NoiseInt](https://gr= eenmask.io/v0.2.0b1/built_in_transformers/standard_transformers/noise_int/)= - These transformers support both random and deterministic engines, offeri= ng dynamic mode parameters that control the noise thresholds within the `mi= n` and `max` range. Unlike previous implementations which used a single `ra= tio` parameter, the new release features `min_ratio` and `max_ratio` parame= ters to define noise values more precisely. Utilizing the `hash` engine in = these transformers enhances security by complicating statistical analysis f= or attackers, especially when the same salt is used consistently over long = periods. * [NoiseNumeric](https://greenmask.io/v0.2.0b1/built_in_transformers/standa= rd_transformers/noise_numeric/) - A newly implemented transformer, sharing = features with `NoiseInt` and `NoiseFloat`, but specifically designed for nu= meric values (large integers or floats). It provides a `decimal` parameter = to handle values with fractions. * [RandomChoice](https://greenmask.io/v0.2.0b1/built_in_transformers/standa= rd_transformers/random_choice/) - Now supports the `hash` engine * [RandomDate](https://greenmask.io/v0.2.0b1/built_in_transformers/standard= _transformers/random_date/), [RandomFloat](https://greenmask.io/v0.2.0b1/bu= ilt_in_transformers/standard_transformers/random_float/), [RandomInt](https= ://greenmask.io/v0.2.0b1/built_in_transformers/standard_transformers/random= _int/) - Now enhanced with hash engine support. Threshold parameters `min` = and `max` have been updated to support dynamic mode, allowing for more flex= ible configurations. * [RandomNumeric](https://greenmask.io/v0.2.0b1/built_in_transformers/stand= ard_transformers/random_numeric/) - A new transformer specifically designed= for numeric types (large integers or floats), sharing similar features wit= h `RandomInt` and `RandomFloat`, but tailored for handling huge numeric val= ues. * [RandomString](https://greenmask.io/v0.2.0b1/built_in_transformers/standa= rd_transformers/random_string/) - Now supports hash engine mode * [RandomUnixTimestamp](https://greenmask.io/v0.2.0b1/built_in_transformers= /standard_transformers/random_unix_timestamp/) - This new transformer gener= ates Unix timestamps with selectable units (`second`, `millisecond`, `micro= second`, `nanosecond`). Similar in function to `RandomDate`, it supports th= e hash engine and dynamic parameters for `min` and `max` thresholds, with t= he ability to override these units using `min_unit` and `max_unit` paramete= rs. * [RandomUuid](https://greenmask.io/v0.2.0b1/built_in_transformers/standard= _transformers/random_uuid/) - Added hash engine support * [RandomPerson](https://greenmask.io/v0.2.0b1/built_in_transformers/standa= rd_transformers/random_person/) - Implemented a new transformer that replac= es `RandomName`, `RandomLastName`, `RandomFirstName`, `RandomFirstNameMale`= , `RandomFirstNameFemale`, `RandomTitleMale`, and `RandomTitleFemale`. This= new transformer offers enhanced customizability while providing similar fu= nctionalities as the previous versions. It generates personal data such as = `FirstName`, `LastName`, and `Title`, based on the provided `gender` parame= ter, which now supports dynamic mode. Future minor versions will allow for = overriding the default names database. * Added [tsModify](https://greenmask.io/v0.2.0b1/built_in_transformers/adva= nced_transformers/custom_functions/core_functions/#tsmodify) - a new templa= te function for time.Time objects modification * Introduced a new [RandomIp](https://greenmask.io/v0.2.0b1/built_in_transf= ormers/standard_transformers/random_ip/) transformer capable of generating = a random IP address based on the specified netmask. * Added a new [RandomMac](https://greenmask.io/v0.2.0b1/built_in_transforme= rs/standard_transformers/random_mac/) transformer for generating random Mac= addresses. * Deleted transformers include `RandomMacAddress`, `RandomIPv4`, `RandomIPv= 6`, `RandomUnixTime`, `RandomTitleMale`, `RandomTitleFemale`, `RandomFirstN= ame`, `RandomFirstNameMale`, `RandomFirstNameFemale`, `RandomLastName`, and= `RandomName` due to the introduction of more flexible and unified options. # Useful Links * Public [Roadmap](https://github.com/orgs/GreenmaskIO/projects/6) * Explore detailed [Documentation](https://greenmask.io/v0.2.0b1/installati= on/) * Access the Latest [Release on GitHub](https://github.com/GreenmaskIO/gree= nmask/releases/tag/v0.2.0b1) * Contact us for support at [Email Support](mailto:support@greenmask.io) --===============0612736970004896121== Content-Type: text/html; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Greenmask v0.2.0b1 Release
 

Greenmask v0.2.0b1 Release

Greenmask v0.2.0b1 Release

PostgreSQL Logical Dump and Ano= nymization Tool

This major beta release in= troduces new features and refactored transformers, significantly enhancing = Greenmask's flexibility to better meet business needs. Help us improve Gree= nMask and tailor it to meet community needs. We welcome your feedback in th= e release discussion on GitH= ub.

Greenmask Overview

Greenmask is a versatile open-source tool f= or database backup, anonymization, and restoration. Written in pure Go with= ported PostgreSQL libraries, it is platform-independent and stateless, req= uiring no schema modifications. It is customizable and compatible with exis= ting PostgreSQL utilities.

Greenmask is ideally suited for= :

  • Routine backup = and restoration tasks, ensuring data integrity and availability.
  • Anonymization a= nd data masking for staging environments and analytics, protecting = sensitive information while maintaining data utility.

Key features

  • Determi= nistic transformers =E2=80=94 deterministic approach to data trans= formation based on the hash functions. This ensures that the same input data will always produce the = same output data. Almost each transformer supports either random or hash engine making it= universal for any use case.
  • Dynamic= parameters =E2=80=94 almost each transformer supports dynamic par= ameters, allowing to parametrize the transformer dynamically from the table column value. This is helpful for = resolving the functional dependencies=20 between columns and satisfying the constraints.
  • Databas= e type safe - Ensures data integrity by validating data and utiliz= ing the database driver for encoding and decoding operations. This approach guarantees the preservati= on of data formats.
  • Transfo= rmation validation and easy maintainable - During obfuscation deve= lopment, Greenmask provides validation warnings and a transformation diff feature, allowing you to monitor and m= aintain transformations effectively throughout the software lifecycle.
  • Partiti= oned tables transformation inheritance - Define transformation con= figurations once and apply them to all partitions within partitioned tables, simplifying the obfuscation process= .
  • Statele= ss - Greenmask operates as a logical dump and does not impact your= existing database schema.
  • Backwar= d compatible - It fully supports the same features and protocols a= s existing vanilla PostgreSQL utilities. Dumps created by Greenmask can be successfully restored using the pg_rest= ore utility.
  • Extensi= ble - Users have the flexibility to implement domain-based transfo= rmations in any programming language or use predefined templates.
  • Provide= a variety of storage - Greenmask offers a variety of storage opti= ons for local and remote data storage, including directories and S3-like storage solutions.

Playground usage for the beta v= ersion

If you want to run a Greenmask playground for the beta version execute:

git checkout tags/v0.2.0b1 -b v0.2.0b1 docker-compose run greenmask-from-source

Changes overview

  • Introduced dynamic paramete= rs in the transformers
    • Most transforme= rs now support dynamic parameters where applicable.
    • Dynamic paramet= ers are strictly enforced. If you need to cast values to another type, Gree= nmask provides templates and predefined cast functions accessible via cast_to. These functions cover frequent operations such as Un= ixTimestampToDate and IntToBool.
  • The transformat= ion logic has been significantly refactored, making transformers more custo= mizable and flexible than before.

  • Introduced transformation engines

    • random - generates transformer values based on pseudo-random algorithms.
    • hash - generates transformer values using hash functions. Currently, it utili= zes sha3 hash functions, which are secure but perform slowly. = In the stable release, there will be an option to choose between sha3= and SipHash.

  • Introduced static parameters value template

Notable changes

Core

  • Introduced the = Parametrizer interface, now implemented for both dynamic and s= tatic parameters.
  • Renamed most of= the toolkit types for enhanced clarity and comprehensive documentation cov= erage.
  • Refactored the = Driver initialization logic.
  • Added validatio= n warnings for overridden types in the Driver.
  • Migrated existi= ng built-in transformers to utilize the new Parametrizer inter= face.
  • Implemented a n= ew abstraction, TransformationContext, as the first step towar= ds enabling new feature transformation conditions (#34).
  • Optimized most = transformers for performance in both dynamic and static modes. While dynami= c mode offers flexibility, static mode ensures performance remains high. Us= ing only the necessary transformation features helps keep transformation ti= me predictable.

Documentation

Documentation has been significantly refact= ored. New information about features and updates to transformer description= s have been added.

Transformers

  • RandomEmail - Introduces a new trans= former that supports both random and deterministic engines. It allows for f= lexible email value generation; you can use column values in the template a= nd choose to keep the original domain or select any from the domains<= /code> parameter.

  • NoiseDate, NoiseFloat, NoiseIn= t - These transformers support both random and deterministic engines, o= ffering dynamic mode parameters that control the noise thresholds within th= e min and max range. Unlike previous implementati= ons which used a single ratio parameter, the new release featu= res min_ratio and max_ratio parameters to define = noise values more precisely. Utilizing the hash engine in thes= e transformers enhances security by complicating statistical analysis for a= ttackers, especially when the same salt is used consistently over long peri= ods.

  • NoiseNumeric - A newly implemented = transformer, sharing features with NoiseInt and NoiseFlo= at, but specifically designed for numeric values (large integers or = floats). It provides a decimal parameter to handle values with= fractions.

  • RandomChoice - Now supports the hash engine

  • RandomDate, RandomFloat, Rand= omInt - Now enhanced with hash engine support. Threshold parameters min and max have been updated to support dynamic mod= e, allowing for more flexible configurations.

  • RandomNumeric - A new transformer = specifically designed for numeric types (large integers or floats), sharing= similar features with RandomInt and RandomFloat,= but tailored for handling huge numeric values.

  • RandomString - Now supports hash en= gine mode

  • RandomUnixTimestamp - This = new transformer generates Unix timestamps with selectable units (seco= nd, millisecond, microsecond, nanose= cond). Similar in function to RandomDate, it supports t= he hash engine and dynamic parameters for min and max thresholds, with the ability to override these units using min_u= nit and max_unit parameters.

  • RandomUuid - Added hash engine suppor= t

  • RandomPerson - Implemented a new tr= ansformer that replaces RandomName, RandomLastName, RandomFirstName, RandomFirstNameMale, Ra= ndomFirstNameFemale, RandomTitleMale, and RandomT= itleFemale. This new transformer offers enhanced customizability whi= le providing similar functionalities as the previous versions. It generates= personal data such as FirstName, LastName, and <= code>Title, based on the provided gender parameter, whi= ch now supports dynamic mode. Future minor versions will allow for overridi= ng the default names database.

  • Added tsModi= fy - a new template function for time.Time objects modification

  • Introduced a new RandomIp transformer c= apable of generating a random IP address based on the specified netmask.

  • Added a new RandomMac transformer for = generating random Mac addresses.

  • Deleted transformers include RandomMa= cAddress, RandomIPv4, RandomIPv6, Ra= ndomUnixTime, RandomTitleMale, RandomTitleFemale<= /code>, RandomFirstName, RandomFirstNameMale, RandomFirstNameFemale, RandomLastName, and Ran= domName due to the introduction of more flexible and unified options= .

Useful Links
This email was sent to you from Greenmask.io. It was delivered on their beh= alf by the PostgreSQL project. Any questions about the content of the message shou= ld be sent to Greenmask.io.

You were sent this email as a subscriber of the pgsql-announce mai= linglist, for for one of the content tags Related Open Source or Security. To unsubscribe from further emails, or change which emails you want to receive, please click th= e personal unsubscribe link that you can find in the headers of this email, or visit https://lists.postgresql.org/unsubscribe/.
 
--===============0612736970004896121==-- --===============3563550150148469430==--