Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jOCVK-0003VN-KD for pgsql-bugs@arkaria.postgresql.org; Tue, 14 Apr 2020 03:44:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jOCVJ-0003A7-5a for pgsql-bugs@arkaria.postgresql.org; Tue, 14 Apr 2020 03:44:53 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jOCVI-0003A0-UV for pgsql-bugs@lists.postgresql.org; Tue, 14 Apr 2020 03:44:52 +0000 Received: from mail-pg1-x52a.google.com ([2607:f8b0:4864:20::52a]) by magus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1jOCVG-0000HT-Ki for pgsql-bugs@lists.postgresql.org; Tue, 14 Apr 2020 03:44:52 +0000 Received: by mail-pg1-x52a.google.com with SMTP id h69so2883821pgc.8 for ; Mon, 13 Apr 2020 20:44:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=uGupFY9TIa7C5EXZVLUAMiWDFeLxDenVj4gY5TTfDc8=; b=mm5zybLh9K+U8QMFqgE1loVF7mtkMDG5ebTOqK5UOJURNB8zmFgxjJVB1NZIexoJdc IDCJnqtzahmNWFzFyKYokw0BfdeiniWoZ8tGfyS2pWO0FTo3IosaKKuBtq3dM34ls8+o qhdhVf8V5DDB6BfMl/tLHC0RI2WIDlLLdj+2FgB0k33iUwg2sViCUEbTEtc10COlw2Km A5sS/9YWudnoawY+T1hO1ZFo1mFQOxaJrNCaNInxVc+nResYpX9fjGtmjW1S1xs9ETGT 5DG18KBbB0rOz3y8Mqk4J8JaiaUghW9dUFAtRISkouwgKYV5n7kTNyUoYYN75pwUUD7v 7swQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=uGupFY9TIa7C5EXZVLUAMiWDFeLxDenVj4gY5TTfDc8=; b=WGBuLMPcB+WT4M/UDmyE1LwnFTWtVRdiYXeVEpaQ7crVkKKXzRHwH8CfxrNT/Gj1bO qN1H0rldZEvYirdz+oC5CuMJc9UY0iRK65bs4wIU2DmgOlUFEpciuZ7MWLP6Ahmov7qx M2RhBP0P+v8vtHUYrdiovh+RgohYetwxOzwDNAK91Uj/dk5+uPo/YQCCCoGWOQtqKKw1 8KDzewiJD5e7FhI5r19UrkDLy2fU4u2YtuE/9SvgNRv8OJyopN9uOq3jMLjdWhEFhayn cnp2YWSJ9+vc40sqA2XevBy3dRhP85farIKDDYIu7Blr2AewzpRB0vMvAo/ipKtyvHx2 7IiQ== X-Gm-Message-State: AGi0Pub14v35okQegMZvgp8ykoVndXz24N8HOpZ0GFl79poxs7gERI26 HBoemdEwAoCiEsF/QaE98mQ= X-Google-Smtp-Source: APiQypKm+XTMo7Dd3VAFJeOUGz45PKyltqnm+kg7sIcvoRc0TK5G+RoYUJHSl85B79+4Ai0YKw+gwQ== X-Received: by 2002:a63:894a:: with SMTP id v71mr21006714pgd.314.1586835888438; Mon, 13 Apr 2020 20:44:48 -0700 (PDT) Received: from [192.168.2.111] (113x32x180x194.ap113.ftth.ucom.ne.jp. [113.32.180.194]) by smtp.gmail.com with ESMTPSA id l15sm9160737pgk.59.2020.04.13.20.44.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 13 Apr 2020 20:44:47 -0700 (PDT) Subject: Re: BUG #16337: Finnish Ispell dictionary cannot be created To: Kyotaro Horiguchi Cc: tomas.vondra@2ndquadrant.com, matti.linnanvuori@portalify.com, pgsql-bugs@lists.postgresql.org, noreply@postgresql.org References: <20200403085509.bwrvm7hfh6h4vgrl@development> <20200413.173610.1847967467851370073.horikyota.ntt@gmail.com> From: Artur Zakirov Message-ID: <7ce82fff-5368-47b8-671e-31ea340b0cde@gmail.com> Date: Tue, 14 Apr 2020 12:44:44 +0900 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: <20200413.173610.1847967467851370073.horikyota.ntt@gmail.com> Content-Type: multipart/mixed; boundary="------------5F7D1F4A41F829698D47D893" Content-Language: en-US List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------5F7D1F4A41F829698D47D893 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hello Horiguchi-san, On 4/13/2020 5:36 PM, Kyotaro Horiguchi wrote: > Looking man 5 ispell, "Any character with special meaning to parser > can be changed to an uniterpreted token by backslashing it". It > depends on how we sholud be strict on that, but I think it is safer > that we think that any character prefixed by a backslash is an word > character. (I don't understand how '-' can be in a word by the > definition in the .affix file, though.) > > Since a escaped character is intended to be a part of a word, there's > no point in identifying minus-sign ad-hockerly, I think. Thank you to pay attention to the patch. I don't mind if the patch will work in more broad cases. But I tested ispell utility with other characters other than '-' before. It seems that it ignores such affixes or doesn't work properly. But in general maybe it is better to stick closer with the man page description. I attached new version of the patch. It fixes only PAE_INFIND and PAE_INREPL cases. I think we shouldn't allow to escape all cases and it is safer to have some exceptions: - In PAE_WAIT_MASK we shouldn't escape comment string which starts with '#' - PAE_INMASK case is handled by regcomp.c separately and maybe it is better to leave the string as-is - PAE_WAIT_FIND can start only with '-' - I don't think that there is a sense in escaping PAE_WAIT_REPL And in PAE_INFIND and PAE_INREPL I think we shouldn't allow to escape ',' and '#'. The condition: if (t_iseq(str, '\\') && !isescaped && (state == PAE_INFIND || state == PAE_INREPL)) maybe is not great, but I cannot come up with a better solution. -- Artur --------------5F7D1F4A41F829698D47D893 Content-Type: text/plain; charset=UTF-8; name="tsearch_escape_hyphen_v2.patch" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="tsearch_escape_hyphen_v2.patch" ZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3RzZWFyY2gvZGljdHMvaXNwZWxsX3NhbXBsZS5h ZmZpeCBiL3NyYy9iYWNrZW5kL3RzZWFyY2gvZGljdHMvaXNwZWxsX3NhbXBsZS5hZmZpeApp bmRleCBmMjkwMDRmZjFkLi45M2I4ZDU0MzVjIDEwMDY0NAotLS0gYS9zcmMvYmFja2VuZC90 c2VhcmNoL2RpY3RzL2lzcGVsbF9zYW1wbGUuYWZmaXgKKysrIGIvc3JjL2JhY2tlbmQvdHNl YXJjaC9kaWN0cy9pc3BlbGxfc2FtcGxlLmFmZml4CkBAIC01LDYgKzUsOSBAQCBwcmVmaXhl cwogZmxhZyAqQjoKIAkuICAgICAgID4gICBSRSAgICAgICMgQXMgaW4gZW50ZXIgPiByZWVu dGVyCiAKK2ZsYWcgRToKKwkuICAgICAgID4gICBJTlwtICAgICMgQXMgaW4gbGF3ID4gaW4t bGF3CisKIGZsYWcgVToKICAgICAuICAgICAgID4gICBVTiAgICAgICMgQXMgaW4gbmF0dXJh bCA+IHVubmF0dXJhbAogCmRpZmYgLS1naXQgYS9zcmMvYmFja2VuZC90c2VhcmNoL2RpY3Rz L2lzcGVsbF9zYW1wbGUuZGljdCBiL3NyYy9iYWNrZW5kL3RzZWFyY2gvZGljdHMvaXNwZWxs X3NhbXBsZS5kaWN0CmluZGV4IDQ0ZGYxOTY3YTYuLjU1OGM3MTU0YzEgMTAwNjQ0Ci0tLSBh L3NyYy9iYWNrZW5kL3RzZWFyY2gvZGljdHMvaXNwZWxsX3NhbXBsZS5kaWN0CisrKyBiL3Ny Yy9iYWNrZW5kL3RzZWFyY2gvZGljdHMvaXNwZWxsX3NhbXBsZS5kaWN0CkBAIC01LDQgKzUs NSBAQCBmb290L1pTCiBmb290YmFsbC9aCiBiYWxsL1NaXAoga2x1YmJlci9aCitsYXcvRQog c2t5L0EKZGlmZiAtLWdpdCBhL3NyYy9iYWNrZW5kL3RzZWFyY2gvc3BlbGwuYyBiL3NyYy9i YWNrZW5kL3RzZWFyY2gvc3BlbGwuYwppbmRleCA4YWFiOTZkM2IwLi5jY2I0NjMzOWUyIDEw MDY0NAotLS0gYS9zcmMvYmFja2VuZC90c2VhcmNoL3NwZWxsLmMKKysrIGIvc3JjL2JhY2tl bmQvdHNlYXJjaC9zcGVsbC5jCkBAIC05MDksMTEgKzkwOSwyNSBAQCBwYXJzZV9hZmZlbnRy eShjaGFyICpzdHIsIGNoYXIgKm1hc2ssIGNoYXIgKmZpbmQsIGNoYXIgKnJlcGwpCiAJY2hh cgkgICAqcG1hc2sgPSBtYXNrLAogCQkJICAgKnBmaW5kID0gZmluZCwKIAkJCSAgICpwcmVw bCA9IHJlcGw7CisJYm9vbAkJaXNlc2NhcGVkID0gZmFsc2U7CiAKIAkqbWFzayA9ICpmaW5k ID0gKnJlcGwgPSAnXDAnOwogCiAJd2hpbGUgKCpzdHIpCiAJeworCQlpZiAodF9pc2VxKHN0 ciwgJ1xcJykgJiYgIWlzZXNjYXBlZCAmJgorCQkJKHN0YXRlID09IFBBRV9JTkZJTkQgfHwg c3RhdGUgPT0gUEFFX0lOUkVQTCkpCisJCXsKKwkJCS8qCisJCQkgKiBOZXh0IGNoYXJhY3Rl ciBzaG91bGQgYmUgZXNjYXBlZC4gQ3VycmVudGx5IGl0IGlzIGFwcGx5ZWQgb25seSB0bwor CQkJICogUEFFX0lORklORCBhbmQgUEFFX0lOUkVQTC4gV2UgZXhwZWN0IHRoYXQgPGZpbmQ+ IGNhbiBzdGFydCBvbmx5CisJCQkgKiB3aXRoICctJyBhbmQgPHJlcGxhY2U+IGNhbiBzdGFy dCBvbmx5IHdpdGggYW4gYWxwaGFiZXQgY2hhcmFjdGVyLgorCQkJICovCisJCQlpc2VzY2Fw ZWQgPSB0cnVlOworCQkJc3RyICs9IHBnX21ibGVuKHN0cik7CisJCQljb250aW51ZTsKKwkJ fQorCiAJCWlmIChzdGF0ZSA9PSBQQUVfV0FJVF9NQVNLKQogCQl7CiAJCQlpZiAodF9pc2Vx KHN0ciwgJyMnKSkKQEAgLTk2Miw3ICs5NzYsNyBAQCBwYXJzZV9hZmZlbnRyeShjaGFyICpz dHIsIGNoYXIgKm1hc2ssIGNoYXIgKmZpbmQsIGNoYXIgKnJlcGwpCiAJCQkJKnBmaW5kID0g J1wwJzsKIAkJCQlzdGF0ZSA9IFBBRV9XQUlUX1JFUEw7CiAJCQl9Ci0JCQllbHNlIGlmICh0 X2lzYWxwaGEoc3RyKSkKKwkJCWVsc2UgaWYgKHRfaXNhbHBoYShzdHIpIHx8IGlzZXNjYXBl ZCkKIAkJCXsKIAkJCQlDT1BZQ0hBUihwZmluZCwgc3RyKTsKIAkJCQlwZmluZCArPSBwZ19t YmxlbihzdHIpOwpAQCAtOTk2LDcgKzEwMTAsNyBAQCBwYXJzZV9hZmZlbnRyeShjaGFyICpz dHIsIGNoYXIgKm1hc2ssIGNoYXIgKmZpbmQsIGNoYXIgKnJlcGwpCiAJCQkJKnByZXBsID0g J1wwJzsKIAkJCQlicmVhazsKIAkJCX0KLQkJCWVsc2UgaWYgKHRfaXNhbHBoYShzdHIpKQor CQkJZWxzZSBpZiAodF9pc2FscGhhKHN0cikgfHwgaXNlc2NhcGVkKQogCQkJewogCQkJCUNP UFlDSEFSKHByZXBsLCBzdHIpOwogCQkJCXByZXBsICs9IHBnX21ibGVuKHN0cik7CkBAIC0x MDA5LDYgKzEwMjMsNyBAQCBwYXJzZV9hZmZlbnRyeShjaGFyICpzdHIsIGNoYXIgKm1hc2ss IGNoYXIgKmZpbmQsIGNoYXIgKnJlcGwpCiAJCWVsc2UKIAkJCWVsb2coRVJST1IsICJ1bnJl Y29nbml6ZWQgc3RhdGUgaW4gcGFyc2VfYWZmZW50cnk6ICVkIiwgc3RhdGUpOwogCisJCWlz ZXNjYXBlZCA9IGZhbHNlOwogCQlzdHIgKz0gcGdfbWJsZW4oc3RyKTsKIAl9CiAKZGlmZiAt LWdpdCBhL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvdHNkaWN0cy5vdXQgYi9zcmMvdGVz dC9yZWdyZXNzL2V4cGVjdGVkL3RzZGljdHMub3V0CmluZGV4IGM4MDQyOTMxNDIuLjVhMDM1 M2Y1NTIgMTAwNjQ0Ci0tLSBhL3NyYy90ZXN0L3JlZ3Jlc3MvZXhwZWN0ZWQvdHNkaWN0cy5v dXQKKysrIGIvc3JjL3Rlc3QvcmVncmVzcy9leHBlY3RlZC90c2RpY3RzLm91dApAQCAtNzEs NiArNzEsMTkgQEAgU0VMRUNUIHRzX2xleGl6ZSgnaXNwZWxsJywgJ3VuYm9vaycpOwogIHti b29rfQogKDEgcm93KQogCistLSBUZXN0IGZvciBoeXBoZW4gZXNjYXBpbmcKK1NFTEVDVCB0 c19sZXhpemUoJ2lzcGVsbCcsICdpbi1sYXcnKTsKKyB0c19sZXhpemUgCistLS0tLS0tLS0t LQorIHtsYXd9CisoMSByb3cpCisKK1NFTEVDVCB0c19sZXhpemUoJ2lzcGVsbCcsICdsYXcn KTsKKyB0c19sZXhpemUgCistLS0tLS0tLS0tLQorIHtsYXd9CisoMSByb3cpCisKIFNFTEVD VCB0c19sZXhpemUoJ2lzcGVsbCcsICdmb290a2x1YmJlcicpOwogICAgdHNfbGV4aXplICAg IAogLS0tLS0tLS0tLS0tLS0tLQpkaWZmIC0tZ2l0IGEvc3JjL3Rlc3QvcmVncmVzcy9zcWwv dHNkaWN0cy5zcWwgYi9zcmMvdGVzdC9yZWdyZXNzL3NxbC90c2RpY3RzLnNxbAppbmRleCBk ZGM2YzdmNDQ1Li5jZjkwMWVjMDhiIDEwMDY0NAotLS0gYS9zcmMvdGVzdC9yZWdyZXNzL3Nx bC90c2RpY3RzLnNxbAorKysgYi9zcmMvdGVzdC9yZWdyZXNzL3NxbC90c2RpY3RzLnNxbApA QCAtMTksNiArMTksMTAgQEAgU0VMRUNUIHRzX2xleGl6ZSgnaXNwZWxsJywgJ3VuYm9va2lu Z3MnKTsKIFNFTEVDVCB0c19sZXhpemUoJ2lzcGVsbCcsICd1bmJvb2tpbmcnKTsKIFNFTEVD VCB0c19sZXhpemUoJ2lzcGVsbCcsICd1bmJvb2snKTsKIAorLS0gVGVzdCBmb3IgaHlwaGVu IGVzY2FwaW5nCitTRUxFQ1QgdHNfbGV4aXplKCdpc3BlbGwnLCAnaW4tbGF3Jyk7CitTRUxF Q1QgdHNfbGV4aXplKCdpc3BlbGwnLCAnbGF3Jyk7CisKIFNFTEVDVCB0c19sZXhpemUoJ2lz cGVsbCcsICdmb290a2x1YmJlcicpOwogU0VMRUNUIHRzX2xleGl6ZSgnaXNwZWxsJywgJ2Zv b3RiYWxsa2x1YmJlcicpOwogU0VMRUNUIHRzX2xleGl6ZSgnaXNwZWxsJywgJ2JhbGx5a2x1 YmJlcicpOwo= --------------5F7D1F4A41F829698D47D893--