Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jVEBP-0002sP-Mk for pgsql-admin@arkaria.postgresql.org; Sun, 03 May 2020 12:57:23 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.92) (envelope-from ) id 1jVEBN-0000yj-Mx for pgsql-admin@arkaria.postgresql.org; Sun, 03 May 2020 12:57:21 +0000 Received: from makus.postgresql.org ([2001:4800:3e1:1::229]) by malur.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jVEBN-0000xA-F5 for pgsql-admin@lists.postgresql.org; Sun, 03 May 2020 12:57:21 +0000 Received: from mout.perfora.net ([74.208.4.196]) by makus.postgresql.org with esmtps (TLS1.3:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.92) (envelope-from ) id 1jVEBK-0007De-Dp for pgsql-admin@postgresql.org; Sun, 03 May 2020 12:57:20 +0000 Received: from [192.168.1.143] ([98.169.208.7]) by mrelay.perfora.net (mreueus003 [74.208.5.2]) with ESMTPSA (Nemesis) id 0LbLJc-1ikIVC3KlG-00kxgQ for ; Sun, 03 May 2020 14:57:16 +0200 To: "pgsql-admin@postgresql.org" From: MichaelDBA Subject: "strange" pg_rewind error Message-ID: <939b25f7-bc1b-ba32-fd3f-afed0f7c0702@sqlexec.com> Date: Sun, 3 May 2020 08:57:13 -0400 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 PostboxApp/7.0.14 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="------------56431B315E809EF239BFE3ED" Content-Language: en-US X-Provags-ID: V03:K1:qkwjRYMKxh2oU1y8ozX7DsXPk/uUpdsQbytRA7Kcf8AXGKyuyXI HGRS6IH1j0rI2uAkQuj7vBB8UkupkjU3gV6jEArmKUKAsxST7jvwmMXF0cvziUQbl4ws1Za QDi6gjgRMtMEMXTeqFgOaV2B+0kyqfUuzH7PN0W5Jw0T70UCt2fzjK/YZsrLphnd7Ccl6EJ 7YZQfeBeB4BG8SmGeg/Hg== X-Spam-Flag: NO X-UI-Out-Filterresults: notjunk:1;V03:K0:UHRgxIC5Cl0=:k5UuV5tHw7Xualm6BzOBbq nCI4jAv8rxQOubDmtBh81u1++Ohaon66H+h7cSZhOFd04Vy4KnMLo7tHCwhNehbPDnsGFCD1+ 5KsnMiSsHSaJYyADr551nxroJZzPiKLGJBM8UEc4YOPZXXY/yb6KczNQYEiNxjTBxFs5qyTN2 D2HB6NTAtIgz3UkChwo01OHfu409ZWd+0LeDXJysh7HGJdEabSexQ9Di9U3b6jgeeV2FtpBZR AgtHNvqHx/n65H5G1tAHUt/gPO4jMwUYCgiVdg46CmVx7P7MfZV0tYrUPtlFZsBNl+RHU0fHR xzoqm9Tt25i9GdfRH51SSTZX0xfg3+2Yg2vfTX6l37R+lhyrkk1513C4snREVqLbBHwIFsb3a DTCjXWJzTyXPqnTaKyxeqWhQscf+wgJ2NundnZTP6PZAc7pvWvKPf8zagXafk3XRhxAjTub96 ksKnE+NkjHnBTFLGKmYH+F0Ecli8PuJuCELyglDtfOPEIczYfDFYaMuqymz97myBwd7zp0Tn9 NPvyh/sJkqgFm53vJTDJnFw878w0w0JIkSDA+jNxukFVcf1xGkwOkI18F4hCPIvJt0vc7K/nu bhooV7icjM/4gVfPozX3AwKmPyFRFlvEfasgw8KNblGtKbFYZf4twzajyvpzQzWjhb2jumpb5 Ut2brmlxkx9hdI54ioJz7XxNyODmUWZDOBSxBtsWScTIftEPrdc/XeVvUAMJVGf6hxlayKb9t UGPdWU66pnH0/n1Y7OzFr1g0LQmBzU+AwFspxcJCpj+Hc8ckZZo/s7/JLGvsZZaSIGHAq2uJs ZAdiICrM+GpG3Phc+AEBFdhrc6a3v6ql3Fc3i7PncI4z3Yx0meVKhZ1kc7WPRXuQWfQC5tf List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk This is a multi-part message in MIME format. --------------56431B315E809EF239BFE3ED Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Here is my pg_rewind command.  Trying to make a former primary follow the newly-promoted standby... /usr/pgsql-12/bin/pg_rewind -D '/apps/opt/postgres/pg12/data' --source-server='host=mypromotedstandby user=repmgr dbname=repmgr port=5432 password=mypassword connect_timeout=2' Now I don't know if this matters but that log directory is actually a symlink pointing to a location that has log files in both the source and target. Here is the error I get: pg_rewind: servers diverged at WAL location 1/2F0000A0 on timeline 6 pg_rewind: rewinding from last common checkpoint at 1/2F000028 on timeline 6 pg_rewind: fatal: "/apps/opt/postgres/pg12/data/log" is not a directory Here is the reference from the PostgreSQL source code where it is noted as "strange" https://github.com/postgres/postgres/blob/master/src/bin/pg_rewind/filemap.c process_source_file(const char *path, file_type_t type, size_t newsize,                     const char *link_target) { ... ... ...     switch (type)     {         case FILE_TYPE_DIRECTORY:             if (exists && !S_ISDIR(statbuf.st_mode) && strcmp(path, "pg_wal") != 0)             {                 /* it's a directory in source, but not in target. Strange.. */                 pg_fatal("\"%s\" is not a directory", localpath);             }             if (!exists)                 action = FILE_ACTION_CREATE;             else                 action = FILE_ACTION_NONE;             oldsize = 0;             break;         case FILE_TYPE_SYMLINK:             if (exists && Regards, Michael Vitale --------------56431B315E809EF239BFE3ED Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit Here is my pg_rewind command.  Trying to make a former primary follow the newly-promoted standby...
/usr/pgsql-12/bin/pg_rewind -D '/apps/opt/postgres/pg12/data' --source-server='host=mypromotedstandby user=repmgr dbname=repmgr port=5432 password=mypassword connect_timeout=2'

Now I don't know if this matters but that log directory is actually a symlink pointing to a location that has log files in both the source and target.

Here is the error I get:
pg_rewind: servers diverged at WAL location 1/2F0000A0 on timeline 6
pg_rewind: rewinding from last common checkpoint at 1/2F000028 on timeline 6
pg_rewind: fatal: "/apps/opt/postgres/pg12/data/log" is not a directory

Here is the reference from the PostgreSQL source code where it is noted as "strange"
https://github.com/postgres/postgres/blob/master/src/bin/pg_rewind/filemap.c
process_source_file(const char *path, file_type_t type, size_t newsize,
                    const char *link_target)
{
...
...
...
    switch (type)
    {
        case FILE_TYPE_DIRECTORY:
            if (exists && !S_ISDIR(statbuf.st_mode) && strcmp(path, "pg_wal") != 0)
            {
                /* it's a directory in source, but not in target. Strange.. */
                pg_fatal("\"%s\" is not a directory", localpath);
            }

            if (!exists)
                action = FILE_ACTION_CREATE;
            else
                action = FILE_ACTION_NONE;
            oldsize = 0;
            break;

        case FILE_TYPE_SYMLINK:
            if (exists &&

Regards,
Michael Vitale


--------------56431B315E809EF239BFE3ED--