Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h9tBe-00010h-Hz for pgsql-hackers@arkaria.postgresql.org; Fri, 29 Mar 2019 15:12:54 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.89) (envelope-from ) id 1h9tBc-0008J8-VD for pgsql-hackers@arkaria.postgresql.org; Fri, 29 Mar 2019 15:12:52 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h9tBc-0008Iy-H8 for pgsql-hackers@lists.postgresql.org; Fri, 29 Mar 2019 15:12:52 +0000 Received: from mail-qt1-x843.google.com ([2607:f8b0:4864:20::843]) by magus.postgresql.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_CBC_SHA1:256) (Exim 4.89) (envelope-from ) id 1h9tBZ-0001rY-EN for pgsql-hackers@lists.postgresql.org; Fri, 29 Mar 2019 15:12:52 +0000 Received: by mail-qt1-x843.google.com with SMTP id t28so2947721qte.6 for ; Fri, 29 Mar 2019 08:12:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=2ndquadrant-com.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:mime-version:content-disposition :content-transfer-encoding:in-reply-to:user-agent; bh=azRmMBf3X8p/kdPWp3jW1BKrGRhMH4L4UhdwukuB02Q=; b=t1J2L6Sk522ivgbc+zhBPfur7vwVmObGYep6aaxBP/RqHk5m5lkgcehU5hSFx1B7O0 vvmxPw4ENBmxDZ7pdUOq6BZYaYrktGcvX1WVze2N0Kah+oGuYkQYgn/Yzjwyw7UqisCl /Ll+JXnqr+0l+D11r+Nv8Sc1mIiO2GxGSN9PZq1P5OnxF1IPD2ZXy2PeN2NJvkgkZwqw QIdDvuoxWRm4neo91flDbBHrTjsmCus1lPJQ9b9MmhvPYq2xvN8uTSRkAVJ79SnG/MsY 30U4aZRO6AH4xP5DpOfANluTq6ge60+5nFcmB3tZ35WfYkp/GxudE6cnQvJWhttWWi50 Oudw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:mime-version :content-disposition:content-transfer-encoding:in-reply-to :user-agent; bh=azRmMBf3X8p/kdPWp3jW1BKrGRhMH4L4UhdwukuB02Q=; b=LKlarSpWVer3eM87jHDFca13mv2HoPXyGbHaYDsMOfraWRU6qd4Lh0mIvtOSpodBTs USwUjJyabkLdeoLUku4nWnY5ot3I9iwGOvPKDBhEHPijiGsVRTnmShwg84np+Fz8qBGz cRKOlIKqDkptYQf4mU7Q3bpFJOg6AtgKF3Pkd3cb1xcH17LQF2by8PCkgalzr7XXR/lb DnALMg72/tf6KSXzNK9BLb9lDzH9c+CyCTu3esVBRRCJnSR5YwfbsxOVfcodUmrIx4YQ omKJwkHCUrhfw8PO2S8vd0KnEIm1AQSneg5JDD+K+Jz4KtedKJRCYS3wcCK52ts2nkTT aRvA== X-Gm-Message-State: APjAAAWNUI/Yz9rRciiUCnkEvGOvKmrQeYY1RlPoeI0d3urII1eJUOR6 i4q09abmGkcH/7BreOpinJyYfg== X-Google-Smtp-Source: APXvYqwDcursjvHED2ItVWAR2ZFYvlA19ARsyZMnPCInqybYVzOq/uazAySMCFm1DAOzoWrnUoA8QQ== X-Received: by 2002:a0c:94d0:: with SMTP id k16mr42005626qvk.158.1553872367408; Fri, 29 Mar 2019 08:12:47 -0700 (PDT) Received: from nimloth.alvh.no-ip.org ([190.121.29.3]) by smtp.gmail.com with ESMTPSA id s43sm1415913qth.47.2019.03.29.08.12.46 (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 29 Mar 2019 08:12:46 -0700 (PDT) Received: by nimloth.alvh.no-ip.org (Postfix, from userid 1000) id 656DB120A6D; Fri, 29 Mar 2019 12:12:44 -0300 (-03) Date: Fri, 29 Mar 2019 12:12:44 -0300 From: Alvaro Herrera To: Robert Haas Cc: Simon Riggs , Pg Hackers , David Fetter , Rahila Syed , Pavan Deolasee , Amit Langote Subject: Re: monitoring CREATE INDEX [CONCURRENTLY] Message-ID: <20190329151244.GA28626@alvherre.pgsql> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20190329150218.GA25010@alvherre.pgsql> User-Agent: Mutt/1.9.4 (2018-02-28) List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Precedence: bulk On 2019-Mar-29, Alvaro Herrera wrote: > I just noticed that the CLUSTER calls index_build, which my patch > modifies to include additional progress metrics; this means that during > the index build phase, the metrics set by CLUSTER will be trashed by the > ones my patch introduces. Indeed: pid | datid | datname | relid | command | phase | cluster_index_relid | heap_tuples_scanned | heap_tuples_written | heap_blks_total | heap_blks_scanned | index_rebuild_count 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 162402 | 162402 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 460362 | 460362 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 754004 | 754004 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1047058 | 1047058 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1356296 | 1356296 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1645321 | 1645321 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 1939920 | 1939920 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 2227450 | 2227450 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 2526116 | 2526116 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 2828468 | 2828468 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 3142982 | 3142982 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 3451494 | 3451494 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 3769799 | 3769799 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 4077513 | 4077513 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 4383255 | 4383255 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 4700286 | 4700286 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5015468 | 5015468 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5324951 | 5324951 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5628172 | 5628172 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 5940862 | 5940862 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 6253778 | 6253778 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 6560474 | 6560474 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 6881248 | 6881248 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 7186555 | 7186555 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 7503888 | 7503888 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 7821706 | 7821706 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 8137334 | 8137334 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 8453280 | 8453280 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 8764097 | 8764097 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 9079908 | 9079908 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 9389465 | 9389465 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 9706288 | 9706288 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10024875 | 10024875 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10344875 | 10344875 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10658531 | 10658531 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 10974097 | 10974097 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 11289868 | 11289868 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 11602965 | 11602965 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 11917624 | 11917624 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 12237998 | 12237998 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 12559406 | 12559406 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 12876413 | 12876413 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 13192793 | 13192793 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 13506188 | 13506188 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 13822722 | 13822722 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 14138624 | 14138624 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 14432335 | 14432335 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 16387 | 14753617 | 14753617 | 0 | 0 | 0 --> pid | datid | datname | relid | command | phase | cluster_index_relid | heap_tuples_scanned | heap_tuples_written | heap_blks_total | heap_blks_scanned | index_rebuild_count 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 743 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 4639 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 8938 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 13457 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 17806 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 22237 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 26594 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 30935 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 35362 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 39705 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 44126 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 48546 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 52966 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 57174 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 61536 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 65868 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 2 | 15000000 | 15000000 | 0 | 66372 | 66193 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 28457 | 12750 | alvherre | 16384 | CLUSTER | index scanning heap | 5 | 15000000 | 15000000 | 0 | 0 | 0 I suppose I can just pick non-overlapping numbers for those columns. -- Álvaro Herrera https://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services