Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uUGbc-000Ygm-5L for pgsql-performance@arkaria.postgresql.org; Wed, 25 Jun 2025 03:15:24 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1uUGba-00Gp0s-85 for pgsql-performance@arkaria.postgresql.org; Wed, 25 Jun 2025 03:15:22 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uUGbZ-00Gp0j-UC for pgsql-performance@lists.postgresql.org; Wed, 25 Jun 2025 03:15:22 +0000 Received: from mail-yw1-x112f.google.com ([2607:f8b0:4864:20::112f]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1uUGbY-003wSQ-24 for pgsql-performance@lists.postgresql.org; Wed, 25 Jun 2025 03:15:22 +0000 Received: by mail-yw1-x112f.google.com with SMTP id 00721157ae682-713fba639f3so11855967b3.1 for ; Tue, 24 Jun 2025 20:15:20 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1750821319; x=1751426119; darn=lists.postgresql.org; h=to:subject:message-id:date:from:mime-version:from:to:cc:subject :date:message-id:reply-to; bh=BwSAMSqqleNaa/HNuGgLpb3E+9Bv9IZ9EftBgZ4LK6s=; b=R6xEgdu6uLAe5qEYh/kD4NHtU8CP4BSOpXFDkKsp0ZJKSDDMuEZ8FSi7O682KkwLv/ 449u4zGPWjSUMUwxxtFw7+AwLG2VpXi0NI3uZ3NKE8TSm3KctxBs1kUkQuEP/VeCYiXn zJJGjU/77IT5pRQqnlWOvJcuQDJKc/t1N0R3FufcbX4std2W3+Uf9he6GkOFA3OvqaLw 3bC3qpFVV73Pyi6bPbnfneZP9mrSgUwnpgHAg4pIDggQ9UiUvMkQY4k1YGAH183rzfB1 pjowZyP/fA78kjD6Lv1vFBQdDqvdg9HeJCIlm1rP/bpzkvlJpDaTgtd1SN/c4JAi1BP2 bUyw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750821319; x=1751426119; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=BwSAMSqqleNaa/HNuGgLpb3E+9Bv9IZ9EftBgZ4LK6s=; b=FMYs03Ayajok9R0M4GbLH6v5kWpaokFJwAgah0/FuE+5NEo1SpPlfeGs8wLVG7eF4s 3OmcNyWg261oaEJ0bOJ+pOQP90XAydaednZflvCvOqdKCygqba1oUH66Z5/sFScj8OD1 nHaU8+boQSTJkD9cXqTCDrtZ9DKrIIDhgONOr8kze9OyPUEv+WqqToCefnZKND87ua3o dRuMd1RA0PiSkS1yJjubo4WEcfOa2XDu6NyMgFG/SjOmLYLqImG9h6d7tJSHdlZIzuGL TqRmAjNNYbaVpwHNDrJrUYovBTjqBxjYhrRgZw++f1RZfsQJ1B+Zlo7pGaeFY5aY1sb7 uGOg== X-Gm-Message-State: AOJu0Yxyb+DVm2kQ8jlGf+50IFMFOIugnMrOgcGUydKEXTNHDXCvIuNH RCBP7nTTxNQPUNE5alHFneNJaei1Gi1jKT8Hd00bPvZl0KfQr4qJJF0y/yG3zMRoXxB57wuYUyP wnyT1xiYbJ4/skvPRfm5SUw/nL070XioxOFdl X-Gm-Gg: ASbGncsVn8JwHr2NxkxRd5pxIjqyxVheZiyy+YCseV7nG8Rrn0gMk2FIP8Kh4TDDQfC hODypmTnHsaCLzZeJ1JOlAOAEXGwoEBmGTvlK0mwzDqeTLWZsNmEoRhw0oz5w/pcUAz3JkjWNb6 rMvX+DTLVzyWYStFUuNlJVMbULUkHMk1rnVhYdkRGjHQ4= X-Google-Smtp-Source: AGHT+IF1KhHWZxq/xcgMc6qvwYuy2yELvTERMjsHwwjtyPvSvsl3uwse1IDU2DXKpTUfJkjdgnit5p7r+YGBI/K+0bc= X-Received: by 2002:a05:690c:3386:b0:714:401:a77 with SMTP id 00721157ae682-71406da0b3fmr20271977b3.20.1750821318947; Tue, 24 Jun 2025 20:15:18 -0700 (PDT) MIME-Version: 1.0 From: James Pang Date: Wed, 25 Jun 2025 11:15:08 +0800 X-Gm-Features: Ac12FXz50gH-fHxaSk8GnhGiCeDqND7Wzpetb4bjCwBh4PbExaltuVjob-LA8aM Message-ID: Subject: many sessions waiting DataFileRead and extend To: pgsql-performance@lists.postgresql.org Content-Type: multipart/alternative; boundary="000000000000c7f87a06385cdadc" List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk --000000000000c7f87a06385cdadc Content-Type: text/plain; charset="UTF-8" pgv14, RHEL8, xfs , we suddenly see tens of sessions waiting on "DataFileRead" and "extend", it last about 2 seconds(based on pg_stat_activity query) , during the waiting time, "%sys" cpu increased to 80% , but from "iostat" , no high iops and io read/write latency increased either. many sessions were running same "DELETE FROM xxxx" in parallel waiting on "extend" and "DataFileRead", there are triggers in this table "After delete" to insert/delete other tables in the tigger. time wait_event count(sessions) 2025-06-24 18:00:11.687245+00 | DataFileFlush | 1 2025-06-24 18:00:11.687245+00 | DataFileWrite | 1 2025-06-24 18:00:11.687245+00 | PgSleep | 1 2025-06-24 18:00:11.687245+00 | DataFileExtend | 7 2025-06-24 18:00:11.687245+00 | ProcArrayGroupUpdate | 1 2025-06-24 18:00:11.687245+00 | WALWrite | 5 2025-06-24 18:00:11.687245+00 | extend | 55 2025-06-24 18:00:11.687245+00 | DataFileRead | 45 2025-06-24 18:00:11.687245+00 | DataFilePrefetch | 2 Thanks, James --000000000000c7f87a06385cdadc Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
=C2=A0 =C2=A0pgv14, RHEL8, xfs , we suddenly see tens of s= essions waiting on "DataFileRead" and "extend", it last= about 2 seconds(based on pg_stat_activity query) , during the waiting time= , "%sys" cpu increased to 80% , but from "iostat" , no = high iops and io read/write latency increased either.=C2=A0=C2=A0
=C2= =A0 =C2=A0many sessions were running same "DELETE FROM xxxx" in p= arallel waiting on "extend" and "DataFileRead", there a= re triggers in this table "After delete" to insert/delete other t= ables in the tigger.=C2=A0

time=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0wait_event=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 count(= sessions)
=C2=A02025-06-24 18:00:11.687245+00 | DataFileFlush=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 1
=C2=A02025-06-24= 18:00:11.687245+00 | DataFileWrite=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 | =C2=A0 =C2=A0 1
=C2=A02025-06-24 18:00:11.687245+00 | PgSleep=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0= =C2=A0 1
=C2=A02025-06-24 18:00:11.687245+00 | DataFileExtend=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 7
=C2=A02025-06-24 18:00:11.687= 245+00 | ProcArrayGroupUpdate | =C2=A0 =C2=A0 1
=C2=A02025-06-24 18:00:1= 1.687245+00 | WALWrite=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 | =C2=A0 =C2=A0 5
=C2=A02025-06-24 18:00:11.687245+00 = | extend=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A055
=C2=A02025-06-24 18:00:11.6872= 45+00 | DataFileRead=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 | =C2= =A0 =C2=A045
=C2=A02025-06-24 18:00:11.687245+00 | DataFilePrefetch=C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0| =C2=A0 =C2=A0 2
=C2=A0
=
Thanks,

James=C2=A0
--000000000000c7f87a06385cdadc--