Received: from malur.postgresql.org ([217.196.149.56]) by arkaria.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uUJAi-001Aa6-Uw for pgsql-performance@arkaria.postgresql.org; Wed, 25 Jun 2025 05:59:48 +0000 Received: from localhost ([127.0.0.1] helo=malur.postgresql.org) by malur.postgresql.org with esmtp (Exim 4.94.2) (envelope-from ) id 1uUJAh-0002cO-13 for pgsql-performance@arkaria.postgresql.org; Wed, 25 Jun 2025 05:59:47 +0000 Received: from magus.postgresql.org ([2a02:c0:301:0:ffff::29]) by malur.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1uUJAg-0002cG-MM for pgsql-performance@lists.postgresql.org; Wed, 25 Jun 2025 05:59:47 +0000 Received: from mail-wm1-x334.google.com ([2a00:1450:4864:20::334]) by magus.postgresql.org with esmtps (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.96) (envelope-from ) id 1uUJAf-003xhR-0a for pgsql-performance@lists.postgresql.org; Wed, 25 Jun 2025 05:59:47 +0000 Received: by mail-wm1-x334.google.com with SMTP id 5b1f17b1804b1-45348bff79fso66039715e9.2 for ; Tue, 24 Jun 2025 22:59:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cybertec.at; s=google; t=1750831184; x=1751435984; darn=lists.postgresql.org; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:from:to:cc:subject:date :message-id:reply-to; bh=eFnH5QBpgP0AJ6DEqhRfyhkJuTPoT6SDAYiGg/NZvZo=; b=IJSQcf3YvOXRS5XPIUm++ahtNHh7JsQ8wyCBUI7u3KZHEwn84tuGxUoKcP8omc6H9N G5U2MUxG5sIqVtVZ3gzxmhhOEwwiVABip2KzXwGknOlmlvD8OfJDpmtASU9clX8DnKYs jxPqYHa+tK46W9Da0YjBNQDxV8any+4MBg3bt44Hx3PQlepUteUJvsJByv2WmrGPY6iC r9cc0cnaQ5KipxoBsCVVGu0DrtYzjN3y1uRVL9GwIKMH2JVBiGotK/5efKevmk06US/P oS6OHi6/DJy01BLxlLlPjcXOWsTI/NNuugVCT8ZPq/+NVAyaJzUcdwEe8e6XLNeJDdVA TrtQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1750831184; x=1751435984; h=mime-version:user-agent:content-transfer-encoding:references :in-reply-to:date:to:from:subject:message-id:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=eFnH5QBpgP0AJ6DEqhRfyhkJuTPoT6SDAYiGg/NZvZo=; b=Ry0DnffwP1gocw/xyWjfl9V6k5ufeWTbuOttdINKldWdx+nz0JyEZZIoQzKzw96ShO SHktRqq1MJsK3wwpm5Fsob9Tcrj/NZ5WQYVpkGg7ilbAA/F8MRG605i4liONusJuV8Tl 9ITwc8FWnrXQLK9GDJSmZOPTeK5CD6EE1TV6i7YHxJb5GjtIv5FSh9eFLMIwVEdiw4J3 QhbIRV1AxuoztIvM2pcgOZvZvrbOvYaHieu3E5EODOhO/phyKO0wuRzLFbuVD4RKfHLd yDasPWjVXLRfcR6nonixeeeZVlUL1kWsaPYsAJU20MbMzam0/qR39ec+C/oEACoN92MQ OWCA== X-Forwarded-Encrypted: i=1; AJvYcCWSWw9gSKf8guTyBoag2uk8ccuRUDTlAQdEiEF8PWEPElHQrmldGCrVTPzs5nQpJpif9hO2ZBPQSZsp/7JuUbxHqA==@lists.postgresql.org X-Gm-Message-State: AOJu0YwEI3yXDRNZw0h+XiF6XSxHdyWDsfCpm4zcBKvFjyi3l+tdvJKM KLCRc7GgznsNWbRPt/+K5ijjNIaQP3D45IjqJ4x7l4c5GlJ0QGb0SoLd4N8yXK5+jZDLB/xtkYH Vz1Af X-Gm-Gg: ASbGnct1X8eGzc718/V/DZi3gx8JUZ5vhvKMJl6bV7j8WZZZEGFxuNAYIk+7mqIw/DZ cu+jR+b18rlndyMKXhePf46QsXisbtvrrw3AheR4CxHGeDnsyuMrzQpQDu+tVdzUTxXGF/N+hw7 p8QGydxiU8xxwZ5IyWm22Ogki8UJlJsNZFg+LRQ0gSZYqGB2gttQLQpp+2nVn5oM46xn5tKGXcf AtYLTFqFAxDrT3oJV+1JelDkmELpwNxtFSH2BBam51JMT/WqjShIjz5odBpgUG9OqaJEzYR4djd IErx524rIpdTmBDM4NzhL/iTJslKPHQKom5Z+IpitQ0vf1cREPstG5AUz6gahf4= X-Google-Smtp-Source: AGHT+IEIE/T/Hyq+UAzqdoArgcq6jChT9L9b4fyLlTmvf79YLuSGbu6BbZzWwIn8e9tilaixnbGLzQ== X-Received: by 2002:a05:6000:2803:b0:3a5:8934:4940 with SMTP id ffacd0b85a97d-3a6ed651d66mr574481f8f.50.1750831184049; Tue, 24 Jun 2025 22:59:44 -0700 (PDT) Received: from laurenz.albe-K4N0CV00F97414D ([213.208.157.35]) by smtp.gmail.com with ESMTPSA id ffacd0b85a97d-3a6e805dc00sm3757968f8f.31.2025.06.24.22.59.43 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 24 Jun 2025 22:59:43 -0700 (PDT) Message-ID: <66879d8bd44148f2ef1dcde1eff056e6c671306e.camel@cybertec.at> Subject: Re: many sessions waiting DataFileRead and extend From: Laurenz Albe To: James Pang , pgsql-performance@lists.postgresql.org Date: Wed, 25 Jun 2025 07:59:41 +0200 In-Reply-To: References: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable User-Agent: Evolution 3.56.2 (3.56.2-1.fc42) MIME-Version: 1.0 List-Id: List-Help: List-Subscribe: List-Post: List-Owner: List-Archive: Archived-At: Precedence: bulk On Wed, 2025-06-25 at 11:15 +0800, James Pang wrote: > pgv14, RHEL8, xfs , we suddenly see tens of sessions waiting on "DataFile= Read" and > "extend", it last about 2 seconds(based on pg_stat_activity query) , duri= ng the > waiting time, "%sys" cpu increased to 80% , but from "iostat" , no high i= ops and > io read/write latency increased either. Run "sar -P all 1" and see if "%iowait" is high. Check if you have transparent hugepages enabled: cat /sys/kernel/mm/transparent_hugepage/enabled If they are enabled, disable them and see if it makes a difference. I am only guessing here. > many sessions were running same "DELETE FROM xxxx" in parallel waiting on= "extend" > and "DataFileRead", there are triggers in this table "After delete" to in= sert/delete > other tables in the tigger.=C2=A0 One thing that almost certainly would improve your situation is to run fewe= r concurrent statements, for example by using a reasonably sized connection p= ool. Yours, Laurenz Albe