Subject: Re: CPU 100% usage caused by unknown postgres process..
To: Justin Pryzby <pryzby@telsasoft.com>,
 Laurenz Albe <laurenz.albe@cybertec.at>
Cc: Dinesh Chandra 12108 <Dinesh.Chandra@cyient.com>,
 "pgsql-performance@postgresql.org" <pgsql-performance@postgresql.org>
References: <68dd64b2a09f4fd8afe9582a4e23641e@cyient.com>
 <1513161371.5128.3.camel@cybertec.at> <20171213121952.GA16691@telsasoft.com>
From: Tomas Vondra <tomas.vondra@2ndquadrant.com>
Message-ID: <96e72d15-7edf-96bd-69e1-59ef562f4ca8@2ndquadrant.com>
Date: Wed, 13 Dec 2017 16:22:47 +0100
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101
 Thunderbird/52.3.0
MIME-Version: 1.0
In-Reply-To: <20171213121952.GA16691@telsasoft.com>
Content-Type: text/plain; charset=utf-8
Content-Language: en-US
Content-Transfer-Encoding: 7bit
Precedence: bulk


On 12/13/2017 01:19 PM, Justin Pryzby wrote:
> On Wed, Dec 13, 2017 at 11:36:11AM +0100, Laurenz Albe wrote:
>> Dinesh Chandra 12108 wrote:
>>> My CPU utilization is going to 100% in PostgreSQL because of one unknown process /x3303400001 is running from postgres user.
>>>  
>>> PID   USER      PR  NI    VIRT    RES    SHR S %CPU %MEM   TIME+   COMMAND
>>> 19885 postgres 20   0  192684   3916   1420 S 99.3  0.1   5689:04  x3303400001 
>>>  
>>> The same file is automatically created in Postgres Cluster also. I am using Postgresql-9.3.
>>>  
>>> Kindly suggest how can I resolve this issue.
>>
>> I don't know, but the same problem has been reported on Stackoverflow:
>> https://stackoverflow.com/q/46617329/6464308
>>
>> If your queries look similar, then you might indeed be the victim of an attack.
>>
>> Figure out where the function and the executable come from.
>>
>> In case of doubt, disconnect the server from the network.
> 
> Looks suspicious; I would look at (and save) things like these:
> 
> ls -l /proc/19885/exe
> ls -l /proc/19885/fd
> ls -l /proc/19885/cwd
> 
> sudo lsof -n -p 19885
> sudo netstat -anpe |grep 19885
> 
> Stacktrace with gcore/gdb is a good idea.
> Save a copy of your log/postgres logfiles and try to figure out where it came
> from.  Since an attacker seems to control the postgres process, your data may
> have been compromized (leaked or tampered with).
> 

Any details about the x3303400001 file (is it a shell script or some
kind of binary)?

FWIW the queries (listed in the stackoverflow post) are running under
postgres, which I assume is superuser. The backend has full access to
the data directory, of course, so it may create extra files (using
adminpack extension, for example).

If that's the case (and if it's indeed an attack), it either means the
attacker likely already has access to all the data. So presumably
x3303400001 is doing something else at the OS level.

regards

-- 
Tomas Vondra                  http://www.2ndQuadrant.com
PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services