public inbox for [email protected]
help / color / mirror / Atom feedRe: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
8+ messages / 3 participants
[nested] [flat]
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
@ 2006-03-26 19:13 Magnus Hagander <[email protected]>
2006-03-26 22:15 ` Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Marc G. Fournier <[email protected]>
2006-03-27 01:43 ` Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Tom Lane <[email protected]>
0 siblings, 2 replies; 8+ messages in thread
From: Magnus Hagander @ 2006-03-26 19:13 UTC (permalink / raw)
To: Marc G. Fournier <[email protected]>; +Cc: pgsql-www
[moved to -www]
> Does anyone here know how to use the latest version of
> Bittorrent, from a server vs client, perspective? The one we
> are currently running is *ancient*, and I'd like to get that
> vServer upgraded and off the templates, but without someone
> that knows how to setup the newer bittorrent, well, that's
> just a scary proposition ;(
Ancient is a bit strong - it's about a year old. By that definition,
there's a whole lot of things that are a lot worse off on our servers.
That said, it's not good. But the fact that something changed on the box
that caused it to stop working in the past couple of days can't be
because of the version, since we've been running that version for a
year.
Moving on, some figures: In the active log now, we have about 4 million
FTP downloads and 190,000 bittorrent downloads. Meaning bittorrent is
only ~ 5% of our downloads. OTOH 190,000 downloads since Jan 11th, 2005
is still more than 400 downloads per day...
But the bottom line of that reasoning is, how much time do we want to
spend hacking around the bittorrent stuff? We're better off now than
before we had an automated system, but is it really something the end
user wants. (For example, we don't know how many of those hits are hits
by search engines and mirroring scripts - that could easily be a couple
of hundreds aday..)
As for migrating it to a differnt vserver - just make sure you have a
"close enough" version of python, and it should be no problem to just
tar up the whole directory and move it over. I wouldn't be particularly
worried about that part.
//Magnus
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
2006-03-26 19:13 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
@ 2006-03-26 22:15 ` Marc G. Fournier <[email protected]>
1 sibling, 0 replies; 8+ messages in thread
From: Marc G. Fournier @ 2006-03-26 22:15 UTC (permalink / raw)
To: Magnus Hagander <[email protected]>; +Cc: pgsql-www
On Sun, 26 Mar 2006, Magnus Hagander wrote:
> [moved to -www]
>
>> Does anyone here know how to use the latest version of
>> Bittorrent, from a server vs client, perspective? The one we
>> are currently running is *ancient*, and I'd like to get that
>> vServer upgraded and off the templates, but without someone
>> that knows how to setup the newer bittorrent, well, that's
>> just a scary proposition ;(
>
> Ancient is a bit strong - it's about a year old. By that definition,
> there's a whole lot of things that are a lot worse off on our servers.
>
> That said, it's not good. But the fact that something changed on the box
> that caused it to stop working in the past couple of days can't be
> because of the version, since we've been running that version for a
> year.
'k, I looked at the server based on your report, and everything looks fine
... so, can you let me know how to reproduce the problem you are seeing?
> As for migrating it to a differnt vserver - just make sure you have a
> "close enough" version of python, and it should be no problem to just
> tar up the whole directory and move it over. I wouldn't be particularly
> worried about that part.
tar up what directory? its the bittorrent binaries that I'm concerned
about ... at some point, they merged all the various "utilities" into one
'bittorrent' script ... there is no longer a 'btlaunchmany.py' script, for
instance ...
And not sure what you are talking about concerning 'a year', but
bittorrent was originally installed back in Nov of '03, according to the
file stamps:
# ls -lt bt*
-rw-r--r-- 1 root wheel 5833 Nov 24 2003 btmakemetafile.pyc
-rw-r--r-- 1 root wheel 6385 Nov 15 2003 btdownloadheadless.pyc
----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: [email protected] Yahoo!: yscrappy ICQ: 7615664
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
2006-03-26 19:13 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
@ 2006-03-27 01:43 ` Tom Lane <[email protected]>
1 sibling, 0 replies; 8+ messages in thread
From: Tom Lane @ 2006-03-27 01:43 UTC (permalink / raw)
To: Magnus Hagander <[email protected]>; +Cc: pgsql-www
"Magnus Hagander" <[email protected]> writes:
> Moving on, some figures: In the active log now, we have about 4 million
> FTP downloads and 190,000 bittorrent downloads.
*Wow*, that seems like a lotta downloads ... what interval is this
measured over?
regards, tom lane
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
@ 2006-03-27 07:15 Magnus Hagander <[email protected]>
2006-03-27 19:13 ` Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Marc G. Fournier <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Magnus Hagander @ 2006-03-27 07:15 UTC (permalink / raw)
To: Marc G. Fournier <[email protected]>; +Cc: pgsql-www
> > [moved to -www]
> >
> >> Does anyone here know how to use the latest version of Bittorrent,
> >> from a server vs client, perspective? The one we are currently
> >> running is *ancient*, and I'd like to get that vServer
> upgraded and
> >> off the templates, but without someone that knows how to setup the
> >> newer bittorrent, well, that's just a scary proposition ;(
> >
> > Ancient is a bit strong - it's about a year old. By that
> definition,
> > there's a whole lot of things that are a lot worse off on
> our servers.
> >
> > That said, it's not good. But the fact that something
> changed on the
> > box that caused it to stop working in the past couple of
> days can't be
> > because of the version, since we've been running that version for a
> > year.
>
> 'k, I looked at the server based on your report, and
> everything looks fine ... so, can you let me know how to
> reproduce the problem you are seeing?
It's automatically reproduced every 10 minutes. There are two
possibilities:
1) The python processes has started crashing. I don't think so, because
the count keeps going up.
2) They are no longer reported in a way that Proc::ProcessTable can
read. See the code at
http://gborg.postgresql.org/cgi-bin/cvsweb.cgi/portal/tools/bt/updatetor
rents.pl?rev=1.8;cvsroot=pgweb for what we're trying to do. It is
*possible* this is done because the processes are swapped out, though I
*think* I tested the code and it handled that.
> > As for migrating it to a differnt vserver - just make sure
> you have a
> > "close enough" version of python, and it should be no
> problem to just
> > tar up the whole directory and move it over. I wouldn't be
> > particularly worried about that part.
>
> tar up what directory? its the bittorrent binaries that I'm
> concerned about ... at some point, they merged all the
> various "utilities" into one 'bittorrent' script ... there is
> no longer a 'btlaunchmany.py' script, for instance ...
Tar up /usr/local/www/bt.postgresql.org.
There are no "bittorrent binaries", they are all python scripts. And
they live under taht directory along with all our utility scripts.
> And not sure what you are talking about concerning 'a year',
> but bittorrent was originally installed back in Nov of '03,
> according to the file stamps:
>
> # ls -lt bt*
> -rw-r--r-- 1 root wheel 5833 Nov 24 2003 btmakemetafile.pyc
> -rw-r--r-- 1 root wheel 6385 Nov 15 2003 btdownloadheadless.pyc
Where are you looking? That's before I started doing the web stuff, and
I'm definitly the one who put in what's tyhere now.
svr4# pwd
/usr/local/www/bt.postgresql.org/auto/bt
svr4# ls -l
total 154
drwxr-xr-x 4 bt robot 1024 Mar 24 2005 BitTorrent-4.0.1
-rw-r--r-- 1 bt wheel 154427 Mar 25 2005 BitTorrent-4.0.1.tar.gz
lrwxr-xr-x 1 root wheel 16 Apr 11 2005 bt -> BitTorrent-4.0.1
(Meaning I may be responsible for the weird groupid usded there :-P)
//Magnus
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
2006-03-27 07:15 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
@ 2006-03-27 19:13 ` Marc G. Fournier <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Marc G. Fournier @ 2006-03-27 19:13 UTC (permalink / raw)
To: Magnus Hagander <[email protected]>; +Cc: pgsql-www
On Mon, 27 Mar 2006, Magnus Hagander wrote:
> It's automatically reproduced every 10 minutes. There are two
> possibilities:
'k, I'm seeing processes running since 2pm Sunday ... so "every 10
minutes", what exactly is happening?
# ps aux | grep btlaunchmany | grep Sun02 | wc -l
36
> 2) They are no longer reported in a way that Proc::ProcessTable can
> read. See the code at
> http://gborg.postgresql.org/cgi-bin/cvsweb.cgi/portal/tools/bt/updatetor
> rents.pl?rev=1.8;cvsroot=pgweb for what we're trying to do. It is
> *possible* this is done because the processes are swapped out, though I
> *think* I tested the code and it handled that.
'k, first ... "processes are swapped out" ... unless something really odd
is happening, I've never heard of a process swapped out that is removed
from the process table ... at least not under FreeBSD ...
second ... have you manually verified that the process(es) *aren't*
running, or just relying on that script? As i stated above, I see 36
processes running for the past 24 hrs +, and a bunch for a shorter period
...
> Tar up /usr/local/www/bt.postgresql.org.
> There are no "bittorrent binaries", they are all python scripts. And
> they live under taht directory along with all our utility scripts.
Ah, okay ... I thought that we were using bittorrent as was installed ages
ago, not a self-maintained version ... but, if that is the case, then I'm
going to get rid of the template stuff over there sometime this week,
since that will be simple enough to do ... python 2.4.2 is auto-installed,
so no problems there ...
If you can be online tomorrow afternoon ... ? that way, when I bring it
back up, you are able to look over it for any issues ...
But, it doesn't look like bittorrent is reaping child processes:
# ps aux | grep "(python)" | wc -l
166
out of:
# ps aux | grep python | wc -l
201
> Where are you looking? That's before I started doing the web stuff, and
> I'm definitly the one who put in what's tyhere now.
/usr/local/bin ...
----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: [email protected] Yahoo!: yscrappy ICQ: 7615664
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
@ 2006-03-27 08:13 Magnus Hagander <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Magnus Hagander @ 2006-03-27 08:13 UTC (permalink / raw)
To: Tom Lane <[email protected]>; +Cc: pgsql-www
> > Moving on, some figures: In the active log now, we have about 4
> > million FTP downloads and 190,000 bittorrent downloads.
>
> *Wow*, that seems like a lotta downloads ... what interval is
> this measured over?
Downloads since Jan 11th, 2005. So ~14 months.
(Note that this of course only counts the downloads that go through our
website. Ppl going directly to ftp.xx.postgresql.org or another local
mirror are not included)
//Magnus
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
@ 2006-03-27 19:48 Magnus Hagander <[email protected]>
2006-03-27 20:17 ` Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Marc G. Fournier <[email protected]>
0 siblings, 1 reply; 8+ messages in thread
From: Magnus Hagander @ 2006-03-27 19:48 UTC (permalink / raw)
To: Marc G. Fournier <[email protected]>; +Cc: pgsql-www
> > It's automatically reproduced every 10 minutes. There are two
> > possibilities:
>
> 'k, I'm seeing processes running since 2pm Sunday ... so
> "every 10 minutes", what exactly is happening?
>
> # ps aux | grep btlaunchmany | grep Sun02 | wc -l
> 36
Yeah. That's a reasonably normal amount of timing.
What happens is that the script detects a lot of btlaunchmany that are
suddenly gone. Either they are still there, and not showing up, or they
are dead. It could well be that they've crashed.
BTW, I've disabled the cronjob now since it sends an email to the slaves
list every 10 minutes :-) So right now, nothing happens.
> > 2) They are no longer reported in a way that Proc::ProcessTable can
> > read. See the code at
> >
> http://gborg.postgresql.org/cgi-bin/cvsweb.cgi/portal/tools/bt/updatet
> > or rents.pl?rev=1.8;cvsroot=pgweb for what we're trying to do. It is
> > *possible* this is done because the processes are swapped
> out, though
> > I
> > *think* I tested the code and it handled that.
>
> 'k, first ... "processes are swapped out" ... unless
> something really odd is happening, I've never heard of a
> process swapped out that is removed from the process table
> ... at least not under FreeBSD ...
Nope, that would be really weird. But it could well be a bug in the perl
library Proc::ProcessTable as well.
> second ... have you manually verified that the process(es)
> *aren't* running, or just relying on that script? As i
> stated above, I see 36 processes running for the past 24 hrs
> +, and a bunch for a shorter period ...
I did that. But it's a bit hard to catch it at the right moment. I
*think* I've seen both.
> > Tar up /usr/local/www/bt.postgresql.org.
> > There are no "bittorrent binaries", they are all python
> scripts. And
> > they live under taht directory along with all our utility scripts.
>
> Ah, okay ... I thought that we were using bittorrent as was
> installed ages ago, not a self-maintained version ... but, if
> that is the case, then I'm going to get rid of the template
> stuff over there sometime this week, since that will be
> simple enough to do ... python 2.4.2 is auto-installed, so no
> problems there ...
Ok. good.
> If you can be online tomorrow afternoon ... ? that way, when
> I bring it back up, you are able to look over it for any issues ...
Afternoon your time or mine ;-)
I'll be online most of the day CET (~GMT), and I can also be online in
the evening if you need it.
> But, it doesn't look like bittorrent is reaping child processes:
>
> # ps aux | grep "(python)" | wc -l
> 166
>
> out of:
>
> # ps aux | grep python | wc -l
> 201
Oh. Could *that* be where the problem comes from? Ithought the ()
indicated they were swapped out, I thin kit does on linux :-)
> > Where are you looking? That's before I started doing the web stuff,
> > and I'm definitly the one who put in what's tyhere now.
>
> /usr/local/bin ...
Oh :-) Didn't even know it was there.
//Magnus
^ permalink raw reply [nested|flat] 8+ messages in thread
* Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync )
2006-03-27 19:48 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
@ 2006-03-27 20:17 ` Marc G. Fournier <[email protected]>
0 siblings, 0 replies; 8+ messages in thread
From: Marc G. Fournier @ 2006-03-27 20:17 UTC (permalink / raw)
To: Magnus Hagander <[email protected]>; +Cc: pgsql-www
On Mon, 27 Mar 2006, Magnus Hagander wrote:
>>> It's automatically reproduced every 10 minutes. There are two
>>> possibilities:
>>
>> 'k, I'm seeing processes running since 2pm Sunday ... so
>> "every 10 minutes", what exactly is happening?
>>
>> # ps aux | grep btlaunchmany | grep Sun02 | wc -l
>> 36
>
> Yeah. That's a reasonably normal amount of timing.
> What happens is that the script detects a lot of btlaunchmany that are
> suddenly gone. Either they are still there, and not showing up, or they
> are dead. It could well be that they've crashed.
>
> BTW, I've disabled the cronjob now since it sends an email to the slaves
> list every 10 minutes :-) So right now, nothing happens.
>
>
>>> 2) They are no longer reported in a way that Proc::ProcessTable can
>>> read. See the code at
>>>
>> http://gborg.postgresql.org/cgi-bin/cvsweb.cgi/portal/tools/bt/updatet
>>> or rents.pl?rev=1.8;cvsroot=pgweb for what we're trying to do. It is
>>> *possible* this is done because the processes are swapped
>> out, though
>>> I
>>> *think* I tested the code and it handled that.
>>
>> 'k, first ... "processes are swapped out" ... unless
>> something really odd is happening, I've never heard of a
>> process swapped out that is removed from the process table
>> ... at least not under FreeBSD ...
>
> Nope, that would be really weird. But it could well be a bug in the perl
> library Proc::ProcessTable as well.
Or, it could be not listing 'non-reaped (ie. Zombie)' processes ... but,
the ps listing doesn't seem to be marking them as Zombie ...
>> If you can be online tomorrow afternoon ... ? that way, when
>> I bring it back up, you are able to look over it for any issues ...
>
> Afternoon your time or mine ;-)
I *think* we are 4 hours off ... always makes it fun to scheduale, I don't
want to make it too late for you, but not too early for me :)
How about 6pm-ish your time, which I htink is around 2pm mine?
> Oh. Could *that* be where the problem comes from? Ithought the ()
> indicated they were swapped out, I thin kit does on linux :-)
It could be ... only place I've ever noticed this is on a system where the
child processes weren't being properly killed off though ...
----
Marc G. Fournier Hub.Org Networking Services (http://www.hub.org)
Email: [email protected] Yahoo!: yscrappy ICQ: 7615664
^ permalink raw reply [nested|flat] 8+ messages in thread
end of thread, other threads:[~2006-03-27 20:17 UTC | newest]
Thread overview: 8+ messages (download: mbox mbox.gz follow: Atom feed)
-- links below jump to the message on this page --
2006-03-26 19:13 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
2006-03-26 22:15 ` Marc G. Fournier <[email protected]>
2006-03-27 01:43 ` Tom Lane <[email protected]>
2006-03-27 07:15 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
2006-03-27 19:13 ` Marc G. Fournier <[email protected]>
2006-03-27 08:13 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
2006-03-27 19:48 Re: Bittorrent (Was: RE: [HACKERS] [SUGGESTION] CVSync ) Magnus Hagander <[email protected]>
2006-03-27 20:17 ` Marc G. Fournier <[email protected]>
This inbox is served by agora; see mirroring instructions
for how to clone and mirror all data and code used for this inbox