Re: Slow File Transfer (was Re: Helpppp!

Andreas Koenig (k@anna.in-berlin.de)
Sun, 10 Nov 1996 11:40:44 +0100 

>>>>> Martijn Koster <m.koster@webcrawler.com> writes:

 martijn> At 9:27 PM 11/7/96, Jim Anderson wrote:
 >> I'm about to have to trash an implementation of a file transfer system
 >> that uses LWP. Without going into the rationale for the chosen design,
 >> I have to say that using HTTP/UserAgent to transfer large files is
 >> impractical--memory usage is over 3 times in excess of the file being
 >> transferred,

 martijn> Surely not if you transfer in chunks, or to a file instead of single scalar?

I fear, Jim's right. libwww-5.04 and perl5.005_07 need a closer
examination. The following process:

% GET ftp://ftp.funet.fi/pub/languages/perl/CPAN/src/5.0/unsupported/perl5.003_07.tar.gz > /dev/null

grew to 10 MB before I killed it. IRIX becomes unusable at that
point. I suspect memory is requested in too big chunks and IRIX has no
detection for this kind of nonsense. Which is why I killed the process
long before my memory was exeeded. Here is the size from ps -elf
output in 5 seconds interval:

    SZ:RSS
  3554:3276
  3762:3484
  3932:3653
  5134:4822
  6470:6192
  7974:7432
  9910:9199
 10594:8778

The last 2 lines were not printed after 5 seconds but after about 60,
I suspect because of priority of malloc or some higher rationale.

I have no other machine with both libwww-5.04 and perl5.005_07
unfortunately.

 >> and it's taking almost 2 orders of magnitude longer than
 >> a simple rcp.

 martijn> Huh? 2x maybe, but surely not much more. Reducing memory usage should
 martijn> help speed too.

 >> Any guidelines or suggestions as to what I may be doing wrong? I know
 >> I haven't given a lot of detail, so please let me know what more is
 >> needed.

 martijn> Code (trimmed to only show the problem), hostnames and network layout
 martijn> between them, OS descriptions, Perl configs, example transfer data,
 martijn> tcpdump traces :-)

:-) And first of all, a better subject line, Jim! Read the regular
posting on c.l.p.* about choosing subject lines. Watch what Martijn
did with yours :-)

 martijn> -- Martijn

Regards,
andreas


 martijn> Email: m.koster@webcrawler.com
 martijn> WWW: http://info.webcrawler.com/mak/mak.html