buffer/select problem (modifying libwww)

Darrell Berry (darrell@shrine.cyber.ad.jp)
Sat, 13 May 1995 20:29:50 +1000 

As some of you are aware, I'm in the process of making a few hacks to
libwww for the Mac...

As a preliminary, I'm modifying wwwhttp'request to use a temp file to store
the data currently read straight into $content (currently returned data
must fit in memory, which is rather a limitation)...

I want to insert code something like the following into the "run_it"
section of request(), immediately after the header processing (replacing
$content = <FS>; )

            [...]

            open(TEMP, ">tmp");

            vec($rin, fileno(FS), 1 ) = 1;

            $more = 1;

            while ($more) {

               $n = select($rout = $rin, undef, undef, undef);
               die "select: $!" unless defined $n;


               # so now have input on some select()ed stream

               if (vec($rout, fileno(FS), 1 ) ) {
                   $more     = read(FS, $fbuf, 1024);
                   print TEMP $fbuf;
                   print "$more bytes read on FS\n";
               }

               # check other streams here with similar code

               print "idle here...\n";
            }

            close(TEMP);

            [...]

Obviously this is just a sketch, and not particularly idiomatic at that
(and the $more is just plain nasty), but let me explain the idea...I want
to use select() to check for async io on other data streams at the same
time as downloading the data returned by the http request, and also do the
download in small chunks, so as to allow plenty of idle time for other
things (remember this is on a Mac)...the final buffer size _will_ probably
be larger than 1k...but on a dialup this gets me back at the select() about
once a second, which is a bearable (one of the other streams is doing
console i/o) tradeoff for smallish files...

The problem is that although this sort of works, the resulting temp file is
invariably missing data at the start, as if the first load of stuff in the
buffer is dissappearing. If I use the code above & hardwire a physical file
handle as FS (rather than an already accessed net-stream), the it works
fine (copies the entire file).

But looking at the code immediately preceeding this in request(), it
appears to leave the buffer clean...the headers have been stripped out by
using an explicitly set record delimiter, which is then undefine()'d...so
there shouldn't be a problem (but there is!). I get the same data returned
whether I use the code shown above or a simple succession of reads with the
buffer size set, so I don't _think_ its just something to do with the
select.

Can anyone suggest what I'm doing wrong?

------------------------------------------------------------------------------
 darrell berry              :                   i r i s / e
 darrell@shrine.cyber.ad.jp :  7-3-48-1 shinjuku shinjuku-ku tokyo 160 japan
..............................................................................
              http://shrine.cyber.ad.jp/~darrell/irise.html
------------------------------------------------------------------------------