LWP / IO hangup through a firewall

Bob Reveal (breveal@c2o.com)
Tue, 01 Dec 1998 16:20:45 +0000 

I have run across a peculiar problem performing a request through a
firewall using LWP.  Here is the situation, when performing the test
from a machine outside of the firewall the http request takes on average
~5 seconds.  When performing the request from a machine inside the
firewall, the request takes ~.500 milliseconds.

Here is my environment:  agent machine (w/LWP) -> firewall (sun solaris
w/checkpoint) -> web server (url is http://www.gmacfs.com/index.htm)

Here are some of the tests I have tried:

- I have tried moving the file to different web servers sitting in front
of and behind the firewall.  The tests produced similar results for both
those machines in front of and behind the firewall (i.e. behind the
firewall took several seconds).
- I have tried different http get libraries from the various servers and
they do not produce similar results (i.e. they are all around the .500
millisecond range).
- I have tried placing files of similar size on the web server and
performed tests from the machine outside of the firewall and did not see
similar results.

After coming to the conclusion that the problem might be local to LWP or
my perl installation, I have determined the problem to occur within the
IO module.  Here is the call chain :

LWP::Protocol->collect

  statement :        while ($content = &$collector, length $$content) {

LWP::Protocol::http->request

  statement :        die "read timeout" if $timeout &&
!$sel->can_read($timeout);

IO::Select->can_read

  statement :         defined($r) && (select($r,undef,undef,$timeout) >
0) ? handles($vec, $r) : ();


The system intermittently hangs for 1-3 seconds on the above select
statement when run from the machine outside of the firewall.


It is almost as if the code has a timing or sequence problem in dealing
with responses from the firewall.  Has anyone run into a similar
problem?  I have included my code below:



#!/usr/local/bin/perl5

use LWP::UserAgent;
use HTTP::Request;
use HTTP::Response;
use HTTP::Message;
use HTTP::Status;
use Time::HiRes qw(gettimeofday tv_interval);

$COUNT = 50;
$total = 0;

### EXAMPLE OF GET

foreach (1 .. $COUNT) {

print ("ATTEMPTING GET REQUEST ...\n");
$ua = new LWP::UserAgent;
$ua->timeout(10);
$StartTime      = [gettimeofday];

$myURL = "http://www.gmacfs.com/index.htm";
$req = new HTTP::Request('GET', $myURL);
$res = $ua->request($req);
if ($res->is_success) { print ("SUCCESS\n"); } else { print ("ERROR\n");
}

$EndTime      = [gettimeofday];
$ResponseTime = tv_interval ($StartTime, $EndTime);
print ("response time is [$ResponseTime]\n\n\n");

$total = $total + $ResponseTime;

}

print ("TOTAL    TIME IS [$total]\n");
$average = $total / $COUNT;
print ("AVERAGE  TIME IS [$average]\n");