LWP::UserAgent redirect with POST

Kit (hykit@jps.net)
Tue, 10 Aug 1999 00:41:30 -0800 

Hi Nathan Good,

I saw Nathan Good's post on the libwww-perl mailing list archive regarding
problems with LWP::UserAgent handling POST redirects and cookies.  I am
wondering if anyone has found out a solution for it yet, cause I am running
into the same problems.  I use LWP::UserAgent to send a POST request to
www.reel.com and didn't get the results I wanted.  Any help would be
appreciated.


Here is Nathan Good's post just to remind you:

Hi.  I have found myself in a situation where I need cookies and am getting
the redirect error and it doesn't look like the UserAgent is sending the
cookies back to the server. The story is that I would like to suck
(responsibly of course) a bunch of movie information from
http://www.reel.com for my research project. Here is what I know:
1)You must have cookies enabled to visit Reel.com and do anything useful
(search moveis etc)
2)When I go to reel.com in my browser and enter in a title, say "revenge" in
the quick search column on the left, the following string shows up in the
location window of my browser:
http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=revenge
&TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICE
RANGE=0&PRODUCTTYPE=0
3)If I go to this string w/o going to http://www.reel.com first (or if I
disable cookies) then I get a page that says "Sorry nothing found" where
there should be results.
4)if I have cookies enabled in the browser and I go to reel, preform a
search "revenge" I get back the same string in the location as above and I
get a list of movies with the title "revenge".

Here's what I've tried in LWP:
1)simulate a browser session; get "Reel.com", then POST.
2)same as above with a GET after the POST
Result is 302 Redirect error from POST
So then I subclassed UserAgent and changed the redirect_ok method to all
ways spit out true:
package MyUserAgent;
require LWP::UserAgent;
@ISA =qw(LWP::UserAgent);
sub redirect_ok{
 return TRUE;
}

3)After this I repeated the above tests.
Result: No more redirect error. kept getting the page saying that there were
no results found, the same as when cookies were disabled in the browser.
4)Tried a single POST Test.
Result: kept getting the page saying that there were no results found.
5)tried various combinations of post strings in
$req->content();
exp:
$req->content('OBJECT=searchresults&STRING=revenge&TYPE=TITLE&STORE=ALL&MODE
=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICERANGE=0&PRODUCTTYPE=0');
and
$req->content(STRING=revenge&TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD
=0&RATING=0&AVAIL=BOTH&PRICERANGE=0&PRODUCTTYPE=0);
Result: Same 'results not found' page.

I am new to the user agent interface (I was just using LWP Simple before) so
I might have made some errors there. I think the problem has something to do
with my mishandling of the cookie-jar, as it seems that reel.com is really
not getting my cookies. Anyway, I've included the errors I get and the code
I wrote.

Code I wrote:
#!/project/gl/install/bin/perl

use lib "/project/gl2/nathan/lib";
#after subclassing I used also included this
#require "/project/gl2/nathan/lib/LWP/MyUserAgent.pm";

use LWP::UserAgent;

       $ua = new LWP::UserAgent;
 #after subclassing I used this:
 #$ua= new MyUserAgent;

$ua->agent("Mozilla/4.0");

#for cookies
$ua->cookie_jar($cookies);

#Go to Reel.com and get a cookie
 my $req = new HTTP::Request 'GET','http://www.reel.com';
       $req->content_type('application/x-www-form-urlencoded');
       my $res = $ua->request($req);
if ($res->is_success) {
   print "Was able to go to reel.com\n";
 } else {
   print "Failed. Didn't make it\n";
}

$r = $res;

print "Code: ", $r->code, "\n";
print "Message: ", $r->message, "\n";
print "Request: ", $r->request, "\n";
print "Base: ", $r->base, "\n";

print "###################Basic URL\n";
#print "As_string: ", $r->as_string, "\n";


#Now try to post to Reel.com
#When using MyUserAgent got back the the 'No search results page'

       my $req = new HTTP::Request
'POST','http://www.reel.com/cgi-bin/search/nph-movie.exe';
       $req->content_type('application/x-www-form-urlencoded');
       $req->content('STRING=revenge&TYPE=TITLE');
       my $res = $ua->request($req);

if ($res->is_success) {
   print "OK Was Able to Post\n";
 } else {
   print "Failed. Posting did not succeed\n";
}

$r = $res;

print "Code: ", $r->code, "\n";
print "Message: ", $r->message, "\n";
print "Request: ", $r->request, "\n";
print "Base: ", $r->base, "\n";
print "#################################### POST URL\n";
print "As_string: ", $r->as_string, "\n";


#Now try to get things from reel.com
#I didn't use this all the time

 my $req = new HTTP::Request
'GET','http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=
revenge&TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOT
H&PRICERANGE=0&PRODUCTTYPE=0';
       $req->content_type('application/x-www-form-urlencoded');
       my $res = $ua->request($req);
if ($res->is_success) {
   print "OK Was Able to grab the page\n";
 } else {
   print "Failed. Grabbing did not succeed\n";
}

$r = $res;

print "Code: ", $r->code, "\n";
print "Message: ", $r->message, "\n";
print "Request: ", $r->request, "\n";
print "Base: ", $r->base, "\n";
print "#################################### The Actual URL\n";
print "As_string: ", $r->as_string, "\n";

<end of code>

This is a list of my results before subclassing UserAgent.pm
####################################
MY Results:

Was able to go to reel.com
Code: 200
Message: ok
Request: HTTP::Request=HASH(0xb2fc8)
Base: http://www.reel.com/
###################Basic URL
Failed. Posting did not succeed
(this changed after I used MyUserAgent. After I started using that I got the
same results as the GET results below:)
Code: 302
Message: Redirect
Request: HTTP::Request=HASH(0x23e558)
Base: http://www.reel.com/cgi-bin/search/nph-movie.exe
#################################### POST URL
As_string: HTTP/1.0 302 (Found) Redirect
Location:
http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=revenge
&TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICE
RANGE=0&PRODUCTTYPE=0
Content-Type: text/html
Client-Date: Sun, 08 Nov 1998 10:17:41 GMT
Client-Peer: 209.185.135.200:80
Set-Cookie: ReelSession=REEL345JPMY45ML4TZJ; path=/; domain=.reel.com;
Set-Cookie: ReelSender=; path=/; domain=.reel.com;

OK Was Able to grab the page
Code: 200
Message: ok
Request: HTTP::Request=HASH(0x31b17c)
Base:
http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=revenge
&TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICE
RANGE=0&PRODUCTTYPE=0
#################################### The Actual URL
As_string: HTTP/1.0 200 (OK) ok
Content-Type: text/html
Client-Date: Sun, 08 Nov 1998 10:17:42 GMT
Client-Peer: 209.185.135.200:80
Set-Cookie: ReelSession=REEL3ET4XDY93ESCNHS; path=/; domain=.reel.com;
Title: Reel | Smart Search Results


(and the rest of the HTML file, the one with no results). So there it is.
Thanks again for all of your help. Nathan Good good@cs.umn.edu Try MovieLens
http://www.movielens.umn.edu