Re: LWP::UserAgent redirect with POST

Kit (hykit@jps.net)
Tue, 10 Aug 1999 04:02:23 -0800
Messages sorted by: [ date ][ thread ][ subject ][ author ]
Next message: Doug Monroe: "Re: LWP replacement of webget"
Previous message: jack velte: "Re: LWP::UserAgent redirect with POST"
I did just like Nathan Good did.

So then I subclassed UserAgent and changed the redirect_ok method to all
ways spit out true:

package MyUserAgent;
require LWP::UserAgent;
@ISA =qw(LWP::UserAgent);
sub redirect_ok{
    return TRUE;
}

This will cause UserAgent to go on with the redirect even if it is type
"POST".  However, a test with reel.com results in nothing being found even
though it suppose to under a web browser like netscape.


----------
>From: "jack velte" <jackv@earthling.net>
>To: "Kit" <hykit@jps.net>, <libwww-perl@ics.uci.edu>
>Subject: Re: LWP::UserAgent redirect with POST
>Date: Tue, Aug 10, 1999, 2:24 AM
>

> you have to change the 'redirect_ok()' function in useragent.pm (??) to not
> return 0 if the type is 'post'.  the code does "the right thing" which is
> wrong in the real world.
>
> -jack
>
>
> ----- Original Message -----
> From: Kit <hykit@jps.net>
> To: <libwww-perl@ics.uci.edu>
> Sent: Tuesday, August 10, 1999 4:41 AM
> Subject: LWP::UserAgent redirect with POST
>
>
>> Hi Nathan Good,
>>
>> I saw Nathan Good's post on the libwww-perl mailing list archive regarding
>> problems with LWP::UserAgent handling POST redirects and cookies.  I am
>> wondering if anyone has found out a solution for it yet, cause I am
> running
>> into the same problems.  I use LWP::UserAgent to send a POST request to
>> www.reel.com and didn't get the results I wanted.  Any help would be
>> appreciated.
>>
>>
>> Here is Nathan Good's post just to remind you:
>>
>> Hi.  I have found myself in a situation where I need cookies and am
> getting
>> the redirect error and it doesn't look like the UserAgent is sending the
>> cookies back to the server. The story is that I would like to suck
>> (responsibly of course) a bunch of movie information from
>> http://www.reel.com for my research project. Here is what I know:
>> 1)You must have cookies enabled to visit Reel.com and do anything useful
>> (search moveis etc)
>> 2)When I go to reel.com in my browser and enter in a title, say "revenge"
> in
>> the quick search column on the left, the following string shows up in the
>> location window of my browser:
>>
> http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=revenge
>>
> &TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICE
>> RANGE=0&PRODUCTTYPE=0
>> 3)If I go to this string w/o going to http://www.reel.com first (or if I
>> disable cookies) then I get a page that says "Sorry nothing found" where
>> there should be results.
>> 4)if I have cookies enabled in the browser and I go to reel, preform a
>> search "revenge" I get back the same string in the location as above and I
>> get a list of movies with the title "revenge".
>>
>> Here's what I've tried in LWP:
>> 1)simulate a browser session; get "Reel.com", then POST.
>> 2)same as above with a GET after the POST
>> Result is 302 Redirect error from POST
>> So then I subclassed UserAgent and changed the redirect_ok method to all
>> ways spit out true:
>> package MyUserAgent;
>> require LWP::UserAgent;
>> @ISA =qw(LWP::UserAgent);
>> sub redirect_ok{
>>  return TRUE;
>> }
>>
>> 3)After this I repeated the above tests.
>> Result: No more redirect error. kept getting the page saying that there
> were
>> no results found, the same as when cookies were disabled in the browser.
>> 4)Tried a single POST Test.
>> Result: kept getting the page saying that there were no results found.
>> 5)tried various combinations of post strings in
>> $req->content();
>> exp:
>>
> $req->content('OBJECT=searchresults&STRING=revenge&TYPE=TITLE&STORE=ALL&MODE
>> =BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICERANGE=0&PRODUCTTYPE=0');
>> and
>>
> $req->content(STRING=revenge&TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD
>> =0&RATING=0&AVAIL=BOTH&PRICERANGE=0&PRODUCTTYPE=0);
>> Result: Same 'results not found' page.
>>
>> I am new to the user agent interface (I was just using LWP Simple before)
> so
>> I might have made some errors there. I think the problem has something to
> do
>> with my mishandling of the cookie-jar, as it seems that reel.com is really
>> not getting my cookies. Anyway, I've included the errors I get and the
> code
>> I wrote.
>>
>> Code I wrote:
>> #!/project/gl/install/bin/perl
>>
>> use lib "/project/gl2/nathan/lib";
>> #after subclassing I used also included this
>> #require "/project/gl2/nathan/lib/LWP/MyUserAgent.pm";
>>
>> use LWP::UserAgent;
>>
>>        $ua = new LWP::UserAgent;
>>  #after subclassing I used this:
>>  #$ua= new MyUserAgent;
>>
>> $ua->agent("Mozilla/4.0");
>>
>> #for cookies
>> $ua->cookie_jar($cookies);
>>
>> #Go to Reel.com and get a cookie
>>  my $req = new HTTP::Request 'GET','http://www.reel.com';
>>        $req->content_type('application/x-www-form-urlencoded');
>>        my $res = $ua->request($req);
>> if ($res->is_success) {
>>    print "Was able to go to reel.com\n";
>>  } else {
>>    print "Failed. Didn't make it\n";
>> }
>>
>> $r = $res;
>>
>> print "Code: ", $r->code, "\n";
>> print "Message: ", $r->message, "\n";
>> print "Request: ", $r->request, "\n";
>> print "Base: ", $r->base, "\n";
>>
>> print "###################Basic URL\n";
>> #print "As_string: ", $r->as_string, "\n";
>>
>>
>> #Now try to post to Reel.com
>> #When using MyUserAgent got back the the 'No search results page'
>>
>>        my $req = new HTTP::Request
>> 'POST','http://www.reel.com/cgi-bin/search/nph-movie.exe';
>>        $req->content_type('application/x-www-form-urlencoded');
>>        $req->content('STRING=revenge&TYPE=TITLE');
>>        my $res = $ua->request($req);
>>
>> if ($res->is_success) {
>>    print "OK Was Able to Post\n";
>>  } else {
>>    print "Failed. Posting did not succeed\n";
>> }
>>
>> $r = $res;
>>
>> print "Code: ", $r->code, "\n";
>> print "Message: ", $r->message, "\n";
>> print "Request: ", $r->request, "\n";
>> print "Base: ", $r->base, "\n";
>> print "#################################### POST URL\n";
>> print "As_string: ", $r->as_string, "\n";
>>
>>
>> #Now try to get things from reel.com
>> #I didn't use this all the time
>>
>>  my $req = new HTTP::Request
>>
> 'GET','http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=
>>
> revenge&TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOT
>> H&PRICERANGE=0&PRODUCTTYPE=0';
>>        $req->content_type('application/x-www-form-urlencoded');
>>        my $res = $ua->request($req);
>> if ($res->is_success) {
>>    print "OK Was Able to grab the page\n";
>>  } else {
>>    print "Failed. Grabbing did not succeed\n";
>> }
>>
>> $r = $res;
>>
>> print "Code: ", $r->code, "\n";
>> print "Message: ", $r->message, "\n";
>> print "Request: ", $r->request, "\n";
>> print "Base: ", $r->base, "\n";
>> print "#################################### The Actual URL\n";
>> print "As_string: ", $r->as_string, "\n";
>>
>> <end of code>
>>
>> This is a list of my results before subclassing UserAgent.pm
>> ####################################
>> MY Results:
>>
>> Was able to go to reel.com
>> Code: 200
>> Message: ok
>> Request: HTTP::Request=HASH(0xb2fc8)
>> Base: http://www.reel.com/
>> ###################Basic URL
>> Failed. Posting did not succeed
>> (this changed after I used MyUserAgent. After I started using that I got
> the
>> same results as the GET results below:)
>> Code: 302
>> Message: Redirect
>> Request: HTTP::Request=HASH(0x23e558)
>> Base: http://www.reel.com/cgi-bin/search/nph-movie.exe
>> #################################### POST URL
>> As_string: HTTP/1.0 302 (Found) Redirect
>> Location:
>>
> http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=revenge
>>
> &TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICE
>> RANGE=0&PRODUCTTYPE=0
>> Content-Type: text/html
>> Client-Date: Sun, 08 Nov 1998 10:17:41 GMT
>> Client-Peer: 209.185.135.200:80
>> Set-Cookie: ReelSession=REEL345JPMY45ML4TZJ; path=/; domain=.reel.com;
>> Set-Cookie: ReelSender=; path=/; domain=.reel.com;
>>
>> OK Was Able to grab the page
>> Code: 200
>> Message: ok
>> Request: HTTP::Request=HASH(0x31b17c)
>> Base:
>>
> http://www.reel.com/cgi-bin/nph-reel.exe?OBJECT=searchresults&STRING=revenge
>>
> &TYPE=TITLE&STORE=ALL&MODE=BEGINS&GENRE=0&PERIOD=0&RATING=0&AVAIL=BOTH&PRICE
>> RANGE=0&PRODUCTTYPE=0
>> #################################### The Actual URL
>> As_string: HTTP/1.0 200 (OK) ok
>> Content-Type: text/html
>> Client-Date: Sun, 08 Nov 1998 10:17:42 GMT
>> Client-Peer: 209.185.135.200:80
>> Set-Cookie: ReelSession=REEL3ET4XDY93ESCNHS; path=/; domain=.reel.com;
>> Title: Reel | Smart Search Results
>>
>>
>> (and the rest of the HTML file, the one with no results). So there it is.
>> Thanks again for all of your help. Nathan Good good@cs.umn.edu Try
> MovieLens
>> http://www.movielens.umn.edu
>>
>
>
Next message: Doug Monroe: "Re: LWP replacement of webget"
Previous message: jack velte: "Re: LWP::UserAgent redirect with POST"