Proxying in LWP .01 (was Re: LWP .01

Martijn Koster (m.koster@nexor.co.uk)
Tue, 18 Jul 1995 18:44:50 +0100 

Sorry I'm responding a bit late. Real work want to get done badly here
:-/

In message <199507180452.AAA28609@fugit.ny.jpmorgan.com>, Joel Scotkin writes:

> This is great stuff.  You blew my evening - I just hadda sit at home
> and play. :)

:-) {Perl,Web}, destroyer of evenings.
 
> except it seems to me that proxying is handled at the wrong level.

Hold on, we may be talking about different things here. The proxying
support that is in the library is designed to proxy one protocol over
another, like the CERN proxy server can do. If you're talking about
application level proxying for protocols themselves through a firewall
that may be a separate thing. Can you explain what exactly you want to
do?

The proxy code has been backwards and forwards a bit. Having had it
running in several different places (including the Protocols) I felt
the current arrangements made the most sense.

I also think it is very important this stuff works, because I'm
not prepared to wait till Protocol/ftp.pm gets done.

> Currently, all proxying is done in UserAgent, and really only kicks in
> if simpleRequest() is called.  Since even a simple program like get
> does the LWP::request() call itself, this doesn't really help very much.

Hold on, the LWP::request() calls LWP::simpleRequest(), so it helps
quite a bit :-) Did you check out test/get-proxy.pl?

 $ua->proxy('ftp', 'http://web.nexor.co.uk/');

 my $url = new URI::URL('ftp://lancaster.nexor.co.uk/welcome.msg');

 my $request = new LWP::Request('GET', $url);
 my $response = $ua->request($request, undef, undef);

You can't make it much more simple than that. The only thing a user
agent needs to do is handle the configuration (from environment
variables or a config file). That can be added to bin/get with a few
lines.
 
> It would seem to be much more useful to move the proxying code into the
> request modules (where you would need to do this once per protocol), or
> even into the Protocol module, so it would be inherited.  This way, 
> proxying would be completely automatic if needed (and the higher up it
> goes in the class tree, the easier it would be to selectively disable).
> 
> Does this make sense, or am I missing something fundamental?

Well, at the end of the day the proxying _is_ handled in the
protocols. Any protocol can decide if it can be used to proxy other
protocols. Currently only http knows how to proxy. Protocols that
don't do not need to be bogged down with extra code.

The only thing that happens in the User-agent.pm is that it know which
protocols to proxy over which others, using which proxy servers. That
can't really be anywhere else.

It was a while back, so I'm not sure I can remember all of my
motivation, but I explicitly moved it out of the Protocol modules
after moving it into them for good reason :-)

> While I'm on the subject of Request, Response and Message, these
> classes seem to be strictly HTTP.  I'm wondering if using these
> names runs the risk of polluting the namespace too fast - maybe they
> should be prefixed with HTTP_, or some version thereof.  This would
> also make it much easier to handle (and test) future versions of
> HTTP, and might reduce confusion when ftp, gopher and whatever make
> it in.

I disagree, did you see my message about the naming of this library?
The model of having everything appear as an HTTP-like Message is the
whole architecture behind this library and its predecessor libwww-perl
0.40. Even gopher will appear as a HTTP-like message. It should
definately not start defining its own messages.

If this turns out not to be the best model in future, then a different
architectured library may be appropriate, and that can be named
differently.

I agree we should be careful about the namespace, that's why I don't
think we should go for WWW::*, but stay behind LWP::. There we can do
what we want (We planted the flag). If a protocol module needs extra
stuff internally it should stick it behind Protocol::complex::foo or
something.

I'm definately going to have to write up the architecture for this
thing... But time is in short supply.

> I'll take a look at making the proxying a more inherent part of protocol
> requests, but I'm not sure if I have the time to do it right...

Again, let us know exactly what your requirement is, and how you
propose to go about making it happen. Maybe our collective brains can
help.

> Thanks for a great tool!

Glad it at least was a fun evening :-)

-- Martijn
__________
Internet: m.koster@nexor.co.uk
X-400: C=GB; A= ; P=Nexor; O=Nexor; S=koster; I=M
WWW: http://web.nexor.co.uk/mak/mak.html