ActiveState Community Site - lwp https requests via proxy - Comments

Re: Quite a deficit on lwp's

PeterLi9295 — Fri, 21 Mar 2008 07:17:12 -0700

What I found is that:
1. When HTTPS URL is accessable, LWP can work normally without setting any $ENV parameters;
2. When HTTPS URL is not accessable, if we want to work as well, I mean, we want to access the HTTPS URL via the proxy, we must set $ENV parameters.

Re: Quite a deficit on lwp's

margol — Mon, 10 Sep 2007 06:51:38 -0700

I don't agree. LWP doesn't need to do it, and the other proxy policy is possibly needed in some situations too. It might be nice if it was documented near the LWP::UserAgent proxies, though...

Quite a deficit on lwp's

andig — Mon, 10 Sep 2007 01:32:53 -0700

Quite a deficit on lwp's side, isn't it? Anyway, finally got it working with your support- thanks! The tricky part is that lwp will still need the http_proxy to handle http requests correctly. This is working for me:

$proxy = 'http://user:pass@server';
$ua->proxy(['http'], $proxy);

$proxy = 'http://server';
$ENV{HTTPS_PROXY}		= $proxy;
$ENV{HTTPS_PROXY_USERNAME}	= user;
$ENV{HTTPS_PROXY_PASSWORD}	= pass;

# working, http & https
my $req = HTTP::Request->new(GET => 'http(s)://site/');
my $res = $ua->request($req);
if ($res->is_success){
	print $res->as_string;
}

Re: lwp https requests via proxy

margol — Wed, 05 Sep 2007 10:45:52 -0700

You want to use LWP as if there were no proxy, and set the environment variable HTTPS_PROXY to your proxy (use HTTPS_PROXY_USERNAME and HTTPS_PROXY_PASSWORD for basic auth on the proxy).

DO NOT use LWP::UserAgent-env_proxy() after doing so.

This will use Crypt::SSLeay's built-in proxy support (which correctly uses CONNECT) rather than LWP's proxy support (which won't)

I believe the 'right' thing

andig — Tue, 14 Aug 2007 09:23:26 -0700

I believe the 'right' thing would be that lwp itself uses CONNECT instead of GET when using HTTPS via proxy- however this doesn't seem to happen?

lwp https requests via proxy

andig — Tue, 14 Aug 2007 09:21:24 -0700

I'm trying to access an https url via proxy using LWP. Without proxy code works fine, with proxy the error message for a get request is:

LWP::UserAgent::request: Simple response: Not Implemented

Reading the web I realize that GET is not the correct method, it should rather be CONNECT. Now, when I CONNECT instead of getting this is fine- only the HTTP response doesn't contain a body. I assume after CONNECT, the encrypted GET request needs to be performed.

So my question is: how to I perform an HTTP request to a remote web server using LWP via proxy?

Current code:

$proxy = 'http://user:pass@server';
$ua->proxy(['http', 'https'], $proxy);

# HTTP/1.0 200 Connection established but no body
my $req = HTTP::Request->new(CONNECT => 'https://site/');
my $res = $ua->request($req);
if ($res->is_success){
	print $res->as_string;
}

# error message: Failed: 501 Not Implemented
my $req = HTTP::Request->new(GET => 'https://site/');
my $res = $ua->request($req);
if ($res->is_success){
	print $res->as_string;
}