Subscribe to the RSS feed

lwp https requests via proxy

Posted by andig on 2007-08-14 09:21
Forums: Perl discussion | OS: Windows XP Pro

I'm trying to access an https url via proxy using LWP. Without proxy code works fine, with proxy the error message for a get request is:

LWP::UserAgent::request: Simple response: Not Implemented

Reading the web I realize that GET is not the correct method, it should rather be CONNECT. Now, when I CONNECT instead of getting this is fine- only the HTTP response doesn't contain a body. I assume after CONNECT, the encrypted GET request needs to be performed.

So my question is: how to I perform an HTTP request to a remote web server using LWP via proxy?

Current code:

$proxy = 'http://user:pass@server';
$ua->proxy(['http', 'https'], $proxy);

# HTTP/1.0 200 Connection established but no body
my $req = HTTP::Request->new(CONNECT => 'https://site/');
my $res = $ua->request($req);
if ($res->is_success){
        print $res->as_string;
}

# error message: Failed: 501 Not Implemented
my $req = HTTP::Request->new(GET => 'https://site/');
my $res = $ua->request($req);
if ($res->is_success){
        print $res->as_string;
}

I believe the 'right' thing
andig | Tue, 2007-08-14 09:23

I believe the 'right' thing would be that lwp itself uses CONNECT instead of GET when using HTTPS via proxy- however this doesn't seem to happen?

Re: lwp https requests via proxy
margol | Wed, 2007-09-05 10:45

You want to use LWP as if there were no proxy, and set the environment variable HTTPS_PROXY to your proxy (use HTTPS_PROXY_USERNAME and HTTPS_PROXY_PASSWORD for basic auth on the proxy).

DO NOT use LWP::UserAgent-env_proxy() after doing so.

This will use Crypt::SSLeay's built-in proxy support (which correctly uses CONNECT) rather than LWP's proxy support (which won't)

Quite a deficit on lwp's
andig | Mon, 2007-09-10 01:32

Quite a deficit on lwp's side, isn't it? Anyway, finally got it working with your support- thanks! The tricky part is that lwp will still need the http_proxy to handle http requests correctly. This is working for me:

$proxy = 'http://user:pass@server';
$ua->proxy(['http'], $proxy);

$proxy = 'http://server';
$ENV{HTTPS_PROXY}               = $proxy;
$ENV{HTTPS_PROXY_USERNAME}      = user;
$ENV{HTTPS_PROXY_PASSWORD}      = pass;

# working, http & https
my $req = HTTP::Request->new(GET => 'http(s)://site/');
my $res = $ua->request($req);
if ($res->is_success){
        print $res->as_string;
}

Re: Quite a deficit on lwp's
margol | Mon, 2007-09-10 06:51

I don't agree. LWP doesn't need to do it, and the other proxy policy is possibly needed in some situations too. It might be nice if it was documented near the LWP::UserAgent proxies, though...

Re: Quite a deficit on lwp's
PeterLi9295 | Fri, 2008-03-21 07:17

What I found is that:
1. When HTTPS URL is accessable, LWP can work normally without setting any $ENV parameters;
2. When HTTPS URL is not accessable, if we want to work as well, I mean, we want to access the HTTPS URL via the proxy, we must set $ENV parameters.

LWP HTTPS request via proxy
richwellman | Fri, 2012-02-03 13:55

I found at our site I needed to use this script to connect to our proxy correctly. Without the Net::SSL parameter the script was always trying to establish the HTTPS connection directly to the proxy. A modern proxy is expecting a HTTP CONNECT request with the https url in the payload.

#!usr/bin/perl

use LWP::UserAgent;

# override HTTPS setting in LWP to work with a proxy
$ENV{PERL_NET_HTTPS_SSL_SOCKET_CLASS} = "Net::SSL";
$ENV{PERL_LWP_SSL_VERIFY_HOSTNAME} = 0;
$ENV{HTTPS_PROXY}               = 'http-proxy.yourcompany.com:80';
$ENV{HTTPS_PROXY_USERNAME}      = 'username';
$ENV{HTTPS_PROXY_PASSWORD}      = 'password';

$ua = new LWP::UserAgent;

# make a https request
my $req = HTTP::Request->new(GET => 'https://mail.google.com/');
my $res = $ua->request($req);
print $res->as_string;