httpclient是apache的一个项目：http://hc.apache.org/

文档比較完好：http://hc.apache.org/httpcomponents-client-ga/tutorial/html/

这里就不啰嗦了，主要是在做demo的时候遇到的一些问题在这里总结一下：

[引用请注明出处http://blog.csdn.net/bhq2010/article/details/9210007]

1、使用连接池

虽说http协议时无连接的，但毕竟是基于tcp的，底层还是须要和server建立连接的。对于须要从同一个网站抓取大量网页的程序，应该使用连接池，否则每次抓取都和Web网站建立连接、发送请求、获得响应、释放连接，一方面效率不高，还有一方面稍不小心就会疏忽了某些资源的释放、导致网站拒绝连接（非常多网站会拒绝同一个ip的大量连接、防止DOS攻击）。

连接池的例程例如以下：

SchemeRegistry schemeRegistry = new SchemeRegistry();
	schemeRegistry.register(new Scheme("http", 80, PlainSocketFactory.getSocketFactory()));
schemeRegistry.register(new Scheme("https", 443, SSLSocketFactory.getSocketFactory()));
PoolingClientConnectionManager cm = new PoolingClientConnectionManager(schemeRegistry);
	cm.setMaxTotal(200);
	cm.setDefaultMaxPerRoute(2);
	HttpHost googleResearch = new HttpHost("research.google.com", 80);
	HttpHost wikipediaEn = new HttpHost("en.wikipedia.org", 80);
	cm.setMaxPerRoute(new HttpRoute(googleResearch), 30);
	cm.setMaxPerRoute(new HttpRoute(wikipediaEn), 50);

从连接池中获取http client也非常方面：

DefaultHttpClient client = new DefaultHttpClient(cm);

2、设置HttpClient參数

HttpClient须要设置合适的參数，才干更好地工作。默认的參数可以应付少量的抓取工作，但找到一组合适的參数往往能改善特定情况下的抓取效果。设置參数的例程例如以下：

		DefaultHttpClient client = new DefaultHttpClient(cm);
		Integer socketTimeout = 10000;
		Integer connectionTimeout = 10000;
		final int retryTime = 3;
		client.getParams().setParameter(CoreConnectionPNames.SO_TIMEOUT, socketTimeout);
		client.getParams().setParameter(CoreConnectionPNames.CONNECTION_TIMEOUT, connectionTimeout);
		client.getParams().setParameter(CoreConnectionPNames.TCP_NODELAY, false);
		client.getParams().setParameter(CoreConnectionPNames.SOCKET_BUFFER_SIZE, 1024 * 1024);
		HttpRequestRetryHandler myRetryHandler = new HttpRequestRetryHandler()
		{
			@Override
			public boolean retryRequest(IOException exception, int executionCount, HttpContext context)
			{
				if (executionCount >= retryTime)
				{
					// Do not retry if over max retry count
					return false;
				}
				if (exception instanceof InterruptedIOException)
				{
					// Timeout
					return false;
				}
				if (exception instanceof UnknownHostException)
				{
					// Unknown host
					return false;
				}
				if (exception instanceof ConnectException)
				{
					// Connection refused
					return false;
				}
				if (exception instanceof SSLException)
				{
					// SSL handshake exception
					return false;
				}
				HttpRequest request = (HttpRequest) context.getAttribute(ExecutionContext.HTTP_REQUEST);
				boolean idempotent = !(request instanceof HttpEntityEnclosingRequest);
				if (idempotent)
				{
					// Retry if the request is considered idempotent
					return true;
				}
				return false;
			}

		};
		client.setHttpRequestRetryHandler(myRetryHandler);

3、设置request header

		HttpResponse response = null;
		HttpGet get = new HttpGet(url);
		get.addHeader("Accept", "text/html");
		get.addHeader("Accept-Charset", "utf-8");
		get.addHeader("Accept-Encoding", "gzip");
		get.addHeader("Accept-Language", "en-US,en");
		get.addHeader("User-Agent", "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.22 (KHTML, like Gecko) Chrome/25.0.1364.160 Safari/537.22");
		response = client.execute(get);
		HttpEntity entity = response.getEntity();
		Header header = entity.getContentEncoding();
		if (header != null)
		{
			HeaderElement[] codecs = header.getElements();
			for (int i = 0; i < codecs.length; i++)
			{
				if (codecs[i].getName().equalsIgnoreCase("gzip"))
				{
					response.setEntity(new GzipDecompressingEntity(entity));
				}
			}
		}
		return response;