在使用HttpClient進行抓取一些網頁的時候，經常會保留從伺服器端發回的Cookie資訊，以便發起其他需要這些Cookie的請求。大多數情況下，我們使用內建的cookie策略，便能夠方便直接地獲取這些cookie。
下面的一小段程式碼，就是訪問http://www.baidu.com，並獲取對應的cookie：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 |
@ Test public void getCookie(){ CloseableHttpClient httpClient = HttpClients.createDefault(); HttpGet get=new HttpGet("http://www.baidu.com"); HttpClientContext context = HttpClientContext.create(); try { CloseableHttpResponse response = httpClient.execute(get, context); try{ System.out.println(">>>>>>headers:"); Arrays.stream(response.getAllHeaders()).forEach(System.out::println); System.out.println(">>>>>>cookies:"); context.getCookieStore().getCookies().forEach(System.out::println); } finally { response.close(); } } catch (IOException e) { e.printStackTrace(); }finally { try { httpClient.close(); } catch (IOException e) { e.printStackTrace(); } } } |
列印結果
|
1 2 3 4 5 6 7 8 9 10 11 12 |
>>>>>>headers: Server: bfe/1.0.8.18 Date: Tue, 12 Sep 2017 06:19:06 GMT Content-Type: text/html Last-Modified: Mon, 23 Jan 2017 13:28:24 GMT Transfer-Encoding: chunked Connection: Keep-Alive Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform Pragma: no-cache Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/ >>>>>>cookies: [version: 0][name: BDORZ][value: 27315][domain: baidu.com][path: /][expiry: null] |
但是也有一些網站返回的cookie並不一定完全符合規範，例如下面這個例子，從列印出的header中可以看到，這個cookie中的Expires屬性是時間戳形式，並不符合標準的時間格式，因此，httpclient對於cookie的處理失效，最終無法獲取到cookie，並且發出了一條警告資訊：“Invalid ‘expires’ attribute: 1505204523”
|
1 2 3 4 5 6 7 8 9 |
警告: Invalid cookie header: "Set-Cookie: yd_cookie=90236a64-8650-494b332a285dbd886e5981965fc4a93f023d; Expires=1505204523; Path=/; HttpOnly". Invalid 'expires' attribute: 1505204523 >>>>>>headers: Date: Tue, 12 Sep 2017 06:22:03 GMT Content-Type: text/html Connection: keep-alive Set-Cookie: yd_cookie=90236a64-8650-494b332a285dbd886e5981965fc4a93f023d; Expires=1505204523; Path=/; HttpOnly Cache-Control: no-cache, no-store Server: WAF/2.4-12.1 >>>>>>cookies: |
雖然我們可以利用header的資料，重新構造一個cookie出來，也有很多人確實也是這麼做的，但這種方法不夠優雅，那麼如何解決這個問題？網上相關的資料又很少，所以就只能先從官方文件入手。在官方文件3.4小節custom cookie policy中講到允許自定義的cookie策略，自定義的方法是實現CookieSpec介面，並通過CookieSpecProvider來完成在httpclient中的初始化和註冊策略例項的工作。好了，關鍵的線索在於CookieSpec介面，我們來看一下它的原始碼：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
public interface CookieSpec { …… /** * Parse the {@ code "Set-Cookie"} Header into an array of Cookies. * * <p> This method will not perform the validation of the resultant<code> * {@link Cookie}s</p> * * @ see #validate * * @param header the {@ code Set-Cookie} received from the server * @param origin details of the cookie origin * @ return an array of {@ code Cookie}s parsed from the header * @throws MalformedCookieException if an exception occurs during parsing */ List parse(Header header, CookieOrigin origin) throws MalformedCookieException; …… } |
在原始碼中我們發現了一個parse方法，看註釋就知道正是這個方法，將Set-Cookie的header資訊解析為Cookie物件，自然地再瞭解一下在httplcient中的預設實現DefaultCookieSpec，限於篇幅，原始碼就不貼了。在預設的實現中，DefaultCookieSpec主要的工作是判斷header中Cookie規範的型別，然後再呼叫具體的某一個實現。像上述這種Cookie，最終是交由NetscapeDraftSpec的例項來做解析，而在NetscapeDraftSpec的原始碼中，定義了預設的expires時間格式為“EEE, dd-MMM-yy HH:mm:ss z”
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
public class NetscapeDraftSpec extends CookieSpecBase { protected static final String EXPIRES_PATTERN = "EEE, dd-MMM-yy HH:mm:ss z"; /** Default constructor */ public NetscapeDraftSpec(final String[] datepatterns) { super(new BasicPathHandler(), new NetscapeDomainHandler(), new BasicSecureHandler(), new BasicCommentHandler(), new BasicExpiresHandler( datepatterns != null ? datepatterns.clone() : new String[]{EXPIRES_PATTERN})); } NetscapeDraftSpec(final CommonCookieAttributeHandler... handlers) { super(handlers); } public NetscapeDraftSpec() { this((String[]) null); } …… } |
到這裡已經比較清楚了，我們只需要將Cookie中expires的時間轉換為正確的格式，然後再送入預設的解析器就可以了。
解決方法：
- 自定義一個CookieSpec類，繼承DefaultCookieSpec
- 重寫parser方法
- 將Cookie中的expires轉換為正確的時間格式
- 呼叫預設的解析方法
實現如下（URL就不公開了，已經隱去）
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 |
public class TestHttpClient { String url = sth; class MyCookieSpec extends DefaultCookieSpec { @ Override public List parse(Header header, CookieOrigin cookieOrigin) throws MalformedCookieException { String value = header.getValue(); String prefix = "Expires="; if (value.contains(prefix)) { String expires = value.substring(value.indexOf(prefix) + prefix.length()); expires = expires.substring(0, expires.indexOf(";")); String date = DateUtils.formatDate(new Date(Long.parseLong(expires) * 1000L),"EEE, dd-MMM-yy HH:mm:ss z"); value = value.replaceAll(prefix + "\\d{10};", prefix + date + ";"); } header = new BasicHeader(header.getName(), value); return super.parse(header, cookieOrigin); } } @ Test public void getCookie() { CloseableHttpClient httpClient = HttpClients.createDefault(); Registry cookieSpecProviderRegistry = RegistryBuilder.create() .register("myCookieSpec", context -> new MyCookieSpec()).build();//註冊自定義CookieSpec HttpClientContext context = HttpClientContext.create(); context.setCookieSpecRegistry(cookieSpecProviderRegistry); HttpGet get = new HttpGet(url); get.setConfig(RequestConfig.custom().setCookieSpec("myCookieSpec").build()); try { CloseableHttpResponse response = httpClient.execute(get, context); try{ System.out.println(">>>>>>headers:"); Arrays.stream(response.getAllHeaders()).forEach(System.out::println); System.out.println(">>>>>>cookies:"); context.getCookieStore().getCookies().forEach(System.out::println); } finally { response.close(); } } catch (IOException e) { e.printStackTrace(); }finally { try { httpClient.close(); } catch (IOException e) { e.printStackTrace(); } } } } |
再次執行，順利地列印出正確的結果，完美！
|
1 2 3 4 5 6 7 8 9 |
>>>>>>headers: Date: Tue, 12 Sep 2017 07:24:10 GMT Content-Type: text/html Connection: keep-alive Set-Cookie: yd_cookie=9f521fc5-0248-4ab3ee650ca50b1c7abb1cd2526b830e620f; Expires=1505208250; Path=/; HttpOnly Cache-Control: no-cache, no-store Server: WAF/2.4-12.1 >>>>>>cookies: [version: 0][name: yd_cookie][value: 9f521fc5-0248-4ab3ee650ca50b1c7abb1cd2526b830e620f][domain: www.sth.com][path: /][expiry: Tue Sep 12 17:24:10 CST 2017] |
本文也發我的個人部落格：fullstackyang