Java爬取並下載酷狗TOP500歌曲
來源:my.oschina.net/gllfeixiang/blog/2995570?p=1
是這樣的,之前買車送的垃圾記錄儀不能用了,這兩天狠心買了好點的記錄儀,帶導航、音樂、藍芽、4G等功能,尋思,既然有這些功能就利用起來,用4G聽歌有點奢侈,就準備去酷狗下點歌聽,居然都是需要辦會員才能下載,而且vip一月只能下載300首,我這麼窮又這麼摳怎麼可能衝會員,於是百度搜了下怎麼免費下載,都是python爬取,雖然也會一點,但是電腦上沒安裝python,再安裝再研究感覺有點費勁,於是就花了半小時做了這個爬蟲,技術一般,只記錄分析實現過程,大牛請繞行。其中用到了一些庫,包括:jsoup、HttpClient、net.sf.json大家可以自行去下載jar包。
1、分析是否能獲得TOP500歌單
首先,開啟酷狗首頁檢視酷狗TOP500,說好的500首,怎麼就只有22首呢?
是真的只讓看這些還是能找到其餘的呢,於是我就看了下這TOP500的連結:
https://www.kugou.com/yy/rank/home/1-8888.html?from=rank
可以看的出home後邊有個1,難道這是代表第一頁的意思?於是我就把1改成2,進入,果然進入了第二頁, 至此可以知道我們可以在網頁裡獲取這500首的歌單。
2、分析找到真正的mp3下載地址(這個有點繞)
點一個歌曲進入播放頁面,使用谷歌瀏覽器的控制檯的Elements,搜一下mp3,很輕鬆就定位到了MP3的位置。
但是使用java訪問的時候爬取的html裡卻沒有該mp3的檔案地址,那麼這肯定是在該頁面的位置使用了js來載入mp3,那麼重新整理下網頁,看網頁載入了哪些東西,載入的東西有點多,著重看一下js、php的請求,主要是看裡面有沒有mp3的地址,分析細節就不用說了。
最終我在列表的
https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&hash=667939C6E784265D541DEEE65AE4F2F8&album_id=0&_=1546235744251
這個請求裡發現了mp3的完整地址:
"play_url": "http:\/\/fs.w.kugou.com\/201812311325\/dcf5b6449160903c6ee48035e11434bb\/G128\/M08\/02\/09\/IIcBAFrZqf2ANOadADn94ubOmaU995.mp3",
那這個js是怎麼判斷是哪首歌的呢,那麼只可能是hash這個引數來決定歌曲的,然後到播放頁面裡找到這個hash的位置,是在下面的js裡:
var dataFromSmarty = [{"hash":"667939C6E784265D541DEEE65AE4F2F8","timelength":"237051","audio_name":"\u767d\u5c0f\u767d - \u6700\u7f8e\u5a5a\u793c","author_name":"\u767d\u5c0f\u767d","song_name":"\u6700\u7f8e\u5a5a\u793c","album_id":0}],//當前頁面歌曲資訊
playType = "search_single";//當前播放
</script>
在去java爬取該網頁,檢視能否爬到這個hash,果然,爬取的html裡有這段js,到現在mp3的地址也找到了,歌單也找到了,那麼下一步就用程式實現就可以了。
3、java實現爬取酷狗mp3
先看一下爬取結果:
找到了資源,程式實現就好說了,其中使用到了自己寫的幾個工具類,自己整理點自己的工具類還是有好處的,以後遇到什麼問題就沒必要重新寫了,直接拿來用就可以了。沒什麼好說的了,下面直接貼出原始碼。
SpiderKugou.java
package com.bing.spider;
import java.io.IOException;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.bing.download.FileDownload;
import com.bing.html.HtmlManage;
import com.bing.http.HttpGetConnect;
import net.sf.json.JSONObject;
public class SpiderKugou {
public static String filePath = "F:/music/";
public static String mp3 = "https://wwwapi.kugou.com/yy/index.php?r=play/getdata&callback=jQuery191027067069941080546_1546235744250&"
+ "hash=HASH&album_id=0&_=TIME";
public static String LINK = "https://www.kugou.com/yy/rank/home/PAGE-8888.html?from=rank";
//"https://www.kugou.com/yy/rank/home/PAGE-23784.html?from=rank";
public static void main(String[] args) throws IOException {
for(int i = 1 ; i < 23 ; i++){
String url = LINK.replace("PAGE", i + "");
getTitle(url);
//download("https://www.kugou.com/song/mfy6je5.html");
}
}
public static String getTitle(String url) throws IOException{
HttpGetConnect connect = new HttpGetConnect();
String content = connect.connect(url, "utf-8");
HtmlManage html = new HtmlManage();
Document doc = html.manage(content);
Element ele = doc.getElementsByClass("pc_temp_songlist").get(0);
Elements eles = ele.getElementsByTag("li");
for(int i = 0 ; i < eles.size() ; i++){
Element item = eles.get(i);
String title = item.attr("title").trim();
String link = item.getElementsByTag("a").first().attr("href");
download(link,title);
}
return null;
}
public static String download(String url,String name) throws IOException{
String hash = "";
HttpGetConnect connect = new HttpGetConnect();
String content = connect.connect(url, "utf-8");
HtmlManage html = new HtmlManage();
String regEx = "\"hash\":\"[0-9A-Z]+\"";
// 編譯正規表示式
Pattern pattern = Pattern.compile(regEx);
Matcher matcher = pattern.matcher(content);
if (matcher.find()) {
hash = matcher.group();
hash = hash.replace("\"hash\":\"", "");
hash = hash.replace("\"", "");
}
String item = mp3.replace("HASH", hash);
item = item.replace("TIME", System.currentTimeMillis() + "");
System.out.println(item);
String mp = connect.connect(item, "utf-8");
mp = mp.substring(mp.indexOf("(") + 1, mp.length() - 3);
JSONObject json = JSONObject.fromObject(mp);
String playUrl = json.getJSONObject("data").getString("play_url");
System.out.print(playUrl + " == ");
FileDownload down = new FileDownload();
down.download(playUrl, filePath + name + ".mp3");
System.out.println(name + "下載完成");
return playUrl;
}
}
HttpGetConnect.java
package com.bing.http;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.security.NoSuchAlgorithmException;
import java.security.cert.CertificateException;
import java.security.cert.X509Certificate;
import javax.net.ssl.SSLContext;
import javax.net.ssl.TrustManager;
import javax.net.ssl.X509TrustManager;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.HttpEntity;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.HttpClient;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.conn.ClientConnectionManager;
import org.apache.http.conn.scheme.Scheme;
import org.apache.http.conn.scheme.SchemeRegistry;
import org.apache.http.conn.ssl.SSLSocketFactory;
import org.apache.http.impl.client.BasicResponseHandler;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.DefaultHttpClient;
import org.apache.http.impl.client.HttpClients;
import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
import org.apache.http.params.HttpParams;
/**
* @說明:
* @author: gaoll
* @CreateTime:2014-11-13
* @ModifyTime:2014-11-13
*/
public class HttpGetConnect {
/**
* 獲取html內容
* @param url
* @param charsetName UTF-8、GB2312
* @return
* @throws IOException
*/
public static String connect(String url,String charsetName) throws IOException{
BasicHttpClientConnectionManager connManager = new BasicHttpClientConnectionManager();
CloseableHttpClient httpclient = HttpClients.custom()
.setConnectionManager(connManager)
.build();
String content = "";
try{
HttpGet httpget = new HttpGet(url);
RequestConfig requestConfig = RequestConfig.custom()
.setSocketTimeout(5000)
.setConnectTimeout(50000)
.setConnectionRequestTimeout(50000)
.build();
httpget.setConfig(requestConfig);
httpget.setHeader("Accept", "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8");
httpget.setHeader("Accept-Encoding", "gzip,deflate,sdch");
httpget.setHeader("Accept-Language", "zh-CN,zh;q=0.8");
httpget.setHeader("Connection", "keep-alive");
httpget.setHeader("Upgrade-Insecure-Requests", "1");
httpget.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36");
//httpget.setHeader("Hosts", "www.oschina.net");
httpget.setHeader("cache-control", "max-age=0");
CloseableHttpResponse response = httpclient.execute(httpget);
int status = response.getStatusLine().getStatusCode();
if (status >= 200 && status < 300) {
HttpEntity entity = response.getEntity();
InputStream instream = entity.getContent();
BufferedReader br = new BufferedReader(new InputStreamReader(instream,charsetName));
StringBuffer sbf = new StringBuffer();
String line = null;
while ((line = br.readLine()) != null){
sbf.append(line + "\n");
}
br.close();
content = sbf.toString();
} else {
content = "";
}
}catch(Exception e){
e.printStackTrace();
}finally{
httpclient.close();
}
//log.info("content is " + content);
return content;
}
private static Log log = LogFactory.getLog(HttpGetConnect.class);
}
HtmlManage.java
package com.bing.html;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.jsoup.Jsoup;
import org.jsoup.nodes.Document;
import org.jsoup.nodes.Element;
import org.jsoup.select.Elements;
import com.bing.http.HttpGetConnect;
/**
* @說明:
* @author: gaoll
* @CreateTime:2014-11-13
* @ModifyTime:2014-11-13
*/
public class HtmlManage {
public Document manage(String html){
Document doc = Jsoup.parse(html);
return doc;
}
public Document manageDirect(String url) throws IOException{
Document doc = Jsoup.connect( url ).get();
return doc;
}
public List<String> manageHtmlTag(Document doc,String tag ){
List<String> list = new ArrayList<String>();
Elements elements = doc.getElementsByTag(tag);
for(int i = 0; i < elements.size() ; i++){
String str = elements.get(i).html();
list.add(str);
}
return list;
}
public List<String> manageHtmlClass(Document doc,String clas ){
List<String> list = new ArrayList<String>();
Elements elements = doc.getElementsByClass(clas);
for(int i = 0; i < elements.size() ; i++){
String str = elements.get(i).html();
list.add(str);
}
return list;
}
public List<String> manageHtmlKey(Document doc,String key,String value ){
List<String> list = new ArrayList<String>();
Elements elements = doc.getElementsByAttributeValue(key, value);
for(int i = 0; i < elements.size() ; i++){
String str = elements.get(i).html();
list.add(str);
}
return list;
}
private static Log log = LogFactory.getLog(HtmlManage.class);
}
FileDownload.java
package com.bing.download;
import java.io.BufferedInputStream;
import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import org.apache.commons.logging.Log;
import org.apache.commons.logging.LogFactory;
import org.apache.http.client.config.RequestConfig;
import org.apache.http.client.methods.CloseableHttpResponse;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.CloseableHttpClient;
import org.apache.http.impl.client.HttpClients;
/**
* @說明:
* @author: gaoll
* @CreateTime:2014-11-20
* @ModifyTime:2014-11-20
*/
public class FileDownload {
/**
* 檔案下載
* @param url 連結地址
* @param path 要儲存的路徑及檔名
* @return
*/
public static boolean download(String url,String path){
boolean flag = false;
CloseableHttpClient httpclient = HttpClients.createDefault();
RequestConfig requestConfig = RequestConfig.custom().setSocketTimeout(2000)
.setConnectTimeout(2000).build();
HttpGet get = new HttpGet(url);
get.setConfig(requestConfig);
BufferedInputStream in = null;
BufferedOutputStream out = null;
try{
for(int i=0;i<3;i++){
CloseableHttpResponse result = httpclient.execute(get);
System.out.println(result.getStatusLine());
if(result.getStatusLine().getStatusCode() == 200){
in = new BufferedInputStream(result.getEntity().getContent());
File file = new File(path);
out = new BufferedOutputStream(new FileOutputStream(file));
byte[] buffer = new byte[1024];
int len = -1;
while((len = in.read(buffer,0,1024)) > -1){
out.write(buffer,0,len);
}
flag = true;
break;
}else if(result.getStatusLine().getStatusCode() == 500){
continue ;
}
}
}catch(Exception e){
e.printStackTrace();
flag = false;
}finally{
get.releaseConnection();
try{
if(in != null){
in.close();
}
if(out != null){
out.close();
}
}catch(Exception e){
e.printStackTrace();
flag = false;
}
}
return flag;
}
private static Log log = LogFactory.getLog(FileDownload.class);
}
到這就結束了,有可能有些程式碼沒貼全,主要程式碼已經差不多,應該可以跑起來,多多指教。
PS:如果覺得團長的分享不錯,歡迎大家隨手點“好看”、轉發。
(完)
Java團長
專注於Java乾貨分享
掃描上方二維碼獲取更多Java乾貨
相關文章
- Java爬蟲系列之實戰:爬取酷狗音樂網 TOP500 的歌曲Java爬蟲
- python 爬蟲 1 爬取酷狗音樂Python爬蟲
- Python從網易雲音樂、QQ 音樂、酷狗音樂等搜尋和下載歌曲Python
- 爬蟲 Scrapy框架 爬取圖蟲圖片並下載爬蟲框架
- python 突破酷狗收費音樂下載Python
- 一篇文章教會你使用Python網路爬蟲下載酷狗音樂Python爬蟲
- python爬取鬥圖啦表情包並下載到本地Python
- 用Python爬取網易雲音樂歌曲Python
- 爬蟲新手入門實戰專案(爬取筆趣閣小說並下載)爬蟲
- 酷狗音樂的爬取,基於python,從無到有完整教程-上:搭建環境及爬取原理講解Python
- 酷狗面試感想面試
- Python爬取網易雲音樂歌單歌曲Python
- 爭取來的酷狗前端面試 共勉之前端面試
- 如何用Python網路爬蟲爬取網易雲音樂歌曲Python爬蟲
- Swift 下載檔案,並讀取Swift
- 爬取網站前3_下載限速網站
- python 爬蟲之requests爬取頁面圖片的url,並將圖片下載到本地Python爬蟲
- 魅藍max怎麼下載歌曲教程
- Python爬蟲 搜尋並下載圖片Python爬蟲
- 使用C#爬取快手作者主頁,並下載影片/圖集(附原始碼及軟體下載連結)C#原始碼
- win10系統下如何去掉酷狗音樂廣告Win10
- 【Python爬蟲實戰】使用Selenium爬取QQ音樂歌曲及評論資訊Python爬蟲
- python 爬取飄花電影 下載地址Python
- Android QQ音樂/酷狗音樂鎖屏控制實現原理,酷狗鎖屏Android
- 如何使用 python 爬取酷我線上音樂Python
- Python爬蟲全網搜尋並下載音樂Python爬蟲
- Python爬蟲訓練:爬取酷燃網視訊資料Python爬蟲
- Java爬取網易雲音樂民謠並匯入Excel分析JavaExcel
- Java爬蟲批量爬取圖片Java爬蟲
- Java爬蟲-爬取疫苗批次資訊Java爬蟲
- 總結獲得【酷我音樂】歌曲URL地址
- Java爬蟲 爬取bing必應每日一圖背景圖下載到本地(HttpClient+Jsoup+Jackson)Java爬蟲HTTPclientJS
- 實現java讀取網頁內容並下載網頁中出現的圖片Java網頁
- 利用python爬取丁香醫生上新型肺炎資料,並下載到本地,附帶經緯度資訊Python
- Python爬蟲:通過做專案,小編了解了酷狗音樂的加密過程Python爬蟲加密
- [Python學習] 簡單爬取CSDN下載資源資訊Python
- Python下載網易雲歌曲(版許可權制的怎麼播放和下載呢?)Python
- 爬取 boss 直聘技術崗並分析