java根据url抓取html页面内容，怎么解析chunked 方式

Question

如题所述

Answer 1

先获取页面
String html = getContent(url, Constants.ENCODING_UTF8);
解析页面 Document doc=Jsoup.parse(html);
然后你获取相应的标签String tag =doc.getElementsByTag("title").first().text();
如果标签很多不一样你就得判断了,还有看看有什么相同的地方吧,我抓取网页数据的时候最烦的就是格式不一样的,好多标签不一样的,只能判断,找到共同点,个别的单个处理,

Answer 2

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URL;

public class Test {

public static String getHtml(String urlString) {
try {
StringBuffer html = new StringBuffer();
URL url = new URL(urlString);
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
InputStreamReader isr = new InputStreamReader(conn.getInputStream());
BufferedReader br = new BufferedReader(isr);
String temp;
while ((temp = br.readLine()) != null) {
html.append(temp).append("\n");
}
br.close();
isr.close();
return html.toString();
} catch (Exception e) {
e.printStackTrace();
return null;
}
}

public static void main(String[] args) {
System.out.println(Test.getHtml(http://192.168.1.1:8080/test/test.jsp));
}

}