Java 简单网络爬虫

1、首先我们先创建一个maven项目，找到Mavne Project ，然后Next。

2、勾选Create a simple project (skip archetype selection) 然后Next。

3、在Group框写域名，在ArtifactId里写项目名。 然后Finish。这个maven项目就创建完了。

4、然后就是创建项目包了，如图：

5、构建pom.xml:<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> <modelVersion>4.0.0</modelVersion> <groupId>com.baidu</groupId> <artifactId>PaChong1</artifactId> <version>0.0.1-SNAPSHOT</version> <packaging>jar</packaging> <!-- 在maven项目(jar或war都可以)的pom.xml中加入如下代码，导入jar包，我用的是阿里云的镜像库 --> <dependencies> <dependency> <groupId>org.apache.httpcomponents</groupId> <artifactId>httpclient</artifactId> <version>4.3.1</version> </dependency> <!-- SLF4J --> <dependency> <groupId>org.slf4j</groupId> <artifactId>slf4j-api</artifactId> <version>1.7.21</version> </dependency> <!-- Logback --> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-core</artifactId> <version>1.1.3</version> </dependency> <dependency> <groupId>ch.qos.logback</groupId> <artifactId>logback-classic</artifactId> <version>1.1.3</version> </dependency> <dependency> <groupId>org.slf4j</groupId> <artifactId>log4j-over-slf4j</artifactId> <version>1.7.21</version> </dependency> <!-- https://mvnrepository.com/artifact/org.projectlombok/lombok --> <dependency> <groupId>org.projectlombok</groupId> <artifactId>lombok</artifactId> <version>1.16.20</version> </dependency> </dependencies></project>

6、创建完毕，整个项目框架如图，然后在com.baidu里建方法类。

7、写类名,这个看个人爱好：

8、在类里写实现代码：package com.baidu;import java.io.BufferedReader;import java.io.IOException;import java.io.InputStreamReader;import org.apache.http.HttpEntity;import org.apache.http.HttpStatus;import org.apache.http.client.methods.CloseableHttpResponse;import org.apache.http.client.methods.HttpGet;import org.apache.http.impl.client.CloseableHttpClient;import org.apache.http.impl.client.HttpClients;public class HttpGetUtils { public static void main(String[] args) { // TODO Auto-generated method stub //https://v.qq.com/ http://www.youku.com/ http://m.sunlands.com String str=get("http://m.sunlands.com"); System.out.println(str); } private static String get(String url) { String result = ""; try { //获取httpclient实例 CloseableHttpClient httpclient = HttpClients.createDefault(); //获取方法实例。GET HttpGet httpGet = new HttpGet(url); //执行方法得到响应 CloseableHttpResponse response = httpclient.execute(httpGet); try { //如果正确执行而且返回值正确，即可解析 if (response != null && response.getStatusLine().getStatusCode() == HttpStatus.SC_OK) { System.out.println(response.getStatusLine()); HttpEntity entity = response.getEntity(); //从输入流中解析结果 result = readResponse(entity, "utf-8"); } } finally { httpclient.close(); response.close(); } }catch (Exception e){ e.printStackTrace(); } return result; } private static String readResponse(HttpEntity resEntity, String charset) { StringBuffer res = new StringBuffer(); BufferedReader reader = null; try { if (resEntity == null) { return null; } reader = new BufferedReader(new InputStreamReader( resEntity.getContent(), charset)); String line = null; while ((line = reader.readLine()) != null) { res.append(line); } } catch (Exception e) { e.printStackTrace(); } finally { try { if (reader != null) { reader.close(); } } catch (IOException e) { } } return res.toString(); }}

9、运行出现这个就说明我们成功了。