31 lines
2.0 KiB
Markdown
31 lines
2.0 KiB
Markdown
## 新闻网站爬虫
|
|
|
|
多线程爬取，获得的结构化数据包括标题、来源、时间、正文
|
|
|
|
### 支持的搜索页面
|
|
|
|
| 网站 | URL |
|
|
| -------- | ------------------------------------------------------------ |
|
|
| 新浪新闻 | [https://search.sina.com.cn](https://search.sina.com.cn) |
|
|
| 搜狗搜索 | [https://www.sogou.com/sogou?interation=1728053249&query=](https://www.sogou.com/sogou?interation=1728053249&query=) |
|
|
|
|
|
|
|
|
### 支持的网站页面
|
|
|
|
| 网站 | URL |
|
|
| ---------- | ------------------------------------------------------------ |
|
|
| 搜狐 | [www.sohu.com](www.sohu.com) |
|
|
| 网易 | [www.163.com](www.163.com), [dy.163.com](https://www.sogou.com/sogou?interation=1728053249&query=%s), [news.163.com](https://www.sogou.com/sogou?interation=1728053249&query=%s) |
|
|
| 腾讯 | [new.qq.com](https://www.sogou.com/sogou?interation=1728053249&query=%s) |
|
|
| 凤凰网 | [news.ifeng.com](https://www.sogou.com/sogou?interation=1728053249&query=%s), [i.ifeng.com](https://www.sogou.com/sogou?interation=1728053249&query=%s), [mil.ifeng.com](https://www.sogou.com/sogou?interation=1728053249&query=%s) |
|
|
| 中国新闻网 | www.chinanews.com |
|
|
| 中国青年网 | [news.youth.cn](https://www.sogou.com/sogou?interation=1728053249&query=%s) |
|
|
| 新华网 | [www.news.cn](www.news.cn), [xinhuanet.com](https://www.sogou.com/sogou?interation=1728053249&query=%s), [www.xinhuanet.com](www.xinhuanet.com) |
|
|
| 环球网 | [world.huanqiu.com](https://www.sogou.com/sogou?interation=1728053249&query=%s), [mil.huanqiu.com](https://www.sogou.com/sogou?interation=1728053249&query=%s), [china.huanqiu.com](https://www.sogou.com/sogou?interation=1728053249&query=%s) |
|
|
| 中华网 | [news.china.com](https://www.sogou.com/sogou?interation=1728053249&query=%s) |
|
|
|
|
### 环境
|
|
|
|
JDK17
|