Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update README.md #25

Merged
merged 1 commit into from
Feb 2, 2024
Merged

Update README.md #25

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,12 @@
> 本项目仅供学习交流使用, 请勿用于商业及非法用途, 由此引起的一切后果与作者无关



https://github.com/iiicebearrr/spiders-for-all/assets/110714291/32e098d3-c85e-44c9-b54c-4820365b0235




# Menu

- [Menu](#menu)
Expand Down Expand Up @@ -139,4 +145,4 @@ pip install spiders-for-all # python 版本 >= 3.12
- [x] (已解决)小红书爬取用户投稿的笔记时, 由于小红书签名算法的问题尚未解决, 只能爬取用户投稿的首页数据, 需要下拉加载的数据暂时无法爬取(签名算法借鉴了[Spider-XHS](https://github.com/cv-cat/Spider_XHS.git))
- [x] 低版本的sqlite可能不支持`ON CONFLICT DO UPDATE`语法, 如果遇到该问题请尝试升级sqlite版本
- [x] 大批量下载视频时可能偶现下载中断导致失败的情况, 这种情况请在所有任务下载完成后, 运行`grep ERROR logs/bilibili.log`可以查看有哪些视频下载失败, 然后收集bvid重新下载即可。未来会考虑处理这种情况
- [x] (已解决)bilibili近期(2024-01-20左右)更新了用户投稿接口的反爬策略, 新增了`dm_img_inter`参数, 目前作者还没有时间实现该参数的生成算法, 所以该接口暂时无法使用, 但是如果你实现了该参数的生成算法, 可以通过修改`spiders_for_all.spiders.bilibili.AuthorSpider`中的`get_request_args`方法来继续使用该接口
- [x] (已解决)bilibili近期(2024-01-20左右)更新了用户投稿接口的反爬策略, 新增了`dm_img_inter`参数, 目前作者还没有时间实现该参数的生成算法, 所以该接口暂时无法使用, 但是如果你实现了该参数的生成算法, 可以通过修改`spiders_for_all.spiders.bilibili.AuthorSpider`中的`get_request_args`方法来继续使用该接口
Loading