Skip to content

docs: correct concurrent method name in README #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

docs: correct concurrent method name in README #2

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -220,7 +220,7 @@ For each queue you can define the maximum number of concurrent http requests, ti
];

new Loader()
.setHttpThrottle(10) // Process 10 urls at a time
.setMaxConcurrentConnections(10) // Process 10 urls at a time
.setHttpTimeout(2000) // Retry loads that do not complete after 2 seconds
.setNumRetries(5) // Should any errors occur, retry 5 times. After this the load is flagged as error and will not have a result
.load(queue, function(err, loaderItems){
Expand Down Expand Up @@ -323,7 +323,7 @@ const saveToDB = function(loaderItem, result, next){
When loading hundreds or even thousands of pages from the same host eventually the requests are bound to be rejected. This happens for a number of reasons, typically because the host has a maximum number of requests per IP address per time unit.
I've tested the tool and have loaded 20,000+ pages from the same host, but only one request at a time. It took 8 hours but completed without any errors or warnings. Just go easy and set the http throttle to 1:
```js
myLoader.setHttpThrotte(1);
myLoader.setMaxConcurrentConnections(1);
```

### VPN
Expand Down