npm search _something_ doesn't work

[thegiantbeast]

If I execute: npm search _my_package_ it gives me an error:

npm ERR! SyntaxError: Unexpected token C
npm ERR! Cannot GET /-/all/since?stale=update_after&startkey=1386675384891
npm ERR!
npm ERR! at Object.parse (native)
npm ERR! at RegClient. (/usr/local/lib/node_modules/npm/node_modules/npm-registry-client/lib/request.js:235:23)
npm ERR! at Request.self.callback (/usr/local/lib/node_modules/npm/node_modules/request/index.js:142:22)
npm ERR! at Request.EventEmitter.emit (events.js:98:17)
npm ERR! at Request. (/usr/local/lib/node_modules/npm/node_modules/request/index.js:856:14)
npm ERR! at Request.EventEmitter.emit (events.js:117:20)
npm ERR! at IncomingMessage. (/usr/local/lib/node_modules/npm/node_modules/request/index.js:808:12)
npm ERR! at IncomingMessage.EventEmitter.emit (events.js:117:20)
npm ERR! at _stream_readable.js:895:16
npm ERR! at process._tickCallback (node.js:415:13)

It seems that the output from sinopia is not compatible with the parser of npm-search (I only tried with a non-existent package, I can't test with an existing one right now).
This are my current versions:

node --version
v0.10.5

npm --version
1.2.18

sinopia --version
0.5.5

[rlidwka]

"Unexpected token C" is from "Cannot GET /-/all/...", it's default express 404 handler. This feature is missing entirely.

I'd say it's a "patch welcome" kind of thing.

What is expected to be in the output anyway? Should somebody be able to search in local (restricted) packages? Or just proxy an output from npmjs.org and get it over with?

[thegiantbeast]

I would say that people can search yeah, the same way that they can install them, so I don't see the point to restrict access. But local packages would have priority over npmjs.org.

I can write a patch for it if you agree with this workflow :)

[rlidwka]

I can write a patch for it if you agree with this workflow :)

Sinopia does not use database. Currently each package is separated to it's own folder on the filesystem. If you try to implement search on all local packages, it'll either require some kind of a database to keep track of all packages (bye simplicity) or open a door to a DoS.

So I don't see how complete search can be implemented at this point.

[yannickcr]

A solution for a complete search can be to make the search on registry.npmjs.org then, for private packages, search in ./storage for directories with a mtime newer than the startkey in the search request (this can easily be done on *nix with find, more complicated on Windows). Finally, append their informations to the registry.npmjs.org response (using the package.json).

There is certainly some edge cases to address (first build with an empty local index, how reliable can be the mtime ?) but it worked fine in my tests and it does not require a database.

[jalateras]

At least it should proxy the request to http://registry.npmjs.org

[thegiantbeast]

@rlidwka if your concern is DoS what it can be done is create some kind of cached response which would have a limited lifetime and that lifetime could be controlled by the config.yml like:

cache_search: 5000 (5s)

That way even if someone tried to hack the system it would work properly.
But as you can see, most of the guys that are requesting these feature are for "internal" use, so DoS generally isn't an issue I believe.

Does someone have a better/different solution?

[ghost]

We could alternatively limit the number of concurrent search requests to something sane (configurable) and return a HTTP 503 if the limit is exceeded. For normal use cases this is unlikely to happen and in the case of DoS - only the search feature would be affected.