Using isURL to check if the TLD is valid

[ssmirr]

I believe validating TLDs is not supported in isURL (unless I missed something in the README). I see require_tld: true option but no require_valid_tld: true.

I have a use case for validating TLD and if this sounds like something that you would consider merging, I can work on sending a PR to add this feature?

[profnandaa]

New TLDs keep being added every other day, maintaining this will be a little hard. Or do you want to have the user supply their allowed TLDs? That could be a viable option I think 🤔

[ssmirr]

I was thinking to use this list to know which TLDs are valid. But you're right, keeping this up to date will require additional regular maintenance. I'm not sure which one is better though, having a possibly out of date list which contains most of the TLDs, or not having a list? 🤔

I think Supplying allowed TLDs could work and sounds like a simple and viable way to handle this!

[profnandaa]

I see. Or we could start with that list, and then have user supply an additional list? ./na

On Wed, Nov 6, 2019 at 7:00 AM Samim Mirhosseini ***@***.***> wrote: I was thinking to use this list <https://data.iana.org/TLD/tlds-alpha-by-domain.txt> to know which TLDs are valid. But you're right, keeping this up to date will require additional regular maintenance. I'm not sure which one is better though, having a possibly out of date list which contains most of the TLDs, or not having a list? 🤔 I think Supplying allowed TLDs could work and sounds like a simple and viable way to handle this! — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1193?email_source=notifications&email_token=AAB7ZEKRHBLQWT2JBK422OTQSLLXHA5CNFSM4JIBTPTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDG2MPI#issuecomment-550348349>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB7ZENWJX25CFCH6L2FCSTQSLLXHANCNFSM4JIBTPTA> .

-- Sent from a tiny device while on the move.

[AubreyHewes]

Cleaner would be just allowing the developer to supply a list i.e. then they can decide which list to apply.

Including support for the IANA gTLD list would entail caching it or including it per version (something the library should not do; out of scope). It does not change that often, but, including it just produces more issues.

• require_tld could be a boolean (bc) or an array and the developer supplies the array of allowed tld values.. if it is an array then the current tld check (there is a TLD) is performed and then a check on the TLD in the supplied array. How, why or which values the developer uses is superfluous.

#1247

[isik-kaplan]

Hello! I'm writing a very small service to allow users to scrape some data from websites they enter.

I'm trying to use isURL and then fetch it, but I'd prefer to just trust that it is a valid url and send requests to valid urls instead of fetching a bunch of random tlds.

I can write a ts decorator or a hoc to simply disregard errors in fetch but I'd rather not sending the request if it is definitely going to end up as an error. So this feature would be really useful in my opinion.

[AubreyHewes]

@isik-kaplan You are wanting to push your application logic in to a library. This is not the way.

A valid URL is already tested. You want to test a valid URL also has a registered/known TLD. This can only be tested by validating against the IANA list (which can change). Including this list in the library is pure unnecessary bloat; bloat for all consumers. Introducing an unnecessary maintenance burden (as it has to be updated).

Your "very small service" could do this itself after a valid URL validation.. or implement a secondary library/service that validates TLD after a valid URL.

[isik-kaplan]

Of course it can do it itself, it can also simply not use a library. That's not the point, though.

I imagine people who validates urls don't validate them for the sake of validating them. They most probably are using the urls somehow after validation, and I am willing to bet that most of the usage after validating is to send a request to validated urls.

If libraries stopped including features because some dependency(in this case it would be a tld list and not some other code) could change in the feature, we probably would have a lot less libraries.

If validatorjs is willing for this:

Cleaner would be just allowing the developer to supply a list i.e. then they can decide which list to apply.

• require_tld could be a boolean (bc) or an array and the developer supplies the array of allowed tld values.. if it is an array then the current tld check (there is a TLD) is performed and then a check on the TLD in the supplied array. How, why or which values the developer uses is superfluous.

kind of feature, I'm willing to create a package that has the tld list for developers to validate against.

But validating first, then validating again doesn't make sense to me, so even in this version, I think it is best that validatorjs at least allows users to give a tld list to validate against.

Edi: Also, no idea why did you went "very small service", instead of simply saying your service.

[profnandaa]

I'm a little at home with the idea of a user supplying a TLD list, better than the overhead of us maintaining such... I don't know what the rest think? ./na

On Tue, Dec 1, 2020 at 6:13 AM Işık Kaplan ***@***.***> wrote: Of course it can do it itself, it can also simply not use a library. That's not the point, though. I imagine people who validates urls don't validate them for the sake of validating them. They most probably are using the urls somehow after validation, and I am willing to bet that most of the usage after validating is to send a request to validated urls. If libraries stopped including features because some dependency(in this case it would be a tld list and not some other code) could change in the feature, we probably would have a lot less libraries. If validatorjs is willing for this: Cleaner would be just allowing the developer to supply a list i.e. then they can decide which list to apply. - require_tld could be a boolean (bc) or an array and the developer supplies the array of allowed tld values.. if it is an array then the current tld check (there is a TLD) is performed and then a check on the TLD in the supplied array. How, why or which values the developer uses is superfluous. kind of feature, I'm willing to create a package that has the tld list for developers to validate against. But validating first, then validating again doesn't make sense to me, so even in this version, I think it is best that validatorjs at least allows users to give a tld list to validate against. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1193 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAB7ZEJCVHOHLJXFRUXUWYDSSRNPHANCNFSM4JIBTPTA> .

-- Sent from a tiny device while on the move.

[AubreyHewes]

@isik-kaplan I must have completely misunderstood you, sorry about that!

Including a TLD list within the library is pure unnecessary bloat and an extra maintenance overhead. The current validation itself is an URL validation.. i.e. You know the URL is a valid URL (as it is tested according to the spec; spec has no place for specific tlds).

In your case; You are expecting that an URL has a TLD.. this is not the case. A valid URL is a valid URL (by spec).

My proposal (that you quoted) was to allow adding an array of tlds to the validation method.. so a library consumer can apply their own TLD requirements applied after the spec URL requirement.

note "very small service" was just quoting yourself... @isik-kaplan

[YPCrumble]

Would maintainers consider a user-supplied TLD list but also allow configuring a default dependency like this repo which seems to be updated and would keep the periodic update of the TLD list out of scope of this library? https://github.com/stephenmathieson/node-tlds

[braaar]

Would maintainers consider a user-supplied TLD list but also allow configuring a default dependency like this repo which seems to be updated and would keep the periodic update of the TLD list out of scope of this library? https://github.com/stephenmathieson/node-tlds

Wouldn't it be fairly trivial to implement your own method that wraps the isUrl function and takes care of passing the node-tlds list to validator.js' isUrl for you?

[YPCrumble]

Thank you for your note @braaar - I did some more research and agree - I'm likely going to use global-tld-list per this helpful snippet in the Yup documentation: jquense/yup#784 (comment)