Add more checks in Travis CI

[say-yawn]

About this Issue

To make the PR tests be more robust and comprehensive we should add additional checks/tests in Travis CI to check for the following:

Acceptance Criteria

• Check the diff between shavar-prod-lists and disconnect-tracking-protection and have Travis CI fail if it ever finds domains in our list that aren't in Disconnect's
• Check the list files updated (run shavar-list-creation) from the changes in the shavar-prod-lists, see Get diff between branches to indicate list changes #171 for more details
• Check that the size of the files updated does not exceed the max file size (4Mb), see Add File Size Checks from the Latest Changes  #170 for more details.
• Remove the "tag" counts in the json_verify.py test as it is an outdated format

[dtlight]

hello @skim1102, I'd love to work on this! May I?

[say-yawn]

@dave-light yes! Please let me know if you have any questions.

[dtlight]

Hi @skim1102, the 4th acceptance criteria has been satisfied.

I'd just like to clarify that in the 1st acceptance criteria, where you mention disconnect-tracking-protection, are you referring to disconnect-blacklist.json (are we talking about disconnect-entitylist.json too)?
I'm considering adding another argument in to the run() method in json_verify.py so it takes disconnect-blacklist.json and compares it to the shavar-prod-lists. I'd love to know your thoughts on whether or not I'm going about this the right way.

[say-yawn]

@dave-light I linked a work-in-progress (WIP) PR in shavar-list-creation repo that will do most of what the first acceptance criteria:

Check the diff between shavar-prod-lists and disconnect-tracking-protection and have Travis CI fail if it ever finds domains in our list that aren't in Disconnect's

is asking for. You are welcome to take this code and refactor to suit the need of the first criterion. Does this answer your question?

[dtlight]

@skim1102 it does indeed. Thanks 😃

[dtlight]

Hi @skim1102, this is more of a sanity check and to ensure I've understood your code correctly.

Looking at category_domains.py, I see on line 44 a txt file called domain-diff-category is created. I also see a comment printed to the txt (such as line 53) beginning with exclamation marks.

Is it ok for me to assume that all comments like this will begin with an exclamation mark? The reason I ask is that anything else in the txt should only be there because a difference has been detected and so I can cause Travis CI to fail if it finds this.

Please correct me if I've stated something that isn't quite right, or doesn't make sense.

[say-yawn]

☝️ #154 fulfills the a/c:

Remove the "tag" counts in the json_verify.py test as it is an outdated format

[say-yawn]

@dave-light, rather than using the .txt file as is to check for the differences, I suggest using the code to check for the differences between the Shavar and Disconnect lists and assert that there are no differences. And if there is a difference between the two, the assertion should fail and indicate what the differences are as it was done here and here for the block lists. Same should be applied to the entity lists.

Did you also look at where you will implement this? To follow the infrastructure seen in this Travis build, I suggest moving this script to scripts folder in shavar-prod-lists. Wdyt?

[dtlight]

@skim1102 I was a little unsure about where to implement it. But I agree with moving category_domains.py to scripts in shavar-prod-lists.

I'm happy to add the code to shavar-prod-lists in my next commit, please let me know if you'd rather move it across yourself.

[dtlight]

@skim1102 I noticed a handful of unused variables in category_domains.py (e.g. content_domains, fingerprinting_domains, sorted_data), shall I tidy the code up by removing them? Or are they supposed to be used somewhere?

[dtlight]

@skim1102 category_domains requires imports from list2safebrowsing, it also requires disconnect_mapping.json, both of which are in shavar-list-creation. Initially, I thought I had figured out a way to 'import' or at least reference them by reading https://stackoverflow.com/questions/9357442/github-linking-with-other-repos but I'm a little reserved about following any of the suggestions there. Do you have any suggestions? I don't want to duplicate the required files but would it be feasible to move them to shavar-prod-lists?

[shreyagupta30]

@dave-light @skim1102 will it be okay if I take up one of the to-do of this issue?
I am interested to take "Check the diff between shavar-prod-lists and disconnect-tracking-protection and have Travis CI fail if it ever finds domains in our list that aren't in Disconnect's"

[dtlight]

@dave-light @skim1102 will it be okay if I take up one of the to-do of this issue?
I am interested to take "Check the diff between shavar-prod-lists and disconnect-tracking-protection and have Travis CI fail if it ever finds domains in our list that aren't in Disconnect's"

Hey @shreyagupta30, thanks for your interest. I've been writing code to address that piece of acceptance criteria. I have a few questions regarding my approach, which I'm waiting for @skim1102 to get back to me on.

[shreyagupta30]

All right, thank you so much for letting me know. :)

[svensevenslow]

@skim1102 This task was listed as internship task in the outreachy project description. Will this still be a part of the outreachy project or not? Some of the subtasks seem to be done

[say-yawn]

@svensevenslow yes there are additional Travis CI check work needed and will be reasonable to assume that it will still be part of the Outreachy project. I have created #170 and #171 to indicate those work will be additional work that need to be done.