Odd size revamp

[sanjanag]

1. The major change in this PR is the logic behind odd size issue type.
   Previous logic:
   Any image which is x times larger/smaller than the median size in the dataset, where x was hardcoded to 10.
   New logic:
   Now we compute the q1 = 25th percentile and q3 = 75th percentile for the dataset
   An image is marked as oddly sized if size > q1 + 3 * IQR or size < q3 - 3 * IQR.

For scoring the issues, distance of each image is computed from the midpoint, where midpoint = (q1+q3) / 2 and normalized such that the images within the range have a score of 0.5 or less and outlier images have score > 0.5. There is one edge case where q1 = median = q3 and in this case the images with size = median are assigned a score of 1.0 and rest of scores are scaled accordingly, for this case the threshold is 1.0. This threshold can be supplied as a hyperparameter.

Since cleanvision shows low scores for problematic images, these scores are flipped and clipped between 0 and 1.

2. Another change is the output of imagelab.info['statistics']. Earlier this had raw values of image properties, now it contains statistics from pd.Series.describe() instead of values for the whole dataset.

Here's an example of how it would look

3. In the visualization of odd sized images, changed the title to size instead of original size
4. Changed the signature of internal method ImageProperty.mark_issue from
   def mark_issue(self, scores: pd.DataFrame, threshold: float, issue_type: str): to
   def mark_issue(self, scores: pd.DataFrame, issue_type: str, threshold: Optional[float] = None)
   This makes threshold arg optional more visibly.

Results on cleanvision test/demo dataset

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (53ee321) 95.93% compared to head (5d51a9f) 96.10%.
Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #247      +/-   ##
==========================================
+ Coverage   95.93%   96.10%   +0.16%     
==========================================
  Files          16       16              
  Lines         985     1001      +16     
  Branches      194      195       +1     
==========================================
+ Hits          945      962      +17     
+ Misses         20       19       -1     
  Partials       20       20              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[elisno]

LGTM.

Just curious if you think a size distribution that is multi-modal (e.g. mostly 8Mb and 12Mb images, but a handful of, say, 16Mb images) should work well with this IQR rule?

[elisno]

Should this link be "stable" or are these the same?

[elisno]

But both should work.

[sanjanag]

Tutorial installs the latest version (main branch), so I think this link is good.

[sanjanag]

Just curious if you think a size distribution that is multi-modal (e.g. mostly 8Mb and 12Mb images, but a handful of, say, 16Mb images) should work well with this IQR rule?

It wouldn't unless you change the iqr hyperparameter. Say, most of the images are 8MB and 12MB such that 25% is 8 and 75% is 12, this would not detect 16MB since the iqr multiplier is 3, I used 3 instead of 1.5 the standard one, as it looked like a better threshold across the benchmark datasets I tested with. 1.5 gave too many examples as odd sized.