Provide feedback on how many submissions will be invalidated when republishing

[LesRamer]

Problem

As a kata author, I introduced a fix for a reported issue. My fix disallowed a problematic thing that manifested as the reported issue. The fix proved the wrong thing to do since after adding the fix I later discovered that it had invalidated 90% of the solutions. (So I removed the test that disallowed the problematic thing and found another acceptable approach which maintained the integrity of the kata and its existing solutions.)

Proposal

Introduce a confirmation step/stage where the kata-author is provided with information about invalidation stats (counts and percentage-of-the-whole for: newly invalidated solutions, existing invalidated solutions, reinstated solutions) for kata which are not in Beta. Foreseeing the impact might have provided me a way out, by helping me to recognize and also abandon a bad thing without (in this case temporarily) impacting those who had solved my kata.

Thoughts?

The only big drawback seems the work/delay of rerunning all the solutions. Also, with the random tests that most kata have, this could mean inconsistent pass/fail and inconsistent stats from one run to the next.

[hobovsky]

The only big drawback seems the work/delay of rerunning all the solutions.

Would you have an estimate how big would this drawback be? For example, how many solutions did your kata have at the time of republishing? How difficult would it be to create such statistic for your kata at the time, or in general?

[hobovsky]

I personally think that efforts should go in exactly opposite direction: authors and maintainers should hesitate less when improving quality of the content and not refrain from improvements to save something of such a miniscule value like solutions to a kata which is expected to collect them anew anyway. I would say that removing bugs and problems from kata is much more valuable than keeping solutions. Many people gain from improved quality, and no one loses anything by having a solution invalidated. Keeping old solutions means condoning to the past and affects (in a hardly noticeable way) a finite amount of users. Fixing a kata affects the future and improves lives of much larger amount of users: solvers who wont be affected by issues anymore, and maintainers who won;t have to reply to support requests anymore.

Granted, situation could look differently for difficult kata where every solution can be interesting in its own way. But anything white and yellow, I would expect it to get solutions back in not a long time. Currently many kata are bad because maintainers refrain from improving them concerned by losing old solutions. But what is even wrong in changing "round to 4 decimal places"+assert_equals to a proper fuzzy equality comparison in a 6 kyu kata with 1000 solutions from users who don't even care about them and are never going to look back at them?

Let's just stop worrying about useless things like old solutions and lets not let them hold us from doing actually useful stuff beneficial to everyone, like fixing problems in kata.

[LesRamer]

I really appreciate your well articulated counterpoint. How can we encourage a cultural shift like that happen? Doesn't the system notify folks whose submissions were invalidated by kata updates? People experience loss as having greater weight than gains even when psychological studies have constructed them to be identical. I have more questions than answers.