Autolab partial feedback endpoints #1645
Conversation
…aming-output-frontend
…b/Autolab into streaming-output-frontend
… problem that has an autograder result, redirect to that page
| } else if (!res.is_assigned) { | ||
| /* if there's no partial feedback and no queue position, | ||
| refresh the page to see if full feedback is available */ | ||
| location.reload(); |
There was a problem hiding this comment.
I think this is the line that is causing the refresh loop. Basically I think my tango is acting weird (partialfeedback endpoint was not setup properly, so it doesn't work), and thus there is no partial feedback and no queue position.
There was a problem hiding this comment.
I have figured out how to replicate this bug:
- Set up Tango without the PR that introduces partial feedback
- Submit an assignment normally
- Because partial feedback isn't working (the endpoint doesn't exist), so it gives
There was a problem hiding this comment.
Basically, there should be error handling when the partial feedback endpoint throws an error.
Here's what currently happens when autograding fails (other parts of it)
|
Tangentially related: With the removal of the
This is because the div has a
So that it renders properly
Note: I did a grep and this seems to be the only instance of a |
There was a problem hiding this comment.
Followed testing plan, semantic config works too, code generally looks good
Requesting the following changes
- Add
s12to handin formdiv - Update logic for
@problemReleased - Remove extraneous schema changes
- Address off-center rotation of refresh button
Commented some other nits too
| <p> | ||
| Press the refresh button to get updates. | ||
| </p> | ||
| <% else %> |
There was a problem hiding this comment.
Nit: The format for an autograder success seems pretty different from the format for an autograder error / manual grading:
Success:
Manual grading
Error:
Could we make the formats consistent?
(Rough mockup)
| } else if (res.queue_position != null) { | ||
| $('#partial-feedback').html(` | ||
| <div class="autograding-in-progress"> | ||
| <pre>You are at position ${(res.queue_position) + 1} / ${(res.queue_length)} in the autograding queue.</pre> |
There was a problem hiding this comment.
Nit: Do students need to know / care about the queue_length? Perhaps displaying only queue_position would be sufficient.
| Your submission is currently in the autograding queue, waiting to be graded. | ||
| </p> | ||
| <p> | ||
| Press the refresh button to get updates. |
There was a problem hiding this comment.
Nit: Click is more semantically correct than press for a software button
Description
Calls the Tango partial output and live jobs endpoints from Autolab to render more detailed job status and display partial output.
New feedback view states
In Progress submissions:
Users are able to press a refresh button to get updated partial output, which gets displayed in the feedback section. The refresh button is rate-limited from the frontend such that users may only call a refresh every 5 seconds. If there is no longer any partial output, the refresh will trigger a page reload to check for full output.
Queued submissions:
If a submission is queued, a "queued" page will get displayed, as well as the student's queue position. There is similarly a rate-limited refresh button.
Completed submissions:
Once autograding is complete, the feedback is displayed as normal, with no refresh button.
Notable behavior changes
When the autograder is still running, the
--in the grades table temporarily becomes clickable. If users click into it, they are led to the view feedback page, where the possible views mentioned above are shown.After the autograder finishes running, the auto-graded scores are updated to numbers and remain clickable. Non-auto-graded scores go back to the original behavior.
If a user has clicked into the viewFeedback page of a non-auto-graded problem while the file was still being autograded, and then refreshes on it after autograding is complete, they will be redirected with a "No feedback for requested score" error.
Motivation and Context
This change allows students to view feedback as soon as it is updated, rather than just at the end. This is especially important for classes with long test cases, where students may not want to wait until their code is completely done to be able to see individual test case results.
How Has This Been Tested?
Setup
redis-server,python3 restful_tango/server.py,python jobManager.py)Test scenarios:
Types of changes
Checklist:
overcommit --install && overcommit --signto use pre-commit hook for linting