Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mAP zero while training #82

Closed
1 task done
rexayyy opened this issue Aug 20, 2022 · 13 comments · Fixed by ultralytics/yolov5#9056
Closed
1 task done

mAP zero while training #82

rexayyy opened this issue Aug 20, 2022 · 13 comments · Fixed by ultralytics/yolov5#9056
Labels
bug Something isn't working

Comments

@rexayyy
Copy link

rexayyy commented Aug 20, 2022

Search before asking

  • I have searched the HUB issues and found no similar bug report.

HUB Component

Training

Bug

@glenn-jocher the issue still prevails as i have tried training using a different dataset and even different model and tried both hub and custom colab notebook

Environment

No response

Minimal Reproducible Example

No response

Additional

No response

@rexayyy rexayyy added the bug Something isn't working label Aug 20, 2022
@glenn-jocher
Copy link
Member

@rexayyy this isn't a HUB issue, this is related to YOLOv5 and has been resolved in ultralytics/yolov5#9037, so all you need to do is start from a fresh Colab notebook or update your local YOLOv5, i.e. git pull from inside your YOLOv5 directory.

@rexayyy
Copy link
Author

rexayyy commented Aug 20, 2022

I will try now and let you know right now

@rexayyy
Copy link
Author

rexayyy commented Aug 20, 2022

Screen Shot 2022-08-21 at 4 04 55 AM
it is still the same

@glenn-jocher
Copy link
Member

@rexayyy yes I'm able to reproduce. Am investigating now.

Screenshot 2022-08-21 at 02 06 16

glenn-jocher added a commit to ultralytics/yolov5 that referenced this issue Aug 21, 2022
Resolves ultralytics/hub#82

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
glenn-jocher added a commit to ultralytics/yolov5 that referenced this issue Aug 21, 2022
Resolves ultralytics/hub#82

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
@glenn-jocher
Copy link
Member

@rexayyy I was able to fix a zero-map issue in ultralytics/yolov5#9056, but after training a new HUB model I'm getting a strange effect of later epochs showing zero map but the final val run showing correct mAP. It seems mAP-during training is still incorrect as in my original screenshot.

To receive this update:

  • Gitgit pull from within your yolov5/ directory or git clone https://github.com/ultralytics/yolov5 again
  • PyTorch Hub – Force-reload model = torch.hub.load('ultralytics/yolov5', 'yolov5s', force_reload=True)
  • Notebooks – View updated notebooks Open In Colab Open In Kaggle
  • Dockersudo docker pull ultralytics/yolov5:latest to update your image Docker Pulls

Thank you for spotting this issue and informing us of the problem. Please let us know if this update resolves the issue for you, and feel free to inform us of any other issues you discover or feature requests that come to mind. Happy trainings with YOLOv5 🚀!

@rexayyy
Copy link
Author

rexayyy commented Aug 21, 2022

I will do it again and let you know the progress

@rexayyy
Copy link
Author

rexayyy commented Aug 21, 2022

image
I am still getting the issue

@MGIbrahim
Copy link

I am also having the same problem now
Screenshot 2022-08-21 033303

@glenn-jocher
Copy link
Member

@rexayyy @MGIbrahim sorry about the persistent problems. I think I've resolved this now in ultralytics/yolov5#9058

I created a new HUB model and trained in a new Colab notebook (YOLOv5n on COCO128 for 10 epochs) and everything looks fine now.

Screenshot 2022-08-21 at 03 39 30

@MGIbrahim
Copy link

MGIbrahim commented Aug 21, 2022

@glenn-jocher Thanks for your quick response on this! I did a git pull and tried training again, but it still shows the mAP of 0.
Screenshot 2022-08-21 041514

@rexayyy
Copy link
Author

rexayyy commented Aug 21, 2022

Screen Shot 2022-08-21 at 7 21 38 AM
Yes, it is still not fixed. I don`t know what has happened as it was working totally fine a few hours ago.

@glenn-jocher
Copy link
Member

@MGIbrahim @rexayyy zero-mAP problem is being tracked down in ultralytics/yolov5#9059

@robotaiguy
Copy link

robotaiguy commented Aug 21, 2022

@glenn-jocher do you notice that in the hub display in the above post, his Images and Labels coutns are messed up like mine are? But in the Hub image a few posts up, they're normal. And I haven't used Hub yet, only Colab and local. I wonder if there's some off-by-1 error. But it shouldn't be with Hub, because I have yet to use it. I'm looking forward to trying it, but I haven't yet.

@glenn-jocher glenn-jocher reopened this Aug 21, 2022
ctjanuhowski pushed a commit to ctjanuhowski/yolov5 that referenced this issue Sep 8, 2022
Resolves ultralytics/hub#82

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>

Signed-off-by: Glenn Jocher <glenn.jocher@ultralytics.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants