per_class_accuracy too low

[dorothy400]

Hi,
I have been training the model with about 5000 images. The labels in my images are barriers, symbols of bicycle lanes and bicycle lanes. The training log shows that per_class_accuracy is very low for several classes. Increase of iterations didn't seem to be able to improve the accuracy.
I am wondering where the problems might be and how to improve the model.

Please see the log below.

I0913 13:33:09.075178 5758 solver.cpp:228] Iteration 14100, loss = 0.057436
I0913 13:33:09.075223 5758 solver.cpp:244] Train net output #0: accuracy = 0.936784
I0913 13:33:09.075237 5758 solver.cpp:244] Train net output #1: loss = 0.0574362 (* 1 = 0.0574362 loss)
I0913 13:33:09.075245 5758 solver.cpp:244] Train net output #2: per_class_accuracy = 0.935188
I0913 13:33:09.075251 5758 solver.cpp:244] Train net output #3: per_class_accuracy = 0
I0913 13:33:09.075258 5758 solver.cpp:244] Train net output #4: per_class_accuracy = 0
I0913 13:33:09.075264 5758 solver.cpp:244] Train net output #5: per_class_accuracy = 0
I0913 13:33:09.075270 5758 solver.cpp:244] Train net output #6: per_class_accuracy = 0.983284
I0913 13:33:09.075279 5758 sgd_solver.cpp:106] Iteration 14100, lr = 0.001
I0913 13:33:27.954026 5758 solver.cpp:228] Iteration 14120, loss = 0.103527
I0913 13:33:27.954146 5758 solver.cpp:244] Train net output #0: accuracy = 0.933682
I0913 13:33:27.954170 5758 solver.cpp:244] Train net output #1: loss = 0.103527 (* 1 = 0.103527 loss)
I0913 13:33:27.954183 5758 solver.cpp:244] Train net output #2: per_class_accuracy = 0.934803
I0913 13:33:27.954195 5758 solver.cpp:244] Train net output #3: per_class_accuracy = 0.728985
I0913 13:33:27.954205 5758 solver.cpp:244] Train net output #4: per_class_accuracy = 0
I0913 13:33:27.954216 5758 solver.cpp:244] Train net output #5: per_class_accuracy = 0.843473
I0913 13:33:27.954226 5758 solver.cpp:244] Train net output #6: per_class_accuracy = 0
I0913 13:33:27.954241 5758 sgd_solver.cpp:106] Iteration 14120, lr = 0.001
I0913 13:33:45.547899 5758 solver.cpp:228] Iteration 14140, loss = 0.0768097
I0913 13:33:45.547950 5758 solver.cpp:244] Train net output #0: accuracy = 0.918521
I0913 13:33:45.547963 5758 solver.cpp:244] Train net output #1: loss = 0.0768099 (* 1 = 0.0768099 loss)
I0913 13:33:45.547971 5758 solver.cpp:244] Train net output #2: per_class_accuracy = 0.91189
I0913 13:33:45.547977 5758 solver.cpp:244] Train net output #3: per_class_accuracy = 0
I0913 13:33:45.547983 5758 solver.cpp:244] Train net output #4: per_class_accuracy = 0
I0913 13:33:45.547989 5758 solver.cpp:244] Train net output #5: per_class_accuracy = 0
I0913 13:33:45.547996 5758 solver.cpp:244] Train net output #6: per_class_accuracy = 0.994607
I0913 13:33:45.548004 5758 sgd_solver.cpp:106] Iteration 14140, lr = 0.001
I0913 13:34:03.113821 5758 solver.cpp:228] Iteration 14160, loss = 0.149893
I0913 13:34:03.114506 5758 solver.cpp:244] Train net output #0: accuracy = 0.961221
I0913 13:34:03.114523 5758 solver.cpp:244] Train net output #1: loss = 0.149893 (* 1 = 0.149893 loss)
I0913 13:34:03.114531 5758 solver.cpp:244] Train net output #2: per_class_accuracy = 0.963949
I0913 13:34:03.114543 5758 solver.cpp:244] Train net output #3: per_class_accuracy = 0
I0913 13:34:03.114550 5758 solver.cpp:244] Train net output #4: per_class_accuracy = 0
I0913 13:34:03.114557 5758 solver.cpp:244] Train net output #5: per_class_accuracy = 0.938256
I0913 13:34:03.114563 5758 solver.cpp:244] Train net output #6: per_class_accuracy = 0
I0913 13:34:03.114573 5758 sgd_solver.cpp:106] Iteration 14160, lr = 0.001
I0913 13:34:20.676877 5758 solver.cpp:228] Iteration 14180, loss = 0.105678
I0913 13:34:20.676918 5758 solver.cpp:244] Train net output #0: accuracy = 0.912685
I0913 13:34:20.676931 5758 solver.cpp:244] Train net output #1: loss = 0.105679 (* 1 = 0.105679 loss)
I0913 13:34:20.676939 5758 solver.cpp:244] Train net output #2: per_class_accuracy = 0.912661
I0913 13:34:20.676944 5758 solver.cpp:244] Train net output #3: per_class_accuracy = 0
I0913 13:34:20.676951 5758 solver.cpp:244] Train net output #4: per_class_accuracy = 0.922742
I0913 13:34:20.676957 5758 solver.cpp:244] Train net output #5: per_class_accuracy = 0
I0913 13:34:20.676964 5758 solver.cpp:244] Train net output #6: per_class_accuracy = 0
I0913 13:34:20.676971 5758 sgd_solver.cpp:106] Iteration 14180, lr = 0.001
^CI0913 13:34:25.132899 5758 solver.cpp:454] Snapshotting to binary proto file /home/hdp/SegNet-Tutorial-master/wtRun/Training/segnet-bike-all40000_iter_14186.caffemodel
I0913 13:34:26.044842 5758 sgd_solver.cpp:273] Snapshotting solver state to binary proto file /home/hdp/SegNet-Tutorial-master/wtRun/Training/segnet-bike-all40000_iter_14186.solverstate
I0913 13:34:26.259492 5758 solver.cpp:301] Optimization stopped early.
I0913 13:34:26.274672 5758 caffe.cpp:254] Optimization Done.