My loss function fluctuates like crazy.

[ysig]

First of all thanks a lot for this wavenet code, well written and handed to the community.
My problem is that I cannot understand, if my model is learning and what is it learning.
I am testing it on musical audio samples, keeping silences and with the default settings I see the loss fluctuating like crazy from 3.5 to 0.0.
I cannot understand what that means and how I can fix it or improve it.
Also although it learns fast, I am not sure if it actually reads all the data and what amount of it.
What is a good number of seconds (a.k.a number_of_samples = sample_rate * seconds) that I should use so me model generalizes more effectively.
Thanks in advance.