How to choose loss functions



Let me explicitly state at the very beginning that I'm a complete beginner in neural networks. So, please bear with me if this question appears to be too naive.

There are lots of loss functions available. My problem is that I don't know all of them. Hence, I'm confused to choose between them while implementing a model. I understand that it's preferable to use MSE, if I want to minimise the Euclidean distance between the source and the target. Likewise for Manhattan distance, the choice should be MAE.

So, I was wondering whether there are some references where one can get the basic ideas (definition) about different loss functions, and the situations in which they perform better (or at least, theoretically more justifiable and/or preferable) than the others, and more importantly, why. Unfortunately, I couldn't find anything as such. The best I found is in the PyTorch documentation. This provides definitions almost all the loss functions provided in PyTorch, but not their justifications.

So, if someone can provide me such references, or can explain the loss functions (at least some of them), it will be really helpful.



Simple references:

DON'T make the rookie mistake of going through all, or even two of them, at the same time. Pick one and stick to it. It'll be boring but rewarding.

Resources which make me happy :slight_smile:

(note: the last one doesn't cover neural networks, but it does explain various loss functions well).


Thanks a lot :smiley:


You're welcome! Hey, I didn't notice you're the same guy who asked this nice question:

It looks like you're studying at the ISI. How is it? It looks like a nice college, you do ask some smart questions!


It's nice, actually. In India, it's the best institution for Statistics, more inclined to the theoretical and probabilistic sides. The applied topics are offered mainly in the final year.

closed #6

This topic was automatically closed 7 days after the last reply. New replies are no longer allowed.