How to choose loss functions

[Yarnabrina]

Hi!

Let me explicitly state at the very beginning that I'm a complete beginner in neural networks. So, please bear with me if this question appears to be too naive.

There are lots of loss functions available. My problem is that I don't know all of them. Hence, I'm confused to choose between them while implementing a model. I understand that it's preferable to use MSE, if I want to minimise the Euclidean distance between the source and the target. Likewise for Manhattan distance, the choice should be MAE.

So, I was wondering whether there are some references where one can get the basic ideas (definition) about different loss functions, and the situations in which they perform better (or at least, theoretically more justifiable and/or preferable) than the others, and more importantly, why. Unfortunately, I couldn't find anything as such. The best I found is in the PyTorch documentation. This provides definitions almost all the loss functions provided in PyTorch, but not their justifications.

So, if someone can provide me such references, or can explain the loss functions (at least some of them), it will be really helpful.

Thanks.

[Andrea]

Simple references:

https://www.deeplearningbook.org/

ESLII.pdf

Deep Learning with R

Deep Learning on Computational Accelerators

Deep Learning

Practical Deep Learning for Coders - Practical Deep Learning

https://www.amazon.com/Machine-Learning-Probabilistic-Perspective-Computation/dp/0262018020

DON'T make the rookie mistake of going through all, or even two of them, at the same time. Pick one and stick to it. It'll be boring but rewarding.

Resources which make me happy

https://www.amazon.com/Understanding-Machine-Learning-Theory-Algorithms/dp/1107057132/

https://www.amazon.com/dp/0262039400/

(note: the last one doesn't cover neural networks, but it does explain various loss functions well).

[Yarnabrina]

Thanks a lot

[Andrea]

You're welcome! Hey, I didn't notice you're the same guy who asked this nice question:

What does nnet minimise

I'm instructed by our professor to implement a neural network classifier with single hidden layer on the Wine Quality Data. This is divided in two parts, so I will implement two models and these models have ordered classes. I tried using nnet function in the nnet package for this assignment. I've noted that the process does not always converge, and if it does, they may converge to quite different values. I expected some differences, but those value are really far apart. I wanted to investigate and went through the documentation to know the convergence criterion. But I noted that that's not mentioned anywhere. So my question is that whether it minimises training set error, or it is splitting…

It looks like you're studying at the ISI. How is it? It looks like a nice college, you do ask some smart questions!

[Yarnabrina]

@Andrea

How is it?

It's nice, actually. In India, it's the best institution for Statistics, more inclined to the theoretical and probabilistic sides. The applied topics are offered mainly in the final year.