So this may be a place as good as any -- I've got a decent math background, and am self teaching myself ML while waiting for work to come in.
I'm working on undertstanding CNNs, and I can't seem to find the answer (read: don't know what terms to look for) that explain how you train the convolutional weights.
But in practice, I assume you would want to have these actual weights themselves trained, no?
But, in CNNs, the same convolutional step is executed on the entire input to the convolutional step, you just move around where you take your "inputs".
How do you do the training, then? Do you just do backprop on each variable of the convolution stem from its output, with a really small learning rate, then repeat after shifting over to the next output?
Sorry if this seems like a poorly thought out question, I'm definitely not phrasing this perfectly.
It turns out a convolution can be thought of like a special type of matrix
https://en.wikipedia.org/wiki/Toeplitz_matrix#Discrete_convo...
You could imagine do the backprop through the matrix multiply, and you are going to get several terms with the same coefficent, since the diagonals are the same - and you could just sum up all the gradients from the relevant terms and use that.
Huh, this almost got it for me after ten seconds of looking at it. Still have questions about the nitty gritty, but huge matrix multiplication clearly isn't how this is done in production.
That's right, you basically want to add the contributions of each gradient of the output volume to get the full picture of how a weight affected the entire output.
I'm working on undertstanding CNNs, and I can't seem to find the answer (read: don't know what terms to look for) that explain how you train the convolutional weights.
For instance, a blur might be
[[ 0 0.125 0 ] , [ 0.125 0.5 0.125 ] , [0 0.125 0]]
But in practice, I assume you would want to have these actual weights themselves trained, no?
But, in CNNs, the same convolutional step is executed on the entire input to the convolutional step, you just move around where you take your "inputs".
How do you do the training, then? Do you just do backprop on each variable of the convolution stem from its output, with a really small learning rate, then repeat after shifting over to the next output?
Sorry if this seems like a poorly thought out question, I'm definitely not phrasing this perfectly.