Is a neural network consisting of a single softmax classification layer only a linear classifier?
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty{ margin-bottom:0;
}
up vote
3
down vote
favorite
Since the softmax function is a generalization of the logistic function it is continuous and non-linear.
So the output of the softmax layer is: softmax( weight_matrix * input_activation)
weight_matrix * input_activation is purely linear combination of features.
The question is: if the application of the softmax activation still yields in a linear classifier or is the model then capable of representing non-linear functions?
neural-networks generalized-linear-model softmax
asked Nov 22 at 14:07
tamtam_
363
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
up vote
3
down vote
favorite
Since the softmax function is a generalization of the logistic function it is continuous and non-linear.
So the output of the softmax layer is: softmax( weight_matrix * input_activation)
weight_matrix * input_activation is purely linear combination of features.
The question is: if the application of the softmax activation still yields in a linear classifier or is the model then capable of representing non-linear functions?
neural-networks generalized-linear-model softmax
asked Nov 22 at 14:07
tamtam_
363
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
up vote
3
down vote
favorite
up vote
3
down vote
favorite
Since the softmax function is a generalization of the logistic function it is continuous and non-linear.
So the output of the softmax layer is: softmax( weight_matrix * input_activation)
weight_matrix * input_activation is purely linear combination of features.
The question is: if the application of the softmax activation still yields in a linear classifier or is the model then capable of representing non-linear functions?
neural-networks generalized-linear-model softmax
asked Nov 22 at 14:07
tamtam_
363
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Since the softmax function is a generalization of the logistic function it is continuous and non-linear.
So the output of the softmax layer is: softmax( weight_matrix * input_activation)
weight_matrix * input_activation is purely linear combination of features.
The question is: if the application of the softmax activation still yields in a linear classifier or is the model then capable of representing non-linear functions?
neural-networks generalized-linear-model softmax
neural-networks generalized-linear-model softmax
asked Nov 22 at 14:07
tamtam_
363
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 22 at 14:07
tamtam_
363
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 22 at 14:07
tamtam_
363
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked Nov 22 at 14:07
tamtam_
363
asked Nov 22 at 14:07
tamtam_
363
363
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
tamtam_ is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
up vote
6
down vote
A neural network with no hidden layers and a soft max output layer is exactly logistic regression (possibly with more than 2 classes), when trained to minimize categorical cross-entropy (equivalently maximize the log-likelihood of a multinomial model).
Your explanation is right on the money: a linear combination of inputs learns linear functions, and the soft max function yields a probability vector.
answered Nov 22 at 14:31
Sycorax
37.4k994183
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
6
down vote
A neural network with no hidden layers and a soft max output layer is exactly logistic regression (possibly with more than 2 classes), when trained to minimize categorical cross-entropy (equivalently maximize the log-likelihood of a multinomial model).
Your explanation is right on the money: a linear combination of inputs learns linear functions, and the soft max function yields a probability vector.
answered Nov 22 at 14:31
Sycorax
37.4k994183
add a comment |
up vote
6
down vote
A neural network with no hidden layers and a soft max output layer is exactly logistic regression (possibly with more than 2 classes), when trained to minimize categorical cross-entropy (equivalently maximize the log-likelihood of a multinomial model).
Your explanation is right on the money: a linear combination of inputs learns linear functions, and the soft max function yields a probability vector.
answered Nov 22 at 14:31
Sycorax
37.4k994183
add a comment |
up vote
6
down vote
up vote
6
down vote
A neural network with no hidden layers and a soft max output layer is exactly logistic regression (possibly with more than 2 classes), when trained to minimize categorical cross-entropy (equivalently maximize the log-likelihood of a multinomial model).
Your explanation is right on the money: a linear combination of inputs learns linear functions, and the soft max function yields a probability vector.
answered Nov 22 at 14:31
Sycorax
37.4k994183
A neural network with no hidden layers and a soft max output layer is exactly logistic regression (possibly with more than 2 classes), when trained to minimize categorical cross-entropy (equivalently maximize the log-likelihood of a multinomial model).
Your explanation is right on the money: a linear combination of inputs learns linear functions, and the soft max function yields a probability vector.
answered Nov 22 at 14:31
Sycorax
37.4k994183
edited Nov 22 at 15:36
answered Nov 22 at 14:31
Sycorax
37.4k994183
answered Nov 22 at 14:31
Sycorax
37.4k994183
answered Nov 22 at 14:31
Sycorax
37.4k994183
37.4k994183
add a comment |
add a comment |
tamtam_ is a new contributor. Be nice, and check out our Code of Conduct.
tamtam_ is a new contributor. Be nice, and check out our Code of Conduct.
tamtam_ is a new contributor. Be nice, and check out our Code of Conduct.
tamtam_ is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f378276%2fis-a-neural-network-consisting-of-a-single-softmax-classification-layer-only-a-l%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
