Invariance of results when scaling explanatory variables in logistic regression, is there a proof?
$begingroup$
There is a standard result for linear regression that the regression coefficients are given by
$$mathbf{beta}=(mathbf{X^T X})^{-1}mathbf{X^T y}$$
or
$(mathbf{X^T X})mathbf{beta}=mathbf{X^T y} tag{2}label{eq2}$
Scaling the explanatory variables does not affect the predictions. I have tried to show this algebraically as follows.
The response is related to the explanatory variables via the matrix equation
$mathbf{y}=mathbf{X beta} tag{3}label{eq3}$
$mathbf{X}$ is an $n times (p+1)$ matrix of n observations on p explanatory variables. The first column of $mathbf{X}$ is a column of ones.
Scaling the explanatory variables with a $(p+1) times (p+1) $ diagonal matrix $mathbf{D}$, whose entries are the scaling factors
$
mathbf{X^s} = mathbf{XD} tag{4}label{eq4}$
$mathbf{X^s}$ and $mathbf{beta^s}$ satisfy $eqref{eq2}$:
$$(mathbf{D^TX^T XD})mathbf{beta^s} =mathbf{D^TX^T y}$$
so
$$mathbf{X^T XD}mathbf{beta^s} =mathbf{X^T y}$$
$$Rightarrow mathbf{D beta^s} = (mathbf{X^T X)^{-1}}mathbf{X^T y}=mathbf{beta}$$
$Rightarrow mathbf{beta^s}=mathbf{D}^{-1}mathbf{beta} tag{5}label{eq5}$
This means if an explanatory variable is scaled by $d_i$ then the regression coefficient $beta_i$is scaled by $1/d_i$ and the effect of the scaling cancels out, i.e.
considering predictions based on scaled values, and using $eqref{eq4},eqref{eq5},eqref{eq3}$
$$mathbf{y^s}=mathbf{X^s beta^s} = mathbf{X D D^{-1}beta}=mathbf{X beta}=mathbf{y}$$
as expected.
Now to the question.
For logistic regression without any regularization, it is suggested, by doing regressions with and without scaling the same effect is seen
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.8331 0.4304
mtcars$mpg <- mtcars$mpg * 10
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.83307 0.04304
When the variable mpg is scaled up by 10, the corresponding coefficient is scaled down by 10.
- How could this scaling property be proved (or disproved ) algebraically for logistic regression?
I found a similar question relating to the effect on AUC when regularization is used.
- Is there any point to scaling explanatory variables in logistic regression, in the absence of regularization?
Thanks.
regression logistic regression-coefficients
$endgroup$
add a comment |
$begingroup$
There is a standard result for linear regression that the regression coefficients are given by
$$mathbf{beta}=(mathbf{X^T X})^{-1}mathbf{X^T y}$$
or
$(mathbf{X^T X})mathbf{beta}=mathbf{X^T y} tag{2}label{eq2}$
Scaling the explanatory variables does not affect the predictions. I have tried to show this algebraically as follows.
The response is related to the explanatory variables via the matrix equation
$mathbf{y}=mathbf{X beta} tag{3}label{eq3}$
$mathbf{X}$ is an $n times (p+1)$ matrix of n observations on p explanatory variables. The first column of $mathbf{X}$ is a column of ones.
Scaling the explanatory variables with a $(p+1) times (p+1) $ diagonal matrix $mathbf{D}$, whose entries are the scaling factors
$
mathbf{X^s} = mathbf{XD} tag{4}label{eq4}$
$mathbf{X^s}$ and $mathbf{beta^s}$ satisfy $eqref{eq2}$:
$$(mathbf{D^TX^T XD})mathbf{beta^s} =mathbf{D^TX^T y}$$
so
$$mathbf{X^T XD}mathbf{beta^s} =mathbf{X^T y}$$
$$Rightarrow mathbf{D beta^s} = (mathbf{X^T X)^{-1}}mathbf{X^T y}=mathbf{beta}$$
$Rightarrow mathbf{beta^s}=mathbf{D}^{-1}mathbf{beta} tag{5}label{eq5}$
This means if an explanatory variable is scaled by $d_i$ then the regression coefficient $beta_i$is scaled by $1/d_i$ and the effect of the scaling cancels out, i.e.
considering predictions based on scaled values, and using $eqref{eq4},eqref{eq5},eqref{eq3}$
$$mathbf{y^s}=mathbf{X^s beta^s} = mathbf{X D D^{-1}beta}=mathbf{X beta}=mathbf{y}$$
as expected.
Now to the question.
For logistic regression without any regularization, it is suggested, by doing regressions with and without scaling the same effect is seen
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.8331 0.4304
mtcars$mpg <- mtcars$mpg * 10
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.83307 0.04304
When the variable mpg is scaled up by 10, the corresponding coefficient is scaled down by 10.
- How could this scaling property be proved (or disproved ) algebraically for logistic regression?
I found a similar question relating to the effect on AUC when regularization is used.
- Is there any point to scaling explanatory variables in logistic regression, in the absence of regularization?
Thanks.
regression logistic regression-coefficients
$endgroup$
add a comment |
$begingroup$
There is a standard result for linear regression that the regression coefficients are given by
$$mathbf{beta}=(mathbf{X^T X})^{-1}mathbf{X^T y}$$
or
$(mathbf{X^T X})mathbf{beta}=mathbf{X^T y} tag{2}label{eq2}$
Scaling the explanatory variables does not affect the predictions. I have tried to show this algebraically as follows.
The response is related to the explanatory variables via the matrix equation
$mathbf{y}=mathbf{X beta} tag{3}label{eq3}$
$mathbf{X}$ is an $n times (p+1)$ matrix of n observations on p explanatory variables. The first column of $mathbf{X}$ is a column of ones.
Scaling the explanatory variables with a $(p+1) times (p+1) $ diagonal matrix $mathbf{D}$, whose entries are the scaling factors
$
mathbf{X^s} = mathbf{XD} tag{4}label{eq4}$
$mathbf{X^s}$ and $mathbf{beta^s}$ satisfy $eqref{eq2}$:
$$(mathbf{D^TX^T XD})mathbf{beta^s} =mathbf{D^TX^T y}$$
so
$$mathbf{X^T XD}mathbf{beta^s} =mathbf{X^T y}$$
$$Rightarrow mathbf{D beta^s} = (mathbf{X^T X)^{-1}}mathbf{X^T y}=mathbf{beta}$$
$Rightarrow mathbf{beta^s}=mathbf{D}^{-1}mathbf{beta} tag{5}label{eq5}$
This means if an explanatory variable is scaled by $d_i$ then the regression coefficient $beta_i$is scaled by $1/d_i$ and the effect of the scaling cancels out, i.e.
considering predictions based on scaled values, and using $eqref{eq4},eqref{eq5},eqref{eq3}$
$$mathbf{y^s}=mathbf{X^s beta^s} = mathbf{X D D^{-1}beta}=mathbf{X beta}=mathbf{y}$$
as expected.
Now to the question.
For logistic regression without any regularization, it is suggested, by doing regressions with and without scaling the same effect is seen
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.8331 0.4304
mtcars$mpg <- mtcars$mpg * 10
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.83307 0.04304
When the variable mpg is scaled up by 10, the corresponding coefficient is scaled down by 10.
- How could this scaling property be proved (or disproved ) algebraically for logistic regression?
I found a similar question relating to the effect on AUC when regularization is used.
- Is there any point to scaling explanatory variables in logistic regression, in the absence of regularization?
Thanks.
regression logistic regression-coefficients
$endgroup$
There is a standard result for linear regression that the regression coefficients are given by
$$mathbf{beta}=(mathbf{X^T X})^{-1}mathbf{X^T y}$$
or
$(mathbf{X^T X})mathbf{beta}=mathbf{X^T y} tag{2}label{eq2}$
Scaling the explanatory variables does not affect the predictions. I have tried to show this algebraically as follows.
The response is related to the explanatory variables via the matrix equation
$mathbf{y}=mathbf{X beta} tag{3}label{eq3}$
$mathbf{X}$ is an $n times (p+1)$ matrix of n observations on p explanatory variables. The first column of $mathbf{X}$ is a column of ones.
Scaling the explanatory variables with a $(p+1) times (p+1) $ diagonal matrix $mathbf{D}$, whose entries are the scaling factors
$
mathbf{X^s} = mathbf{XD} tag{4}label{eq4}$
$mathbf{X^s}$ and $mathbf{beta^s}$ satisfy $eqref{eq2}$:
$$(mathbf{D^TX^T XD})mathbf{beta^s} =mathbf{D^TX^T y}$$
so
$$mathbf{X^T XD}mathbf{beta^s} =mathbf{X^T y}$$
$$Rightarrow mathbf{D beta^s} = (mathbf{X^T X)^{-1}}mathbf{X^T y}=mathbf{beta}$$
$Rightarrow mathbf{beta^s}=mathbf{D}^{-1}mathbf{beta} tag{5}label{eq5}$
This means if an explanatory variable is scaled by $d_i$ then the regression coefficient $beta_i$is scaled by $1/d_i$ and the effect of the scaling cancels out, i.e.
considering predictions based on scaled values, and using $eqref{eq4},eqref{eq5},eqref{eq3}$
$$mathbf{y^s}=mathbf{X^s beta^s} = mathbf{X D D^{-1}beta}=mathbf{X beta}=mathbf{y}$$
as expected.
Now to the question.
For logistic regression without any regularization, it is suggested, by doing regressions with and without scaling the same effect is seen
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.8331 0.4304
mtcars$mpg <- mtcars$mpg * 10
fit <- glm(vs ~ mpg, data=mtcars,family=binomial)
print(fit)
Coefficients:
(Intercept) mpg
-8.83307 0.04304
When the variable mpg is scaled up by 10, the corresponding coefficient is scaled down by 10.
- How could this scaling property be proved (or disproved ) algebraically for logistic regression?
I found a similar question relating to the effect on AUC when regularization is used.
- Is there any point to scaling explanatory variables in logistic regression, in the absence of regularization?
Thanks.
regression logistic regression-coefficients
regression logistic regression-coefficients
asked 12 hours ago
PM.PM.
2021210
2021210
add a comment |
add a comment |
2 Answers
2
active
oldest
votes
$begingroup$
Here is a heuristic idea:
The likelihood for a logistic regression model is
$$ ell(beta|y) propto prod_ileft(frac{exp(x_i'beta)}{1+exp(x_i'beta)}right)^{y_i}left(frac{1}{1+exp(x_i'beta)}right)^{1-y_i}
$$
and the MLE is the arg max of that likelihood. When you scale a regressor, you also need to accordingly scale the coefficients to achieve the original maximal likelihood.
$endgroup$
$begingroup$
That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling?
$endgroup$
– PM.
11 hours ago
2
$begingroup$
Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles).
$endgroup$
– Christoph Hanck
11 hours ago
add a comment |
$begingroup$
Christoph has a great answer (+1). Just writing this because I can't comment there.
The crucial point here is that the likelihood only depends on the coefficients $beta$ through the linear term $X beta$. This makes the likelihood unable to distinguish between "$X beta$" and $(XD) (D^{-1}beta)$", causing the invariance you've noticed.
To be specific about this, we need to introduce some notation (which we can do since we're writing an answer!). Let $y_i | x_i stackrel{ind.}{sim} mathrm{bernoulli}left[ mathrm{logit}^{-1} (x_i^T beta) right]$ be independent draws according to the logistic regression model, where $x_i in mathbb{R}^{p+1}$ is the measured covariates. Write the likelihood of the $i^{th}$ observation as $l(y_i, x_i^T beta)$.
To introduce the change of coordinates, write $bar{x}_i = D x_i$, where $D$ is diagonal matrix with all diagonal entries nonzero. By definition of maximum likelihood estimation, we know that maximum likelihood estimators $hat{beta}$ of the data ${y_i | x_i}$ satisfy that $$sum_{i=1}^n l(y_i, x_i^T beta) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) tag{1}$$ for all coefficients $beta in mathbb{R}^p$, and that maximum likelihood estimators for the data ${y_i | bar{x}_i}$ satisfy that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) leq sum_{i=1}^n l(y_i, bar{x}_i^T hatalpha) tag{2}$$ for all coefficients $alpha in mathbb{R}^p$.
In your argument, you used a closed form of the maximum likelihood estimator to derive the result. It turns out, though, (as Cristoph suggested above), all you need to do is work with the likelihood. Let $hat{beta}$ be a maximum likelihood estimator of the data ${y_i | x_i}$. Now, writing $beta = D alpha$, we can use equation (1) to show that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) = sum_{i=1}^n lleft(y_i, (x_i^T D) (D^{-1} beta)right) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) = sum_{i=1}^n l(y_i, bar{x}_i^T D^{-1} hat{beta}).$$ That is, $D^{-1} hat{beta}$ satisfies equation (2) and is therefore a maximum likelihood estimator with respect to the data ${y_i | bar{x}_i}$. This is the invariance property you noticed.
(For what it's worth, there's a lot of room for generalizing this argument beyond logistic regression: did we need independent observations? did we need the matrix $D$ to be diagonal? did we need a binary response? did we need the use logit? What notation would you change for this argument to work in different scenarios?)
New contributor
$endgroup$
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
return StackExchange.using("mathjaxEditing", function () {
StackExchange.MarkdownEditor.creationCallbacks.add(function (editor, postfix) {
StackExchange.mathjaxEditing.prepareWmdForMathJax(editor, postfix, [["$", "$"], ["\\(","\\)"]]);
});
});
}, "mathjax-editing");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "65"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f399318%2finvariance-of-results-when-scaling-explanatory-variables-in-logistic-regression%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
2 Answers
2
active
oldest
votes
2 Answers
2
active
oldest
votes
active
oldest
votes
active
oldest
votes
$begingroup$
Here is a heuristic idea:
The likelihood for a logistic regression model is
$$ ell(beta|y) propto prod_ileft(frac{exp(x_i'beta)}{1+exp(x_i'beta)}right)^{y_i}left(frac{1}{1+exp(x_i'beta)}right)^{1-y_i}
$$
and the MLE is the arg max of that likelihood. When you scale a regressor, you also need to accordingly scale the coefficients to achieve the original maximal likelihood.
$endgroup$
$begingroup$
That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling?
$endgroup$
– PM.
11 hours ago
2
$begingroup$
Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles).
$endgroup$
– Christoph Hanck
11 hours ago
add a comment |
$begingroup$
Here is a heuristic idea:
The likelihood for a logistic regression model is
$$ ell(beta|y) propto prod_ileft(frac{exp(x_i'beta)}{1+exp(x_i'beta)}right)^{y_i}left(frac{1}{1+exp(x_i'beta)}right)^{1-y_i}
$$
and the MLE is the arg max of that likelihood. When you scale a regressor, you also need to accordingly scale the coefficients to achieve the original maximal likelihood.
$endgroup$
$begingroup$
That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling?
$endgroup$
– PM.
11 hours ago
2
$begingroup$
Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles).
$endgroup$
– Christoph Hanck
11 hours ago
add a comment |
$begingroup$
Here is a heuristic idea:
The likelihood for a logistic regression model is
$$ ell(beta|y) propto prod_ileft(frac{exp(x_i'beta)}{1+exp(x_i'beta)}right)^{y_i}left(frac{1}{1+exp(x_i'beta)}right)^{1-y_i}
$$
and the MLE is the arg max of that likelihood. When you scale a regressor, you also need to accordingly scale the coefficients to achieve the original maximal likelihood.
$endgroup$
Here is a heuristic idea:
The likelihood for a logistic regression model is
$$ ell(beta|y) propto prod_ileft(frac{exp(x_i'beta)}{1+exp(x_i'beta)}right)^{y_i}left(frac{1}{1+exp(x_i'beta)}right)^{1-y_i}
$$
and the MLE is the arg max of that likelihood. When you scale a regressor, you also need to accordingly scale the coefficients to achieve the original maximal likelihood.
answered 11 hours ago
Christoph HanckChristoph Hanck
17.8k34274
17.8k34274
$begingroup$
That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling?
$endgroup$
– PM.
11 hours ago
2
$begingroup$
Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles).
$endgroup$
– Christoph Hanck
11 hours ago
add a comment |
$begingroup$
That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling?
$endgroup$
– PM.
11 hours ago
2
$begingroup$
Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles).
$endgroup$
– Christoph Hanck
11 hours ago
$begingroup$
That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling?
$endgroup$
– PM.
11 hours ago
$begingroup$
That's useful +1 thank-you. Anything to add regarding the second point about the reasons for doing (or not doing) the scaling?
$endgroup$
– PM.
11 hours ago
2
2
$begingroup$
Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles).
$endgroup$
– Christoph Hanck
11 hours ago
$begingroup$
Not sure. Given that in practice, MLEs are computed via numerical (e.g. Newton-Raphson) algorithms, they might be more susceptible to stability issues when regressors live on extremely different scales. Also, of course, different users may prefer different units of measurement (say, km vs miles).
$endgroup$
– Christoph Hanck
11 hours ago
add a comment |
$begingroup$
Christoph has a great answer (+1). Just writing this because I can't comment there.
The crucial point here is that the likelihood only depends on the coefficients $beta$ through the linear term $X beta$. This makes the likelihood unable to distinguish between "$X beta$" and $(XD) (D^{-1}beta)$", causing the invariance you've noticed.
To be specific about this, we need to introduce some notation (which we can do since we're writing an answer!). Let $y_i | x_i stackrel{ind.}{sim} mathrm{bernoulli}left[ mathrm{logit}^{-1} (x_i^T beta) right]$ be independent draws according to the logistic regression model, where $x_i in mathbb{R}^{p+1}$ is the measured covariates. Write the likelihood of the $i^{th}$ observation as $l(y_i, x_i^T beta)$.
To introduce the change of coordinates, write $bar{x}_i = D x_i$, where $D$ is diagonal matrix with all diagonal entries nonzero. By definition of maximum likelihood estimation, we know that maximum likelihood estimators $hat{beta}$ of the data ${y_i | x_i}$ satisfy that $$sum_{i=1}^n l(y_i, x_i^T beta) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) tag{1}$$ for all coefficients $beta in mathbb{R}^p$, and that maximum likelihood estimators for the data ${y_i | bar{x}_i}$ satisfy that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) leq sum_{i=1}^n l(y_i, bar{x}_i^T hatalpha) tag{2}$$ for all coefficients $alpha in mathbb{R}^p$.
In your argument, you used a closed form of the maximum likelihood estimator to derive the result. It turns out, though, (as Cristoph suggested above), all you need to do is work with the likelihood. Let $hat{beta}$ be a maximum likelihood estimator of the data ${y_i | x_i}$. Now, writing $beta = D alpha$, we can use equation (1) to show that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) = sum_{i=1}^n lleft(y_i, (x_i^T D) (D^{-1} beta)right) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) = sum_{i=1}^n l(y_i, bar{x}_i^T D^{-1} hat{beta}).$$ That is, $D^{-1} hat{beta}$ satisfies equation (2) and is therefore a maximum likelihood estimator with respect to the data ${y_i | bar{x}_i}$. This is the invariance property you noticed.
(For what it's worth, there's a lot of room for generalizing this argument beyond logistic regression: did we need independent observations? did we need the matrix $D$ to be diagonal? did we need a binary response? did we need the use logit? What notation would you change for this argument to work in different scenarios?)
New contributor
$endgroup$
add a comment |
$begingroup$
Christoph has a great answer (+1). Just writing this because I can't comment there.
The crucial point here is that the likelihood only depends on the coefficients $beta$ through the linear term $X beta$. This makes the likelihood unable to distinguish between "$X beta$" and $(XD) (D^{-1}beta)$", causing the invariance you've noticed.
To be specific about this, we need to introduce some notation (which we can do since we're writing an answer!). Let $y_i | x_i stackrel{ind.}{sim} mathrm{bernoulli}left[ mathrm{logit}^{-1} (x_i^T beta) right]$ be independent draws according to the logistic regression model, where $x_i in mathbb{R}^{p+1}$ is the measured covariates. Write the likelihood of the $i^{th}$ observation as $l(y_i, x_i^T beta)$.
To introduce the change of coordinates, write $bar{x}_i = D x_i$, where $D$ is diagonal matrix with all diagonal entries nonzero. By definition of maximum likelihood estimation, we know that maximum likelihood estimators $hat{beta}$ of the data ${y_i | x_i}$ satisfy that $$sum_{i=1}^n l(y_i, x_i^T beta) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) tag{1}$$ for all coefficients $beta in mathbb{R}^p$, and that maximum likelihood estimators for the data ${y_i | bar{x}_i}$ satisfy that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) leq sum_{i=1}^n l(y_i, bar{x}_i^T hatalpha) tag{2}$$ for all coefficients $alpha in mathbb{R}^p$.
In your argument, you used a closed form of the maximum likelihood estimator to derive the result. It turns out, though, (as Cristoph suggested above), all you need to do is work with the likelihood. Let $hat{beta}$ be a maximum likelihood estimator of the data ${y_i | x_i}$. Now, writing $beta = D alpha$, we can use equation (1) to show that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) = sum_{i=1}^n lleft(y_i, (x_i^T D) (D^{-1} beta)right) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) = sum_{i=1}^n l(y_i, bar{x}_i^T D^{-1} hat{beta}).$$ That is, $D^{-1} hat{beta}$ satisfies equation (2) and is therefore a maximum likelihood estimator with respect to the data ${y_i | bar{x}_i}$. This is the invariance property you noticed.
(For what it's worth, there's a lot of room for generalizing this argument beyond logistic regression: did we need independent observations? did we need the matrix $D$ to be diagonal? did we need a binary response? did we need the use logit? What notation would you change for this argument to work in different scenarios?)
New contributor
$endgroup$
add a comment |
$begingroup$
Christoph has a great answer (+1). Just writing this because I can't comment there.
The crucial point here is that the likelihood only depends on the coefficients $beta$ through the linear term $X beta$. This makes the likelihood unable to distinguish between "$X beta$" and $(XD) (D^{-1}beta)$", causing the invariance you've noticed.
To be specific about this, we need to introduce some notation (which we can do since we're writing an answer!). Let $y_i | x_i stackrel{ind.}{sim} mathrm{bernoulli}left[ mathrm{logit}^{-1} (x_i^T beta) right]$ be independent draws according to the logistic regression model, where $x_i in mathbb{R}^{p+1}$ is the measured covariates. Write the likelihood of the $i^{th}$ observation as $l(y_i, x_i^T beta)$.
To introduce the change of coordinates, write $bar{x}_i = D x_i$, where $D$ is diagonal matrix with all diagonal entries nonzero. By definition of maximum likelihood estimation, we know that maximum likelihood estimators $hat{beta}$ of the data ${y_i | x_i}$ satisfy that $$sum_{i=1}^n l(y_i, x_i^T beta) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) tag{1}$$ for all coefficients $beta in mathbb{R}^p$, and that maximum likelihood estimators for the data ${y_i | bar{x}_i}$ satisfy that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) leq sum_{i=1}^n l(y_i, bar{x}_i^T hatalpha) tag{2}$$ for all coefficients $alpha in mathbb{R}^p$.
In your argument, you used a closed form of the maximum likelihood estimator to derive the result. It turns out, though, (as Cristoph suggested above), all you need to do is work with the likelihood. Let $hat{beta}$ be a maximum likelihood estimator of the data ${y_i | x_i}$. Now, writing $beta = D alpha$, we can use equation (1) to show that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) = sum_{i=1}^n lleft(y_i, (x_i^T D) (D^{-1} beta)right) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) = sum_{i=1}^n l(y_i, bar{x}_i^T D^{-1} hat{beta}).$$ That is, $D^{-1} hat{beta}$ satisfies equation (2) and is therefore a maximum likelihood estimator with respect to the data ${y_i | bar{x}_i}$. This is the invariance property you noticed.
(For what it's worth, there's a lot of room for generalizing this argument beyond logistic regression: did we need independent observations? did we need the matrix $D$ to be diagonal? did we need a binary response? did we need the use logit? What notation would you change for this argument to work in different scenarios?)
New contributor
$endgroup$
Christoph has a great answer (+1). Just writing this because I can't comment there.
The crucial point here is that the likelihood only depends on the coefficients $beta$ through the linear term $X beta$. This makes the likelihood unable to distinguish between "$X beta$" and $(XD) (D^{-1}beta)$", causing the invariance you've noticed.
To be specific about this, we need to introduce some notation (which we can do since we're writing an answer!). Let $y_i | x_i stackrel{ind.}{sim} mathrm{bernoulli}left[ mathrm{logit}^{-1} (x_i^T beta) right]$ be independent draws according to the logistic regression model, where $x_i in mathbb{R}^{p+1}$ is the measured covariates. Write the likelihood of the $i^{th}$ observation as $l(y_i, x_i^T beta)$.
To introduce the change of coordinates, write $bar{x}_i = D x_i$, where $D$ is diagonal matrix with all diagonal entries nonzero. By definition of maximum likelihood estimation, we know that maximum likelihood estimators $hat{beta}$ of the data ${y_i | x_i}$ satisfy that $$sum_{i=1}^n l(y_i, x_i^T beta) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) tag{1}$$ for all coefficients $beta in mathbb{R}^p$, and that maximum likelihood estimators for the data ${y_i | bar{x}_i}$ satisfy that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) leq sum_{i=1}^n l(y_i, bar{x}_i^T hatalpha) tag{2}$$ for all coefficients $alpha in mathbb{R}^p$.
In your argument, you used a closed form of the maximum likelihood estimator to derive the result. It turns out, though, (as Cristoph suggested above), all you need to do is work with the likelihood. Let $hat{beta}$ be a maximum likelihood estimator of the data ${y_i | x_i}$. Now, writing $beta = D alpha$, we can use equation (1) to show that $$sum_{i=1}^n l(y_i, bar{x}_i^T alpha) = sum_{i=1}^n lleft(y_i, (x_i^T D) (D^{-1} beta)right) leq sum_{i=1}^n l(y_i, x_i^T hatbeta) = sum_{i=1}^n l(y_i, bar{x}_i^T D^{-1} hat{beta}).$$ That is, $D^{-1} hat{beta}$ satisfies equation (2) and is therefore a maximum likelihood estimator with respect to the data ${y_i | bar{x}_i}$. This is the invariance property you noticed.
(For what it's worth, there's a lot of room for generalizing this argument beyond logistic regression: did we need independent observations? did we need the matrix $D$ to be diagonal? did we need a binary response? did we need the use logit? What notation would you change for this argument to work in different scenarios?)
New contributor
edited 7 hours ago
New contributor
answered 7 hours ago
user551504user551504
412
412
New contributor
New contributor
add a comment |
add a comment |
Thanks for contributing an answer to Cross Validated!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
Use MathJax to format equations. MathJax reference.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstats.stackexchange.com%2fquestions%2f399318%2finvariance-of-results-when-scaling-explanatory-variables-in-logistic-regression%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown