{"id":456,"date":"2013-09-02T18:20:26","date_gmt":"2013-09-02T18:20:26","guid":{"rendered":"https:\/\/www.wdiam.com\/?p=456"},"modified":"2020-08-06T20:13:36","modified_gmt":"2020-08-06T20:13:36","slug":"deriving-the-linear-regression-solution","status":"publish","type":"post","link":"https:\/\/www.wdiam.com\/b\/2013\/09\/02\/deriving-the-linear-regression-solution\/","title":{"rendered":"Deriving the Linear Regression Solution"},"content":{"rendered":"

In deriving the linear regression solution, we will be taking a closer look at how we “solve” the common linear regression<\/a>, i.e., finding \\beta<\/span> in y = X\\beta + \\epsilon<\/span>.<\/p>\n

I mention “common,” because there are actually several ways you can get an estimate for \\beta<\/span> based on assumptions of your data and how you can correct for various anomalies. “Common” in this case specifically refers to ordinary least squares<\/a>. For this specific case, I assume you already know the punch line, that is, \\beta = (X^{T}X)^{-1}X^{T}y<\/span>. But, what we’re really interested in is how to get to that point.<\/p>\n

The crux is that you’re trying to find a solution \\beta<\/span> that minimizes the sum of the squared errors, i.e., \\min\\limits_{\\beta} \\: \\epsilon^{T}\\epsilon<\/span>. We can find the minimum by taking the derivative and setting it to zero, i.e., \\frac{d}{d\\beta} \\epsilon^{T}\\epsilon = 0<\/span>.<\/p>\n

In deriving the linear regression solution, it helps to remember two things. Regarding derivatives of two vectors, the product rule states that \\frac{d}{dx}u^{T}v = u^{T}\\frac{d}{dx}v + v^{T}\\frac{d}{dx}u<\/span>. See this<\/a> and that<\/a>. And, for matrix transpose, (AB)^{T} = B^{T}A^{T}<\/span>.<\/p>\n

Observe that y = X\\beta + \\epsilon \\implies \\epsilon = y - X\\beta<\/span>. As such, \\frac{d}{d\\beta} \\epsilon^{T}\\epsilon = \\frac{d}{d\\beta} (y-X\\beta)^{T}(y-X\\beta)<\/span>.<\/p>\n

Working it out,
\n\\frac{d}{d\\beta} \\epsilon^{T}\\epsilon \\\\= \\frac{d}{d\\beta} (y-X\\beta)^{T}(y-X\\beta) \\\\= (y-X\\beta)^{T} \\frac{d}{d\\beta}(y-X\\beta) + (y-X\\beta)^{T}\\frac{d}{d\\beta}(y-X\\beta) \\\\= (y-X\\beta)^{T}(-X) + (y-X\\beta)^{T}(-X) \\\\= -2(y-X\\beta)^{T}X \\\\= -2(y^{T} - \\beta^{T}X^{T})X \\\\= -2(y^{T}X - \\beta^{T}X^{T}X)<\/span><\/p>\n

By setting the derivative to zero and solving for \\beta<\/span>, we can find the \\beta<\/span> that minimizes the sum of squared errors.
\n\\frac{d}{d\\beta} \\epsilon^{T}\\epsilon = 0 \\\\ \\implies -2(y^{T}X - \\beta^{T}X^{T}X) = 0 \\\\ \\implies y^{T}X - \\beta^{T}X^{T}X = 0 \\\\ \\implies y^{T}X = \\beta^{T}X^{T}X \\\\ \\implies (y^{T}X)^{T} = (\\beta^{T}X^{T}X)^{T} \\\\ \\implies X^{T}y = X^{T}X\\beta \\\\ \\implies (X^{T}X)^{-1}X^{T}y = (X^{T}X)^{-1}(X^{T}X)\\beta \\\\ \\implies \\beta = (X^{T}X)^{-1}X^{T}y<\/span><\/p>\n

Without too much difficulty, we saw how we arrived at the linear regression solution of \\beta = (X^{T}X)^{-1}X^{T}y<\/span>. The general path to that derivation is to recognize that you’re trying to minimize the sum of squared errors (\\epsilon^{T}\\epsilon<\/span>), which can be done by finding the derivative of \\epsilon^{T}\\epsilon<\/span>, setting it to zero, and then solving for \\beta<\/span>.<\/p>\n","protected":false},"excerpt":{"rendered":"

In deriving the linear regression solution, we will be taking a closer look at how we “solve” the common linear regression, i.e., finding in . I mention “common,” because there are actually several ways you can get an estimate for based on assumptions of your data and how you can correct for various anomalies. “Common” … <\/p>\n

Continue reading “Deriving the Linear Regression Solution”<\/span><\/a><\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[3,6],"tags":[16,22,40],"class_list":["post-456","post","type-post","status-publish","format-standard","hentry","category-lesson","category-statistics","tag-derivation","tag-linear-regression","tag-statistics-2"],"_links":{"self":[{"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/posts\/456","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/comments?post=456"}],"version-history":[{"count":6,"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/posts\/456\/revisions"}],"predecessor-version":[{"id":597,"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/posts\/456\/revisions\/597"}],"wp:attachment":[{"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/media?parent=456"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/categories?post=456"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.wdiam.com\/b\/wp-json\/wp\/v2\/tags?post=456"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}