Deriving the Linear Regression Solution

In deriving the linear regression solution, we will be taking a closer look at how we “solve” the common linear regression, i.e., finding $\beta$ in $y = X\beta + \epsilon$.

I mention “common,” because there are actually several ways you can get an estimate for $\beta$ based on assumptions of your data and how you can correct for various anomalies. “Common” in this case specifically refers to ordinary least squares. For this specific case, I assume you already know the punch line, that is, $\beta = (X^{T}X)^{-1}X^{T}y$. But, what we’re really interested in is how to get to that point.

The crux is that you’re trying to find a solution $\beta$ that minimizes the sum of the squared errors, i.e., $\min\limits_{\beta} \: \epsilon^{T}\epsilon$. We can find the minimum by taking the derivative and setting it to zero, i.e., $\frac{d}{d\beta} \epsilon^{T}\epsilon = 0$.

In deriving the linear regression solution, it helps to remember two things. Regarding derivatives of two vectors, the product rule states that $\frac{d}{dx}u^{T}v = u^{T}\frac{d}{dx}v + v^{T}\frac{d}{dx}u$. See this and that. And, for matrix transpose, $(AB)^{T} = B^{T}A^{T}$.

Observe that $y = X\beta + \epsilon \implies \epsilon = y - X\beta$. As such, $\frac{d}{d\beta} \epsilon^{T}\epsilon = \frac{d}{d\beta} (y-X\beta)^{T}(y-X\beta)$.

Working it out,
$\frac{d}{d\beta} \epsilon^{T}\epsilon \\= \frac{d}{d\beta} (y-X\beta)^{T}(y-X\beta) \\= (y-X\beta)^{T} \frac{d}{d\beta}(y-X\beta) + (y-X\beta)^{T}\frac{d}{d\beta}(y-X\beta) \\= (y-X\beta)^{T}(-X) + (y-X\beta)^{T}(-X) \\= -2(y-X\beta)^{T}X \\= -2(y^{T} - \beta^{T}X^{T})X \\= -2(y^{T}X - \beta^{T}X^{T}X)$

By setting the derivative to zero and solving for $\beta$, we can find the $\beta$ that minimizes the sum of squared errors.
$\frac{d}{d\beta} \epsilon^{T}\epsilon = 0 \\ \implies -2(y^{T}X - \beta^{T}X^{T}X) = 0 \\ \implies y^{T}X - \beta^{T}X^{T}X = 0 \\ \implies y^{T}X = \beta^{T}X^{T}X \\ \implies (y^{T}X)^{T} = (\beta^{T}X^{T}X)^{T} \\ \implies X^{T}y = X^{T}X\beta \\ \implies (X^{T}X)^{-1}X^{T}y = (X^{T}X)^{-1}(X^{T}X)\beta \\ \implies \beta = (X^{T}X)^{-1}X^{T}y$

Without too much difficulty, we saw how we arrived at the linear regression solution of $\beta = (X^{T}X)^{-1}X^{T}y$. The general path to that derivation is to recognize that you’re trying to minimize the sum of squared errors ($\epsilon^{T}\epsilon$), which can be done by finding the derivative of $\epsilon^{T}\epsilon$, setting it to zero, and then solving for $\beta$.

Cost of Replacing a 2012 Ford Edge Key

Cost of replacing a 2012 Ford Edge key can be very expensive if you’re not prepared.

Recently, I misplaced (read: placed on the roof of my car) my 2012 Ford Edge key. Not knowing anything about keys, I assumed it was a relatively straightforward process to get a new copy. Not so much.

First and foremost, the key is a smart key, which implies there’s a chip embedded in the key that effectively talks to the car. I.e., if the key isn’t programmed to your car specifically, it’s useless. This also implies that most likely your local Walmart won’t be able to help you.

Feeling panicky, I got some quotes from the Ford dealer, with all sorts of prices that were effectively at least $500. I ended up finding a parts dealer and ordering the fob ($150) and the key ($25) and felt pretty proud of myself, since I was under the assumption that I could program the second key myself with at least one working key. That came quickly crashing down once I actually pulled out the manual and realized that I needed at least two already programmed keys in order to program another key. So, my second key really wasn’t a spare at all, and a spare would have been the third key had the dealer given me three keys, which obviously they did not. I ended up calling around various Ford dealers to get quotes for “I have one programmed key and one unprogrammed key and need to program the unprogrammed one.” I got various prices ranging from$100 and upwards but eventually I got a dealer who quoted me a price of $50 to reprogram that one key. During all of this, I did more research on keys. Apparently, I don’t really need the$175 magical key from Ford. On ebay, I found various sellers selling blank uncut transponder keys for around $15. As such, I would be able to take my two programmed keys and start programming the cheapo-deapo keys as backups. It didn’t matter that the keys were uncut, since the Ford Edge I had couldn’t be physically started anyway; the most the mechanical key could do was lock and unlock the door, but once in, you couldn’t start the car until the programmed key was within range. The moral of the story as such is that for these new fancy keys, you really don’t have a spare and that you should immediately begin the process of programming a third (if not more) key. And, for the purposes of a true spare, it doesn’t need to be the official Ford key and can be a blank uncut transponder key. Cost of replacing a “spare”:$250
Cost of replacing a true spare: \$15