# Dot Products, Quadratic Forms and Eigenvalues

These lecture notes were part of the instruction material for SE-409 (Fall Semester of 2013).

http://www.uia.no/portaler/student/studierelatert/studiehaandbok/11-12/emner/se-409

#### Andrew Musau

September 03, 2013

## Transcript

1. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.1 Lecture 5 Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. SE-409– Quantitative Methods in Economics and Finance– Fall Semester 2013 Sep. 3, 2013 Andrew Musau University of Agder
2. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.2 Agenda 1 Dot Products 2 Quadratic Forms 3 Eigenvalues and Eigenvectors
3. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.3 Dot product (Inner product) DEFINITION: Dot product Let v and u be two vectors in Rn. The dot product of v and u is the scalar v · u and is given by v · u = v1 u1 + v2 u2 + ... + vn un = n i=1 vi ui . Given the rules of vector multiplication, we cannot multiply two vectors when they are both column vectors. Therefore, we take the transpose of the ﬁrst vector, turning it into a row vector. In vector notation, thus, the dot product of v and u is vTu. Example. Let v =   3 −1 7   and u =   1 1 2  . Find the dot product of the two vectors.
4. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.4 Dot product (Inner product) Solution. (1) Transpose the vector v vT = 3 −1 7 (2) Determine vTu v · u = vTu = 3 −1 7   1 1 2   = 16.
5. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.5 Dot product (Inner product) Properties of the Dot Product If u, v, and w are any vectors and λ is a scalar, then 1 u · v = v · u. 2 (λu) · v = u · (λv) = λ(u · v). 3 u · (v + w) = u · v + u · w. 4 u · u = |u|2 5 0 · u = 0.
6. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.6 Quadratic forms and symmetric matrices A quadratic function takes the form f(x) = ax2 + bx + c (1) where a, b and c are constants. A generalization of (1) is q(xy) = ax2 + bxy + cy2 (2) where (2) reduces to (1) if y = 1. It is possible to express (2) in matrix form.
7. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.7 Quadratic forms and symmetric matrices DEFINITION: Quadratic forms Let A denote an n × n symmetric matrix with real entries and let z denote an n × 1 column vector. Then q(x, y) = zTAz is said to be a quadratic form. Note that q(x, y) = zTAz = z1 · · · zn    a11 · · · a1n . . . ... . . . an1 · · · ann       z1 . . . zn    = z1 · · · zn    n i=1 a1i zi . . . n i=1 ani zi    = (a11 z2 1 + a12 z1 z2 + ... + a1n z1 zn) + (a21 z2 z1 + a22 z2 2 + ... + a2n z2 zn) + ... + (an1 zn z1 + an2 zn z2 + ... + ann z2 n ) = i≤j aij zi zj
8. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.8 Quadratic forms and symmetric matrices Example. Let A= 1 2 2 1 , then q(x, y) = zTAz = z1 z2 1 2 2 1 z1 z2 = z1 + 2z2 2z1 + z2 z1 z2 = z2 1 + 2z1 z2 + 2z1 z2 + z2 2 = z2 1 + 4z1 z2 + z2 2 (3) Thus, (3) is a quadratic form (2) where a = 1, b = 4 and c = 1.
9. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.9 Deﬁnite and semideﬁnite quadratic forms DEFINITION: Deﬁnite and semideﬁnite quadratic forms Let q(x) be a quadratic form. If q(x) > 0 for every non-zero vector x, then q(x) is a positive deﬁnite quadratic form. If q(x) ≥ 0 for every vector x, then q(x) is said to be positive semideﬁnite. The converse holds for negative deﬁnite and negative semideﬁnite forms. Examples. The quadratic form q(x) = x2 1 + x2 2 + ... + x2 n is positive deﬁnite since q(x) > 0 for every non-zero n-vector. The quadratic form q(x1, x2, x3) = x2 1 + (x2 − x3)2 being a sum of squares is non-negative but q(0, 1, 1) = 0. Therefore, it is positive semideﬁnite. The quadratic form q(x1, x2) = x2 1 − x2 2 is neither positive semideﬁnite nor negative semideﬁnite since q(0, 1) = −1 and q(1, 0) = 1.
10. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.10 Quadratic forms and symmetric matrices A symmetric matrix A and its corresponding quadratic form is said to be positive deﬁnite if zTAz > 0 whenever z = 0, and positive semideﬁnite if zTAz ≥ 0 for all z. Similarly, A is positive (semi)deﬁnite if −A is negative (semi)deﬁnite. Testing quadratic forms: 2 × 2 symmetric matrices. Let A be a 2 × 2 symmetric matrix, then A is positive deﬁnite iff its diagonal elements are both positive and its determinant is positive. A is positive semideﬁnite iff its diagonal elements are both non-negative and its determinant is non-negative. A is negative deﬁnite iff its diagonal elements are both negative and its determinant is positive. A is negative semideﬁnite iff its diagonal elements are both non-positive and its determinant is non-negative.
11. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.11 Testing quadratic forms: 2 × 2 symmetric matrices. Example: Determine the deﬁniteness of the symmetric matrices A− E. A = 1 0 0 1 , B = −3 −3 −3 −3 , C = −2 1 1 −2 , D = 3 3 3 3 , E = −2 −3 −3 −2 . Solutions. |A| = 1 > 0, and diag(1, 1) > 0 ⇒ A is positive deﬁnite. |B| = 0, and diag(−3, −3) < 0 ⇒ B is negative semideﬁnite. |C| = 3 > 0, and diag(−2, −2) < 0 ⇒ C is negative deﬁnite. |D| = 0, and diag(3, 3) > 0 ⇒ D is positive semideﬁnite. |E| = −5, and diag(−2, −2) < 0 ⇒ E is neither positive (semi)deﬁnite nor negative semi(deﬁnite).
12. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.12 Testing quadratic forms: Higher dimensions In the general n × n symmetric matrix case, there are several ways of testing deﬁniteness. An n × n symmetric matrix is positive deﬁnite if all its leading entries (pivots) are positive after reducing it to an upper triangular matrix using Gaussian elimination without row exchanges. Similarly, A is positive semideﬁnite if all its leading entries (pivots) are non-negative after reducing it to an upper triangular matrix. As an example, matrix A in the previous example i.e., the 2 × 2 identity matrix is in upper triangular form, and both of its two leading entries are equal to one which is greater than zero. Thus, it is positive deﬁnite. Taking matrix C from the previous example, we perform elementary row operations to reduce it to an upper triangular matrix Take 1 2 × Row 1 + Row 2 and get −2 1 0 −3 2 The matrix is now in upper triangular form but the leading entries −2 and −3 2 are not positive. Therefore, the matrix is not positive deﬁnite. The easiest way to test for negative deﬁniteness or semideﬁniteness of a matrix A is by applying the above criterion to the matrix −A.
13. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.13 Testing quadratic forms: Higher dimensions If we do so for matrix C, we deduce that it is negative deﬁnite since −C is positive deﬁnite. A second method of determining the deﬁniteness of a matrix is by looking at its n upper left determinants. The kth leading entry (pivot) of a matrix is given by |Ak | |Ak−1| where Ak is the upper left k × k submatrix. All the leading entries will be positive iff |Ak | > 0 for all 1 ≤ k ≤ n. Therefore, if all upper left k × k determinants of a symmetric matrix are positive, then the matrix is positive deﬁnite. Example: Determine whether the following matrix is positive deﬁnite.   2 −1 0 −1 2 −1 0 −1 2  
14. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.14 Testing quadratic forms: Higher dimensions Solution. We determine the n upper left determinants of the matrix. 2 = 2 2 −1 −1 2 = 3 2 −1 0 −1 2 −1 0 −1 2 = 4 Recall that in the 3 × 3 case, we apply Gaussian elimination and reduce the matrix to upper triangular form then multiply the diagonal elements to ﬁnd the determinant. Thus, since 2 > 0, 3 > 0, and 4 > 0, all of the upper left determinants of the matrix are positive and thus it is positive deﬁnite.
15. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.15 Eigenvalues and Eigenvectors DEFINITION: Eigenvalue and eigenvector An eigenvector of a square matrix A is a non-zero vector v that, when the matrix is multiplied by v, yields a constant multiple of v, where the multiplier is denoted by λ. That is: Av = λv The scalar λ is called the eigenvalue of A corresponding to v. Notice that because Ix= x where I is the identity matrix, from the deﬁnition above, we can write λIv = Av ⇔ λIv − Av = 0 ⇔ (λI − A)v = 0 and the above equation holds for some non-zero vector v. Again, recalling our deﬁnition of singular and non-singular matrices in the previous lecture, we have that v is an eigenvector of A iff λI − A is a singular matrix. Therefore, it is sensible to ﬁnd the eigenvalues of A by solving the equation |λI − A| = 0 since the determinant of a singular matrix is zero.
16. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.16 Eigenvalues and Eigenvectors Example. Find the eigenvalues and eigenvectors of the following matrix 3 −1 4 −2 Solution. (1) First determine λI − A. In the case of a 2 × 2 matrix A, we have λI − A = λ 1 0 0 1 − 3 −1 4 −2 ⇒ λ 0 0 λ − 3 −1 4 −2 = λ − 3 1 −4 λ + 2 . (2) We now ﬁnd |λI − A| and equate it to zero, then solve for λ. |λI−A| = 0 ⇒ λ−3(λ+2)−(−4) = 0 ⇒ λ2−3λ+2λ−6+4 = 0 ⇒ λ2 − λ − 2 = 0
17. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.17 Eigenvalues and Eigenvectors This is a quadratic equation which may be factored as follows: λ2 − λ − 2 = 0 ⇒ (λ − 2)(λ + 1) = 0 The eigenvalues of matrix A are the roots of the equation, that is, 2 and −1. (3) Finally, we determine the eigenvectors of A corresponding to 2. Av = λv implies that 3 −1 4 −2 v1 v2 = 2v1 2v2 Resulting in the following equations 3v1 − v2 = 2v1 ⇒ v1 = v2 4v1 − 2v2 = 2v2 ⇒ v1 = v2 Thus, the eigenvectors corresponding to 2 are the non-zero multiples of 1 1 . It is also straight-forward to show that the eigenvectors corresponding to −1 are the non-zero multiples of 1 4 .
18. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.18 Diagonalizable Matrices and Eigenvalues DEFINITION: Diagonalizable matrix A square matrix A is called diagonalizable if there exists an invertible matrix P such that P−1AP is a diagonal matrix. An n × n matrix is diagonalizable iff it has n different eigenvalues. How to diagonalize a matrix: Procedure Consider the matrix A =   1 2 0 0 3 0 2 −4 2   We use the procedure we outlined to ﬁnd the eigenvalues of the matrix. Thus, λI − A =   λ − 1 −2 0 0 λ − 3 0 −2 4 λ − 2   Notice that we can apply the co-factor expansion method of Lecture 4 to ﬁnd the determinant of the matrix above. It is wise to choose a row with one or more zeros, say row 2 and thus c22 = (−1)4 [(λ − 1)(λ − 2) − 0]
19. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.19 Diagonalizable Matrices and Eigenvalues Thus, the determinant is given by a22 c22 = (λ − 3)(λ − 1)(λ − 2) The eigenvalues of the matrix A are the roots to the equation (λ − 3)(λ − 1)(λ − 2) = 0, that is, 3, 1, and 2. A is a 3 × 3 matrix with 3 different eigenvalues and hence it is diagonalizable. The eigenvector of A corresponding to 3 is determined as follows Stacked:   1 2 0 0 3 0 2 −4 2     v1 v2 v3   =   3v1 3v2 3v3   In equations: v1 + 2v2 = 3v1 ⇒ v1 = v2 3v2 = 3v2 ⇒ v2 = v2 2v1 − 4v2 + 2v3 = 3v3 ⇒ 2v1 − 4v2 = v3
20. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.20 Diagonalizable Matrices and Eigenvalues For the ﬁnal equation, we can replace 2v1 = 2v2 and get v3 = −2v2 . Thus, the eigenvectors corresponding to 3 are non-zero multiples of   1 1 −2  . Similarly, it can be shown using the same procedure that the other two eigenvectors are non-zero multiples of   0 0 1   corresponding to 2 and   1 0 −2   corresponding to 1. Each of the three eigenvectors constitutes the columns of the matrix P (order does not matter). Therefore, P =   1 0 1 1 0 0 −2 1 −2   Then, P diagonalizes A, as a simple computation conﬁrms. Computing P−1 by augmenting the 3 × 3 identity matrix to the left hand side of P and performing elementary row operations until we obtain the identity matrix on the
21. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.21 Diagonalizable Matrices right hand side, we obtain P−1 =   0 1 0 2 0 1 1 −1 0   and computing P−1AP, one obtains the diagonal matrix   0 1 0 2 0 1 1 −1 0     1 2 0 0 3 0 2 −4 2     1 0 1 1 0 0 −2 1 −2   =   3 0 0 0 2 0 0 0 1   Therefore, we can end with the following remark. As long as A is diagonalizable, and thus there exists an invertible matrix P such that P−1AP is a diagonal matrix, then the elements in the diagonal of P−1AP are the eigenvalues of matrix A. In this example, we have 3, 2, and 1 which is what we found by solving the equation |λI − A| = 0 . This therefore is another method of determining the eigenvalues of a matrix in case one is able to obtain a matrix P that diagonalizes a given matrix A.
22. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.22 How to choose a diagonalizing orthogonal matrix Orthogonal matrices An orthogonal matrix P is a matrix with the property PT = P−1. This implies that a square matrix is orthogonal iff PTP = I. Theorem If all the eigenvalues of a matrix A are real numbers and if A is diagonalizable (that is there exists an invertible matrix P such that P−1AP is a diagonal matrix), then it is possible that P can be chosen such that PT = P−1. This is achieved by taking a matrix V whose columns are eigenvectors of A and multiplying each column v of V by (vTv)−1/2. The result is that P is an orthogonal matrix and P−1AP is a diagonal matrix whose diagonal elements are the eigenvalues of A.
23. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.23 How to choose a diagonalizing orthogonal matrix Example. Let A = 4 1 1 4 . Find a diagonal matrix D and an orthogonal matrix P such that P−1AP = D. Solution. Using the procedure previously outlined, the eigenvalues of the matrix A are 5 and 3 with eigenvectors 1 1 and −1 1 . Setting D = 5 0 0 3 , we have the familiar case P−1AP = D 1 2 1 2 −1 2 1 2 P−1 4 1 1 4 A 1 −1 1 1 P = 5 0 0 3 D . However, in this case, P is not orthogonal, that is PT = P−1.
24. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.24 How to choose a diagonalizing orthogonal matrix To make P orthogonal, we take the two eigenvectors of matrix A, call them v and u and multiply them with (vTv)−1/2 and (uTu)−1/2 respectively. These will be the new columns of the orthogonal P matrix. Thus (vTv)−1/2 = 1 1 1 1 −1/2 = 1 √ 2 . Similarly, (uTu)−1/2 = 1 −1 1 −1 −1/2 = 1 √ 2 . and P is thus P = 1 √ 2 − 1 √ 2 1 √ 2 1 √ 2 P is now orthogonal and P−1AP = D.
25. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. Dot Products Quadratic

Forms Eigenvalues and Eigenvectors 5.25 Eigenvalues and Deﬁniteness Now that we are able to calculate the eigenvalues of a matrix, we come back to testing the deﬁniteness of symmetric matrices and say a few more words. A real symmetric matrix A is 1 positive deﬁnite iff all of its eigenvalues are positive. 2 positive semideﬁnite iff all of its eigenvalues are non-negative. 3 negative deﬁnite iff all of its eigenvalues are negative. 4 negative semideﬁnite iff all of its eigenvalues are non-positive.
26. ### Dot Products, Quadratic Forms, Eigenvalues and Eigenvectors. 5.26 References Pemberton,

Malcolm, and Nicholas Rau. (2007). Mathematics for economists: an introductory textbook.. Manchester University Press. Thomas, George B., and Ross L. Finney. (2002). Thomas’ Calculus, Alternate Edition.. Addison Wesley Publishing Company.