Here is an interesting fact worth pondering. Take a circle and stretch it along some direction. It becomes an ellipse. Now take the ellipse and stretch it again, this time in a different direction. Continue stretching or shrinking in other directions if you like. No matter what directions are used, the final image is an ellipse, having two perpendicular axes (ignoring the degenerate case of a line segment). The direction of these axes depend on the direction and amount of stretching, but the existence of perpendicular axes always remains. This did not seem immediately obvious to me – surely one can be clever about the way one stretches and remove the perpendicularity of the axes!
A linear transformation of a circle is a sequence of stretches along axes, and such a sequence can always be reduced to at most two perpendicular stretches. That is, circles map to ellipses under linear maps.
One way of seeing why this is the case is that the equation of an ellipse is a quadratic form, and applying linear transformations does not change its degree (i.e. a conic remains a conic). Since the only bounded conic sections are ellipses, the result follows. However I would be more satisfied with a coordinate-free proof. Here is one argument that I read recently and outline below. (See more via Trefethen and Bau, Planetmath, and also the nice explanation due to Aubrey Poor here)
Let T be a linear transformation (map) from n-dimensional to m-dimensional space. T is usually represented by an m by n matrix, but we will not need that here. We will assume complex vector spaces for generality, but a 2-dimensional real vector space is easier to visualise. We wish to show that a unit sphere maps to an ellipsoid, by which we mean there exist perpendicular axes of the sphere that each map to perpendicular axes in the m dimensional space.
Consider the image of the unit sphere under T. There exists a vector such that is maximal (by the extreme value theorem). Let where and are unit vectors and (we ignore the degenerate case ). We wish to show that any vector perpendicular to maps to a vector perpendicular to . Suppose where (such an orthogonal decomposition is possible). We identify a vector that maps to something as least as long as . Consider the unit vector
The length of this vector is at least
where equality holds when . In other words, to prevent from having length greater than , we require . This means as we wished to show.
By an inductive argument we may then show that there exist orthonormal vectors in the n-dimensional space and orthonormal vectors in the m-dimensional space such that
In other words there exist orthonormal bases in the row space and column space of T which map to each other. This is a restatement of the singular value decomposition (SVD), that spheres map to ellipsoids (generalised to any dimension). The are known as singular values and are interpreted as the semi-axes lengths of the ellipsoidal image of the unit sphere. Stacking the v’s and u’s into matrices (and adding orthonormal vectors from nullspaces if necessary) gives us the alternate form
where is the matrix representation of the linear transformation T, and has the singular values down its diagonal and 0 entries elsewhere.
The SVD (which applies to any matrix) is the generalisation of the spectral theorem (which applies to normal matrices) and has wide applications, from solving least squares problems (via the pseudo-inverse) to finding low-rank approximations to matrices (enabling compression).