Exercises

Table of Contents

Information Theory, Inference and Learning Algorithms

6.14

Since there is no covariance between the different dimensions, i.e. the exercises_42b3c8f6ed56c7cfb60e6194029bee525c549d11.png are independent of all exercises_cc919856ca77385845d81cfd831bcb8a0384b2af.png with exercises_7f1abbaf6a3a55725c2e37d3f1a30c019c1c9ea1.png, we know that

exercises_5847221dbacd5fcc4c6626e4711fff0d2345188b.png

where each of the exercises_e9f4d218474bc10aa94958ec30139aee865c0173.png are following exercises_c2e2e046f4a55927d355dfcac9514c1d6d8babb2.png, hence

exercises_d4e5b3058dd0872e4b171392bfe24218ed27c7f2.png

Hence,

exercises_e2b940d734704f75fdd927a669bd49e6515da5b9.png

The variance is then given by

exercises_3ee1496c0c3a30b398dfaf3ba881b7304538b334.png

where

exercises_7f8fb1227df30e277eb87d1f505b75f3d8855458.png

But since there is no covariance between the different exercises_e9f4d218474bc10aa94958ec30139aee865c0173.png, the second sum vanishes, and since

exercises_0e5c6fea89db65a303a9ce411444dc44e096a961.png

(which we knew from the hint 6.14 in the book). Hence

exercises_d0fcd548f04da5c23de55d6e45161ce5e4e96b08.png

This all means that for large exercises_e10e2b430f95617381cdd6d6b52aed29fb971dff.png, we will have

exercises_3a369dc6e1f28fc5b53408337b4b40e3eca8e263.png

And since exercises_f88c366859b7bbabd746a8997dd1f1ed63344190.png will be neglible for large exercises_e10e2b430f95617381cdd6d6b52aed29fb971dff.png, compared to exercises_e10e2b430f95617381cdd6d6b52aed29fb971dff.png (of course assuming exercises_f2eebbd62e41be9d5a2457b4bd291dd096896884.png is finite), then

exercises_ffd234869c408419698ae62213c1a44eeb54b3e3.png

as watned. The "thickness" will simply be the exercises_dfe3dbd96d728c8d9f491711ca4b225749fd474d.png, i.e. twice the variance of exercises_aeeb5a2afe5a5accdb81a95a3b50568029180a01.png.

Either by:

  • Computing an exercises_e10e2b430f95617381cdd6d6b52aed29fb971dff.png dimensional integral :)
  • Empirically looking at exercises_e89168996a065100b69f75f3fc121549ab9f209d.png for some exercises_3c314f80373742988ad542a6d4ce66a111b17847.png and making use of the symmetry of the Gaussian to infer that all exercises_ed39d9a397196f8f0ce6388b0ea4e0c1dd8becee.png with same radius have the same probability, and that exercises_1ed25cf746e36b58c8d980488e7f49fe0ccb1b39.png decreases when exercises_ed39d9a397196f8f0ce6388b0ea4e0c1dd8becee.png moves away (in whatever "direction" / dimension) from the mean

We can observe that the majority of the probability mass is clustered about this "shell".

Bibliography

  • [mackay2003information] MacKay, Kay & Cambridge University Press, Information Theory, Inference and Learning Algorithms, Cambridge University Press (2003).