Notes on: Olah, C. (2014): Neural networks, manifolds and topology

Table of Contents

Notation

Definitions

Formally, an ambient isotopy between manifolds NeuralNetworksManifoldsAndTopology_8939ac39ed16887f3b3c033a47d4ab60d120251a.png and NeuralNetworksManifoldsAndTopology_aa45b78ca4fcfb1128b81983bc69cebaddb6486d.png is a continuous function NeuralNetworksManifoldsAndTopology_4fd496402f949fbf07d528338a84c8e2c86d981d.png such that each NeuralNetworksManifoldsAndTopology_5aa5a0542e0ce92515e664cbeaa23eb29abb3ff2.png is a homeomorphism from NeuralNetworksManifoldsAndTopology_0207be880056b9a69e22e729dd37bced29cd174a.png to its range, NeuralNetworksManifoldsAndTopology_cbd58c587c4697c458cfe80562ee38de34575f67.png is the identity function, and NeuralNetworksManifoldsAndTopology_f129ee0ef1a12ebe79e1b8e2a7213d3ad1eb0e59.png maps NeuralNetworksManifoldsAndTopology_8939ac39ed16887f3b3c033a47d4ab60d120251a.png to NeuralNetworksManifoldsAndTopology_aa45b78ca4fcfb1128b81983bc69cebaddb6486d.png. That is, NeuralNetworksManifoldsAndTopology_5aa5a0542e0ce92515e664cbeaa23eb29abb3ff2.png continuously transitions from mapping NeuralNetworksManifoldsAndTopology_8939ac39ed16887f3b3c033a47d4ab60d120251a.png to itself to mapping NeuralNetworksManifoldsAndTopology_8939ac39ed16887f3b3c033a47d4ab60d120251a.png to NeuralNetworksManifoldsAndTopology_aa45b78ca4fcfb1128b81983bc69cebaddb6486d.png.

Overview

  • Suggests that training NNs is iteratively mapping between manifolds (and if you assume the weight-matrix to be non-singular, this mapping is homeomorphic ) with the goal of obtaining a manifold in which the data is separable
  • Mentions that if we assume The Manifold Hypothesis to be true, which says that natural data forms lower-dimensional manifolds in its embedding space, then the task of classification is fundamentally to separate a bunch of tangled manifolds.