Notes on: Bojanowski, P., Grave, E., Joulin, A., & Mikolov, T. (2016): Enriching Word Vectors With Subword Information

Table of Contents

Notation

  • bojanowski16_enric_word_vector_with_subwor_infor_793d8945a4d21a7cae16fc79d18a4fc27aedf9e9.png target word
  • bojanowski16_enric_word_vector_with_subwor_infor_51aba9d00663a37c124018e00f429eb33ebf2bdc.png contextual word of bojanowski16_enric_word_vector_with_subwor_infor_793d8945a4d21a7cae16fc79d18a4fc27aedf9e9.png
  • bojanowski16_enric_word_vector_with_subwor_infor_f4c83b0676d25215377c07549a5f43d8de9be036.png set of negative examples sampled from the vocabulary
  • bojanowski16_enric_word_vector_with_subwor_infor_fb55e6074d8b020fe001357d82cb4b9e3ab91fc9.png set of indices of words surrounding word bojanowski16_enric_word_vector_with_subwor_infor_c1dd93242560e0ab3b0435e3aa7b0477a05193d3.png

Overview

  • Consider each word to be the sum of representations of character n-grams and a special "n-gram" corresponding to the word itself
  • Skipgram mode, where each word is represented as a bag of character n-grams.
  • A vector representation is associated to each character n-gram; words being represented as the sum of these representations.