Idea: Align Two Embedding Sets, Find Words That Do Not Match

The main idea here is to align two embeddings sets and to find words whose embeddings do not match well. Formally, let \(\color{#88a635}{W_1}\color{black}, \color{#547dbf}{W_2}\color{black} \in \mathbb{R}^{d\times |V|}\) be embedding sets trained on different corpora. To align the learned embeddings, the authors find the rotation \[R = \arg \max\limits_{Q^TQ=I}\parallel \color{#547dbf}{W_2}\color{black}Q - \color{#88a635}{W_1}\color{black}\parallel_F.\] This is called Orthogonal Procrustes. Using this rotation, we can align embedding sets and find words that do not match well: these are the words that change meaning with the corpora.

Once the embedding sets are aligned, we can evaluate the semantic displacement. Let \(\color{#88a635}{v_w^1}\) and \(\color{#547dbf}{v_w^2}\) be embedding of a word \(w\) in the two aligned spaces, then the semantic displacement is \(1- \cos (\color{#88a635}{v_w^1}\color{black}, \color{#547dbf}{v_w^2}\color{black}).\) Intuitively, this measures how well embeddings of the same word "match" in the aligned semantic spaces.

Experiments

TL;DR: SGNS Embeddings are Better than PPMI and SVD(PPMI)

The authors looked at historical texts for different time periods and tried to apply the method on top of different embeddings: PPMI matrix, SVD(PPMI) and Word2Vec (SGNS). Below are examples of the top words found for each of the embedding methods.

• bold - real semantic shifts (validated by examining literature)
  E.g., headed shifted from primarily referring to the "top of a body/entity" to referring to "a direction of travel."
• underlined - borderline cases (largely due to global genre/discourse shifts)
  E.g., male has not changed in meaning, but its usage in discussions of “gender equality” is relatively new.
• unmarked - clear corpus artifacts
  E.g., special, cover, and romance are artifacts from the covers of fiction books occasionally including advertisements etc.

Looks like results obtained for SGNS embeddings are better. In the original paper, different kinds of evaluation were used to confirm this more formally.

From the next paper, you will learn how to detect semantic change more easily.

Note: The Alignment Idea is Used for Different Tasks

Note that the idea to linearly map different semantic spaces was also used for other tasks. For example, earlier in the lecture we aligned semantic spaces for different languages to build vocabulary.

For more advanced methods for building cross-lingual embeddings, look here in the Related Papers.

Idea: Train Embeddings, Look at the Neighbors

A very simple approach is to train embeddings (e.g., Word2Vec) and look at the closest neighbors. If a word's closest neighbors are different for the two corpora, the word changed its meaning: remember that word embeddings reflect contexts they saw!

Formally, for each word \(w\) the authors take k nearest neighbors in the two embeddings sets: \(NN_1^k(w)\) and \(NN_2^k(w)\). Then they count how many neighbors are the same and define the change score as follows: \[score^k(w) = -|NN_1^k(w)\cap NN_2^k(w)|\] A large intersection means that the meaning is not different (the score will be low), small intersection - meaning is different (such words will receive a high score).

The Method is Interpretable

By design, the method is interpretable: it explains its decisions (i.e., why the word is used differently) by showing the closest neighbors of the word in the two embedding spaces. These neighbors reflect the word meanings in the two corpora. Look at the examples of found words along with the closest neighbors.

Other Good Things

Compared to the alignment-based methods (e.g., the previous paper), this approach:

• is more stable,
• requires less tuning and word filtering.

For more details, look at the paper.

Lena: Note that while the approach is recent, it is extremely simple and works better than previous more complicated ideas. Never be afraid to try simple things - you'll be surprised how often they work!