Can We Manually Edit the Embedding Vectors?

A discussion about the feasibility of manually editing an embedding

When referring to embedding space, we read samples that an embedding is a vector to hold the key features of an object. Those key features are compatible with addition and subtraction for building a new embedding as the following samples show.

So that

This sample is sourced from the wonderful and educational video — “But what is a GPT? Visual intro to transformers | Chapter 5, Deep Learning” [1]— from 3Blue1Brown channel. As shown in Figure 1.

Figure 1. A conceptual explaination to embedding by 3Blue1Brown

While this sample provides a clean and neat conceptional explaination of what is an embedding vector, it also gives the impression that these embedding vectors are editable, and we can manually manipulate the embedding space for variable purposes.

The question is:

Can we really contruct effective embeddings by manually editing?

This article is trying to use real samples and data to find out.

1. Testing methodology

Assume there is a target item associated with the following description:

target_string = "v-neck t-shirt for causal wear, red color"

And I am going to match it with the search string:

search_string = "red t-shirt for party"

Also, construct the embedding by a weighted(1/4) combination of the breakdown keywords.

breakdown_keywords = [
   "red",
   "t-shirt",
   "for",
   "party"
]

Then manually constructed embedding.

If the embedding editing is effective, I should get a similar dot product using the manually combined embedding to search the target embedding.

I am going to apply this test using ViT-B-32 CLIP model.

2. Embedding editing with a CLIP model

First, let’s load up packages and list all available CLIP models. (You can also use a larger CLIP model from the list names).

import torch
import open_clip
from open_clip import…