数字化转型网人工智能专题
翻译：
The basic principles of GPT Embedding are clearly stated in plain language
Digital Transformation Network Artificial Intelligence topics
Learn with the world’s top AI professionals! Digital Transformation Network has established a research and learning community dedicated to discussing artificial intelligence technology, industry, and academia, and grow together with you! Welcome to scan code to join!
Embedding is a fundamental concept in machine learning, especially in the field of natural language processing (NLP), but they are also widely used in other fields. In general, embedding is a method of transforming discrete classified data into continuous vectors, often in high-dimensional Spaces, transforming complex, difficult-to-process items such as words, images, or user ids into a form that machine learning models can understand and process more efficiently.
What is a vector
In mathematics, a vector (also called a Euclidean vector, a geometric vector, or a vector) is a quantity that has magnitude and direction. It can be visualized as a line segment with an arrow.
The arrow points to the direction of the vector. The length of the line segment represents the magnitude of the vector. The quantity corresponding to a vector is called a quantity (called a scalar in physics), and the quantity (or scalar) has only a magnitude and no direction.
Usually we represent a vector in a coordinate system by a line segment with a length and an arrow. In general, in Cartesian coordinates (plane coordinates) we like to place the vector’s starting point at the origin, the end point at some point in the coordinate system, and then we draw a line segment with an arrow from the origin to that point.
Since there are two lines with length and arrows from the origin of the coordinates, we can calculate the Angle between the two vectors. Of course, the vector in a normal computer is multi-dimensional, such as OpenAI Embeddings is 1536 dimensional, but the logic is the same.
What does the Angle between vectors have to do with embedding and search? The following borrow teacher Wu Jun in mathematics general explanation, easy to understand. I admire teacher Wu Jun very much. Even though I was a mathematics major at that time, I still gained a lot from listening to his class. Understanding from another Angle can make my knowledge more full and three-dimensional.
What’s the use of figuring out the Angle between two vectors? It actually has many applications, such as the automatic classification of text. These two things seem unrelated, how can they be connected? Here we will introduce the principle of automatic text classification by computer.
We know that the theme and content of an article are actually determined by the words used in it. Different articles use different words, but the words used in articles with similar themes have great similarities.
For example, there may be words such as “finance”, “stock”, “trading”, and “economy” in articles about finance, and words such as “software”, “Internet”, and “semiconductor” often appear in articles about computers. If these two parts of the keywords are not repeated, then we can easily separate the two types of text.
What if they duplicate? So we’re going to look at the frequency of each word in these two types of texts. In our experience, even if there are some computer words mixed in a financial article, their word frequency will not be too high, and vice versa.
To illustrate how to distinguish between these two types of articles, let’s assume that there are only eight words in Chinese: “finance”, “stock”, “trading”, “economy”, “computer”, “software”, “Internet” and “semiconductor”. Suppose you have an article about economics with the number of occurrences of these eight words (23, 32, 14, 10, 1, 0, 3, 2), and another article about computers with the number of occurrences of these eight words (3, 2, 4, 0, 41, 30, 31, 12), so that they each form an eight-dimensional vector, which we call V1 and V2.
If we could draw them in eight dimensions, you would see that the Angle between them is very large. I did the math, it’s about 82 degrees, almost vertical, or orthogonal. Since these vectors are positive in every dimension, the maximum Angle between them is 90 degrees, no greater. This means that the Angle between the vectors corresponding to the two different types of articles should be large.
If we assume another article with an eight-word frequency of V3= (1,3,0,2,25,23,14,10), then the Angle between it and the corresponding vector in the second article is only 7.5 degrees. The relationship between these three vectors is outlined in two-dimensional coordinates as follows.
As can be seen from the figure, the angles between the first and second vectors are large, while the angles between the second and third vectors are small. From this, we can roughly determine that the third article should be similar to the second topic, also belongs to the computer class.
Now we have to think about the question, what are the angles between vectors that are small, and what are the angles that are almost orthogonal?
If you compare the above three vectors, you will find this characteristic: when two vectors have larger components in the same dimension, their Angle is small. Conversely, when two vectors have large components in different dimensions, they are nearly orthogonal. For example, the second and third vectors have larger component values in the last four dimensions, so their Angle is small. The first vector has a large component in the first four dimensions and a small component in the last four, which is just staggered from the second vector, so it is almost orthogonal. There are two special cases of angles between vectors that you need to be aware of:
If the components of two vectors are proportional in each dimension, the Angle between them is zero.
If a vector is equal in all dimensions, such as (10, 10, 10, 10, 10, 10), it may not be very close to any of them. We’re going to have to use this property later.
Of course, in the real text classification, there are more than these 8 words, there are 100,000 words of this order of magnitude, so the corresponding vectors of each article are about 100,000 dimensions, and these vectors are called feature vectors.
By using the law of cosine to calculate the Angle between the eigenvectors, we can determine which texts are close and belong to the same class. Vectors can classify not only articles, but also people. Today, many large companies in the recruitment of staff, due to the large number of resumes, will first use the computer to automatically screen resumes, the essence of the method is to quantify people according to the resume, and then calculate the Angle.
2. One-hot unique heat coding
Moving on to the method of turning text into vectors, a common and well-known method of representing classified data is unique thermal coding. This method converts each item in the data set to a vector consisting of zeros and a “1”. Imagine a vocabulary in which each word is assigned a unique position in a vector. For example, in a simple vocabulary consisting of three words, “apple,” “banana,” and “cherry,” the word “apple” might be represented as [1, 0, 0], “banana” as [0, 1, 0], and “cherry” as [0, 0, 1]. This approach is simple to understand, but it has major limitations.
The main problem with monothermal coding is its inefficiency, especially in the case of large vocabularies. Each word requires a vector as long as the entire vocabulary, which results in high memory consumption and sparse representation. In addition, this approach fails to capture any semantic relationships between words. “Apple” and “banana” may be more similar than “apple” and “cherry” (both fruits), but in unique heat coding, all words are equidistant from each other.
This is where embedding comes into play, providing a more nuanced and efficient representation of data. Embedding maps unique thermal coding vectors to low-dimensional Spaces, where similar items are placed more closely together. This transformation is learned from the data and enables the model to discern and encode relationships and patterns in the data. For example, in an embedded space, “apple” and “banana” might be placed closer together, reflecting their semantic similarity as fruits, unlike other non-fruit words.
By leveraging embeddings, machine learning models can gain a deeper understanding of the data, resulting in more accurate and insightful results. Whether it’s capturing linguistic nuances in NLP tasks or identifying patterns of user behavior, embedding offers a more efficient and sophisticated way to process complex data.
3. How does embedding work?
Representation: In NLP, every unique word in a language is represented as a dense vector in a continuous vector space. These vectors usually have a fixed size (such as 100, 300, or 512 dimensions), independent of vocabulary size.
Contextual meaning: Unlike unique thermal coding, which provides sparse and uninformative information (where each word is just a different index in a long vector), embedding can capture more information about the word. Words that have similar meanings or are used in similar contexts tend to be closer together in the embedding space. For example, “king” and “queen” might be represented by vectors that are close to each other.
Training: Embedding can be pre-trained on large text corpora such as news articles, Wikipedia, or web pages using algorithms such as Word2Vec, GloVe, or more advanced methods such as BERT (for contextual embedding). The model learns to place words with similar meanings in this high-dimensional space.
Dimensionality reduction: This indicates that dimensionality reduction is allowed. Instead of dealing with thousands or millions of unique words, the model deals with vectors with much smaller dimensional Spaces.
Use in models: These vector representations can then be fed into various machine learning models (such as neural networks) for tasks such as sentiment analysis, machine translation, or content recommendation.
Beyond words: The concept of embedding goes beyond words to include sentences, paragraphs, user ids, products, etc., where similar items are represented by close vectors in the embedding space.
Why it’s useful:
Efficiency: They provide a compact, dense presentation.
Semantic meaning: They capture the deep semantic meaning of words or things.
Flexibility: Can be used for a variety of machine learning tasks.
Transfer learning: Pre-trained embeddings can be used to improve performance for data-limited tasks.
Let’s use a simple example of word embedding to illustrate how embedding works. Suppose we have a very small vocabulary of five words: [” cat “, “dog”, “fish”, “run”, “swim”]. In traditional methods, we might represent these words using a unique thermal code, where each word is represented by a vector that is “1” in the position corresponding to that word and “0” elsewhere. For example:
cat = [1, 0, 0, 0, 0]
dog = [0, 1, 0, 0, 0]
fish = [0, 0, 1, 0, 0]
run = [0, 0, 0, 1, 0]
swim = [0, 0, 0, 0, 1]
In unique heat coding, each word is at an equal distance from the other words; No concept of similarity or context.
4. The embedding process
Training: We use models to learn embeddings. Let’s say our model learns two-dimensional embeddings (in reality, they are much higher in dimensions, but for simplicity, let’s use two dimensions).
Resulting embeddings: The model might learn to represent these words in the following way:
cat=[0.9, 0.1]
dog = [0.85, 0.15]
fish = [0.1, 0.8]
run = [0.2, 0.7]
swim = [0.15, 0.85]
Embeddings provide a way to capture semantic relationships and similarities between words that single-thermal coding doesn’t have, and now we have:
Similarity: Note that “cat” and “dog” are close to each other in this embedding space (both have higher values in the first dimension and lower values in the second dimension). This reflects their similarity as pets.
Differences: In contrast, “cat” and “swim” are far apart, indicating lower similarity.
Contextual grouping: Activities such as “rum” and “swim” are closer to each other and closer to “fish” (contextually related to swimming) than to “cat” or “dog.”
5. A simple example of how embedding works
Let’s go through a simplified process to understand how to develop such a model using basic machine learning concepts. We’ll use a very basic approach that focuses on clarity rather than performance or scalability.
Suppose you have a small set of sentences, each containing words like “cat,” “dog,” “fish,” “run,” and “swim.” These sentences form a corpus, a miniature world in which “cat and dog are pets” or “fish swim in water” are typical scenarios.
Now, consider analyzing these sentences and breaking them down into individual words. It’s kind of like tokenization, where each word stands on its own. You may even decide to drop common words that don’t add much interest, like “and” or “are.”
Next, give each word its own identity in the form of a two-dimensional vector. These vectors are initially random, a bit like assigning a unique fingerprint to each word without thinking about it.
The real magic starts when you define context for each word. Imagine choosing a window size, say two words on either side of the target word. This is the immediate neighborhood of each word, their social circle in the sentence world.
Then, you begin a simple learning process. The goal is to have words that share context also share similar vectors. It’s kind of like pushing them closer together in numerical space. You can do this by tweaking the vector of a word to make it more like its neighbor’s vector. At the same time, push it away from the vector of random words that do not share their context.
This process is not a one-time thing. It’s more of a loop. You browse through each word in the corpus, updating its vector based on its neighbors. Sometimes, you also make it slightly different from random, unrelated words.
As you keep repeating this process, iterating over and over again, the vector starts to stabilize. They find an equilibrium point at which the adjustment becomes smaller and smaller. This is convergence, where vectors now represent not only random assignments, but also meaningful relationships based on the context in which the word appears. Repeat this process enough times, and it will reveal hidden structures in your corpus, word relationships built from scratch.
6. Working principle in large models such as GPT
Compared with traditional word embeddings in models such as Word2Vec, embeddings in models such as GPT work differently. GPT is a translater-based model that uses a more complex and context-aware approach to embedding. The following is an example of the vector that OpenAI Embedding will be converted to
Imagine that GPT is a master of language, first breaking down text into markup. Unlike the simple method of using whole words, GPT usually selects sub-word units, such as byte pair encoding. This nuanced approach allows it to efficiently process large numbers of words, even those that are rare or never seen before.
Each token is then given an initial identity, which is digital DNA in the form of an embedded vector. These embeddings are not randomly assigned; They are carefully learned during GPT training, providing a foundational understanding for each token.
Now, let’s think about the challenge of understanding language: it’s not just about the words, it’s about the order of the words. GPT solves this problem by injecting positional coding into its embeddings. These are like secret signals that are unique to each position in the sequence, ensuring that the order of words is never lost.
When these initial embeddings and positional codes enter the transformation layer of GPT, the real contextualization begins. Here, during parallel processing, the model examines each tag in the context of each other. This is done through the self-attention mechanism, a sophisticated tool that allows each marker to learn its relevance to the others.
Embeddings evolve as input text moves through Transformer’s continuous layers. With each layer, they absorb more context, enriching their presentation. This is not just a superficial understanding; It is a deep, contextualized insight into each token, influenced by the entire sequence of inputs.
When the text reaches the final layer, a vector representation appears for each tag. These vectors represent more than just independent meanings, they have a deep context that reflects not only the marker itself, but also its relationship to every other marker in the sequence.
These final embeddings can be used for a variety of tasks – text generation, language translation, question answering, and more. The main characteristics of GPT embeddings are that they are context-aware, dynamic, layered, and benefit from subword tokenization. This gives GPT a rich and nuanced understanding of language that goes far beyond simpler models. In essence, the embedding mechanism of GPT is a complex fusion of linguistics and technology that can capture a deep and multifaceted understanding of language.
7. Finally
Embeddings represent an important advance in the field of machine learning and natural language processing. They provide an efficient way to convert classified data, such as words or images, into a digital format that machine learning models can process. The shift from basic thermal coding to more complex embedding techniques reflects significant improvements in machines handling large, complex data sets.
As the field continues to evolve, the role of embeddings in processing and interpreting large amounts of data is likely to become more prominent. Their integration with advanced models such as GPT highlights their importance in current and future AI technologies. Therefore, embeddings remain an important part of the ongoing development of more complex and efficient machine learning systems.
Digital Transformation Network Artificial Intelligence topics
Learn with the world’s top AI professionals! Digital Transformation Network has established a research and learning community dedicated to discussing artificial intelligence technology, industry, and academia, and grow together with you! Welcome to scan code to join!
