Exhibit I

 

What distinguishes generative AI is the consideration of “context”, both in the training phase of a neural network and in its (use) inference phase when the weights of words have already been fixed. In the following we discuss how the transformer architecture is used in the training phase, when the network decodes the raw data and learns the best weights of words.

 

Each word (token) processed consists of a Q(Query), K(Key), V(Value) vector.  

               Where:   Q represents the model’s query vector, or its current word for processing by the decoder.

                              

·                                   K looks for semantic similarity, for all elements of the input sequence question; or how much attention the current token should pay to past tokens.

 

·                                   V stores the actual content of all the tokens, and is the now context-aware as its weighted sum flows through subsequent network layers.

 

Through these mechanisms, the decoder not only selects the appropriate tokens to construct a coherent response but also ensures that the semantic and syntactic relationships between tokens align with the intended meaning of the answer. billparker.ai provides an example, which we shorten:

1)    Embeddings provide the context in which a word is found. Actual embeddings can result in 3,072 dimension vectors. We only assume a simple embedding.

The   [1,0]

cat    [0,1]

sat    [1,1] 

2)    Training adds to word embeddings, weight vectors that minimize the difference between the model’s generated outputs and the actual data. Each word (i) has a –(q,k,v)- computation based upon learned matrices applied to the word’s embedding.

Through this, each token not only attends to relevant preceding tokens but assigns weighted relevance to their relationships, ultimately enriching the contextual understanding of sequences. These matrices—termed W_Q, W_K, and W_V—are responsible for transforming the input embeddings into subspaces which allow query, key, and value computations.

  W_{Q = [1 0]}

                     _{[0 1]  <- learned weight vector}

  W_{K =  [01]}

               _{[10] <- learned weight vector}

  W_{V = [1 1]}

              _{[1-1] <- learned weight vector}

 

The interplay between matrices allows the model to dynamically adjust its focus, enabling nuanced interpretations of context. By leveraging this mechanism, attention layers can effectively balance the importance of immediate proximity and distant dependencies within the sequence. This dynamic recalibration of focus ensures that the relationships between tokens are both fluid and precise.

 

3)    Compute Q, K, V for Each Word.

 

Q_i = E_i*W_Q = [1,0]_The , [0,1]_cat, [1,1]_sat …

 

K_i = E_i*W_K = [0,1]_The, [1,0]_cat, [1,1]_sat …

 

V_i = E_i*W_V = [1,1]_The, [1,-1]_cat , [2,0]_sat …

 

4)    Now the payoff; you can calculate the attention model pays to each word by the dot product between them: Q.K

 

Score “cat” and “The” = Q_cat .K_The = [0,1].[0,1] = 1

Score”cat” and “cat” = Q_cat.K_cat= [0,1].[1,0] = 0

Score “cat” and “sat” = Q_cat.K_sat = [0,1].[1,1] = 1

 

5)    You can now apply a softmax function to the above scores to convert the data into word probabilities, attention weights, etc. The core of generative AI is here.

 

There are several conclusions to be drawn from this simple analysis:

 

1)    The generative AI model is very, very matrix intensive. Nelson (2023) notes that AI designers try to avoid costly matrix multiplication. They try to reduce the number of large matrices by using statistical sampling; use linear algebra to simplify large matrices; and apply cosine (dot product) similarity to words or to whole documents. Progress in AI architecture is occurring at very high rates.

2)    This model does not deal with truth; it deals purely with data, bottom-up. Which means that every inquiry is to a special case, requiring very large computer programs.

3)    It can produce hallucinations, making things up, because of its word-by-word construction.

4)    It is by nature bottom-up, so any “logic” has to be an heuristic add-on. 

5)    These are simple caveats, which may or may not be surmountable in each individual application.

 

_

 

ChatGPT – suggested some technical clarifications and says, “Overall, your description provides a good overview of how attention mechanisms function in generative AI, particularly in how tokens interact and how embeddings and training contribute to contextual understanding.”

 

 

 

 

 

 

 

 

 

 

 

 

 

Cc  33

333                           characterizing