Weiqi Wang Blog

reading_club_RAG

Fri, 02 May 2025 00:00:00 GMT

Hi guys, this reading club is suppoed to be hold on 6th May.

The go thorugh would be too random. Hope you could read by yourself. Or maybe you could find a better blog to read. ^^

It is about the paper of in context RAG arxiv. Cool, let's start.

I have read the paper and I would not cover chapter by chapter but cover the topic I did not know before and then move forward to what I am interested in.

What is RAG? Why we need it? What we should suppose it be able to do?

RAG is a complementary technique to LLM. It is a technique to improve the performance of LLM. It is well-known as a bunch of documents or storage of bunch of knowledge. Either LLMs have access to the knowledge or not, RAG is able to enriched the LLM's attention to this part. but RAG is not a static storage. It is a dynamic storage. It involves a whole process of retrieval, ranking, and reranking.

For an example, LLM might be good at giving the previous history but have no access to news. RAG is able to give the news to LLM. Another example is that despite LLM already learned the knowledge but some time it might mislead to wrong information. RAG could be able to fix that.

I heard some blog said that RAG is not as important as former since LLM is able to provide the answer by itself.

I do not know if I understand is correct or not. It is more like an extra work technique during the infering process. Even with base model performance evolve, RAG should be valuable for some case: not common knowledge, not common question, not common context.

What is in context RAG? What is the difference between in context RAG and RAG?

They said a lot of bad word or difference between in context RAG and RAG. I do not know if it is true or not since when I just summarize and repeat to LLM, they would say that is not so true.

The main difference is that RAG commonly provide a retrieval result as some embedding or vector or some other way. In context RAG is simpler. It just provide the query response to LLM. no extra training. Easy to use.

Beside that, very important thing is that it did not stop at the new structure of RAG, but also systematical analysis of the performance of RAG with a frequent retrieval. (result said more frequent retrieval is better)

What is the performance of in context RAG in this paper?

Improve the performance of Multiple or said all LLM with lower perplexity.

result 2: BM25 which is a variant of TF-IDF is better than all the encoder representation including BERT or other BERT-like model.

They propose a very interesting design of the study. One is changable Retrieval stride that they can change the retrieval to change the frequency of retrieval. retrieval query length: since different LLM might have different context window, they can change the query length to see the performance.

Result 3: they use a stride 4 instead of a common 64. so it is much more frequent retrieval. Result is that the frequency of retrieval is like a effience and effectiveness delimma: you cannot hold both.

retrieval query length does not show large impact as the frequency did. For their task and model the bes one is query with 32 length. From my perspective, the number of the length should vary with different task at least and for model, might vary as well.

Increase the range of the candidate could improve the performance a lot.

Good innovation: LMs as a zero shot reranker

They use a same model to do the reranking. It is more like a compensation for the BM25 since BM25 is not abled with the content understanding.

reading_literature_review

Mon, 28 Apr 2025 00:00:00 GMT

Reading literature review When Software Security Meets Large Language Models: A Survey https://ieeexplore.ieee.org/abstract/document/10846956

This paper is about what llm can do mostly is what it had done in the software security.

Include:

fuzzing
unit testing
code repair
bug reproduction
error detection
error classification

Fuzzing

LLMs can help fuzzing to generate more aimed test cases. There are two ways to use llm to help fuzzing:

Use LLMs to do zero-shot fuzzing.
Use LLMs to replace the Stucture guided fuzzing program, input generation or mutation component. (This is like use llms to replace one of the component of fuzzing)

preprocess

Preprocessing for the fuzzing is to provide the inforamtion that LLMs need. Context could be a component image like button and textbox when it is about UI testing. If the fuzzing is about to test the excution, the preprocessing could be the code coverage.

prompt

Provide the prompt with the unfilled information for llms to generate the test cases.

postprocess

Postprocessing is to evaluate the test cases.

Unit Testing

Traditional tool: EvoSuite, Randoop

LLms are considered to write assertions to test if the execution of the code is correct.

preprocess

Could be Get information about APIs and their associated access paths, annotations, and functional descriptions, as well as other unit-related information such as class signatures, method signatures, and method calls. This may include control flow analysis, data flow analysis, or abstract syntax tree (AST) analysis.(This paper is too vague that what information is for what task is not clear described or classified)

prompt

Preprocessing could generate very large prompt for llms. so there are two ways to solve the problem:

only contain the most important information.
start with a small prompt and then add more information to it.

postprocess

Sometimes the generation is not making sense, so postprocess is needed to fix any bug in the testing such as add the brackets or some other pattern based. some studied also involved the llms in the postprocess.

Comparison and limitation

For

reading_club_04_22

Mon, 21 Apr 2025 00:00:00 GMT

Improving Alignment and Robustness with Circuit Breakers

arxiv

AI system can take harmful actions and could be vulnerable to adversarial attacks. This paper proposed a way called circuit breaking to improve the safety of the system especially for the adversarial attacks the model to do harmful actions.

Current alignment methods such as refusal training is easy to be fooled by attacks.

Binary_CNN_with_bound

Fri, 18 Apr 2025 00:00:00 GMT

如何提高binaryCNN的准确性，这边文章是来自NIPS的Towards Accurate Binary Convolutional Neural 这篇文章有点旧了，时间是NIPS 2017，作者有Wei Pan,因此我可以在曼大找到他，我希望不会有太大的问题，我希望的是如何使用里面的bound 方法去利用bound,因为esbmc 同样使用了bound去找bug,这或许在神经网络训练也可以apply esbmc？

首先来看Abstract: 本文提到他们的贡献是introduce a novel scheme to train binary CNN. binary CNN是一种将原本float或者double的概率或者权重改成[-1,1]的binary CNN.好处是可以reduce memory size accesses.并且用bitwise operations代替了arithmetic operations.作者提到，好处是可以提高计算速度，并且减少功耗。对CNN进行binarization会造成预测精度的严重下降。本文提出两种方法去解决它。

使用多个二值权重基的线性组合来表示近似全精度权重。
使用多个二值激活函数来减少信息丢失。结果是正在ImageNet和forest trail数据集上的表现和全精度CNN的预测精度相当。

第一个方法并不难理解

方程如下： $$ W = \sum_{i=1}^{k} \alpha_i b_i $$ 这里面W是一个全精度权重，而b_i是二进制权重，而alpha是系数。这样如果我们能构造一个a_i可以使得W=B\alpha,那么我们就可以用二进制权重去表示全精度权重。

如何计算这个二进制权重来表示全精度权重

我们得到一个公式，应该让这样一个二进制权重去尽可能的靠近W $$ \min_{\alpha,B} J(\alpha,B) = ||w-B\alpha||^2 , s.t B_(ij) \in {-1,1} $$ 这个是W和B的方差。可是直接要求解这个公式是很难的，因此作者提出了对B_i进行一些简化。 1.用一个固定B_i,可以通过对W进行标准化来生成： $$B_i = F_{u_i}(W) := \text{sign}(\overline{W} + u_i \cdot \text{std}(W))$$ 其中 $$\overline{W} = W-mean(W)$$

将B_i代入到公式中，可以得到： $$ \min_{\alpha} J(\alpha) = ||W-\sum_{i=1}^{k} \alpha_i F_{u_i}(W)||^2 $$ 因此，我们可以通过求解这个公式来得到alpha。

alpha的计算公式可以转化成 $$ \alpha = \frac{W^T B}{B^T B} $$

Algorithm_MCTS

Fri, 18 Apr 2025 00:00:00 GMT

Today is about MCTS. Monte Carlo Tree Search.

First, let's recover the memory of the MCTS. Its algorithm is as follows:

$$ V(s) = \frac{\sum_{i=1}^{N} R_i}{N} $$

where:

$$V(s) $$is the value of state $$s$$,
$$R_i$$ is the reward received from the $$i$$-th simulation,
$$N$$ is the number of simulations from state $$s$$.

so, the Monte Carlo Tree Search is an algorithm with the idea of statistics.

#include <stdio.h>
#include <stdlib.h>
#include <time.h>
int main() {
    long point_inside_circle = 0;
    long i;
    double x, y;
    double distance_squared;
    double radius_squared = 1.0;
    int num_points = 10000000;
    double pi_estimate;
    srand(time(NULL));

    for (i = 0; i < num_points; i++) {
        x = (double)rand() / RAND_MAX;
        y = (double)rand() / RAND_MAX;
        distance_squared = x * x + y * y;
        if (distance_squared <= radius_squared) {
            point_inside_circle++;
        }
    }
    pi_estimate = 4.0 * point_inside_circle / num_points;
    printf("Estimated Pi: %f\n", pi_estimate);
    return 0;
}

Here we use the simulation of random dropping points in the range of [0, 1] to estimate the value of pi.

MCTS is quite good to use in the information hidden environment.

#include <stdio.h>
#include <stdlib.h>
#include <math.h>
#include <time.h>

// Possible actions in the game
#define ROCK 0 // Rock
#define SCISSORS 1 // Scissors
#define PAPER 2 // Paper
#define NUM_ACTIONS 3

// Probability of action of biased opponent
#define ROCK_PROB 50 // 50% of rock
#define SCISSORS_PROB 30 // 30% of scissors
#define PAPER_PROB 20 // 20% of paper

// Node structure
typedef struct {
int visits;
double wins;
double ucb;
} Node;

// Get opponent action based on probability distribution
int get_biased_opponent_action() {
int rand_num = rand() % 100;

if (rand_num < ROCK_PROB) {
return ROCK;
} else if (rand_num < ROCK_PROB + SCISSORS_PROB) {
return SCISSORS;
} else {
return PAPER;
}
}

// Use MCTS to select the best action and return win rate statistics
int mcts_best_action(int simulations, double win_rates[]) {
Node actions[NUM_ACTIONS] = {0};
int total_visits = 0;
int action_counts[NUM_ACTIONS] = {0};

// Main MCTS loop
for (int sim = 0; sim < simulations; sim++) {
// 1. Selection phase - using UCB formula
int selected_action = -1;
double best_ucb = -1.0;

for (int i = 0; i < NUM_ACTIONS; i++) {
if (actions[i].visits == 0) {
selected_action = i;
break;
}

// UCB calculation (exploration and exploitation balance)
actions[i].ucb = (actions[i].wins / actions[i].visits) +
sqrt(2.0 * log(total_visits) / actions[i].visits);

if (actions[i].ucb > best_ucb) {
best_ucb = actions[i].ucb;
selected_action = i;
}
}

// 2. Extension and 3. Simulation - guess opponent actions and simulate in an incomplete information environment
int opponent_action = get_biased_opponent_action();
action_counts[opponent_action]++;

// Calculation result: 0 = draw, 1 = win, -1 = lose
int result = 0;
if ((selected_action == ROCK && opponent_action == SCISSORS) ||
(selected_action == SCISSORS && opponent_action == PAPER) ||
(selected_action == PAPER && opponent_action == ROCK)) {
result = 1; // Victory
} else if (selected_action != opponent_action) {
result = -1; // Failure
}

// 4. Backpropagation
actions[selected_action].visits++;
actions[selected_action].wins += (result + 1) / 2.0; // Convert to [0,1] range
total_visits++;
}

// Calculate the winning rate of each action
for (int i = 0; i < NUM_ACTIONS; i++) {
if (actions[i].visits > 0) {
win_rates[i] = actions[i].wins / actions[i].visits;
} else {
win_rates[i] = 0.0;
}
}

// Print opponent action statistics
printf("Opponent action statistics (based on %d simulations):\n", simulations);
printf("- Rock: %.1f%%\n", (float)action_counts[ROCK] / simulations * 100);
printf("- Scissors: %.1f%%\n", (float)action_counts[SCISSORS] / simulations * 100);
printf("- Paper: %.1f%%\n", (float)action_counts[PAPER] / simulations * 100);
printf("\n");

// Select the action with the highest win rate
int best_action = 0;
double max_win_rate = win_rates[0];

for (int i = 1; i < NUM_ACTIONS; i++) {
if (win_rates[i] > max_win_rate) {
max_win_rate = win_rates[i];
best_action = i;
}
}

return best_action;
}

// Win rate evaluation name
const char* get_win_rate_evaluation(double rate) {
if (rate >= 0.7) return "excellent";
if (rate >= 0.6) return "very good";
if (rate >= 0.5) return "favorable";
if (rate >= 0.4) return "unfavorable";
if (rate >= 0.3) return "very bad";
return "very bad";
}

int main() {
const char* action_names[] = {"rock", "scissors", "cloth"};
srand(time(NULL));

printf("MCTS against preference opponent (50%%rock, 30%%scissors, 20%%cloth)\n");
printf("==================================================\n");

// Store win rate statistics
double win_rates[NUM_ACTIONS] = {0};

// Use MCTS to select actions and obtain win rate statistics
int best_action = mcts_best_action(100000, win_rates);

// Print win rate statistics table
printf("MCTS win rate statistics:\n");
printf("+--------+----------+-----------+-------------+\n");
printf("| Action | Win rate | Expected value | Evaluation |\n");
printf("+--------+----------+-----------+-------------+\n");

for (int i = 0; i < NUM_ACTIONS; i++) {
// Calculate expected value: win = 1, draw = 0, loss = -1 Convert back
double expected_value = (win_rates[i] * 2) - 1;

printf("| %-6s | %.2f%% | %+.2f | %-12s |\n",
action_names[i],
win_rates[i] * 100,
expected_value,
get_win_rate_evaluation(win_rates[i]));
}

printf("+--------+----------+-----------+-------------+\n");
printf("\nBest strategy: %s (win rate: %.2f%%)\n",
action_names[best_action], win_rates[best_action] * 100);

// Print theoretical optimal solution and compare with MCTS results
printf("\nTheoretical optimal strategy analysis:\n");
printf("- Rock: Expected value = 0.5×0(draw) + 0.3×1(win) + 0.2×(-1)(loss) = +0.10\n");
printf("- Scissors: Expected value = 0.5×(-1)(loss) + 0.3×0(draw) + 0.2×1(win) = -0.30\n");
printf("- Paper: Expected value = 0.5×1(win) + 0.3×(-1)(loss) + 0.2×0(draw) = +0.20\n");

return 0;
}

This is a rock-scissors-paper game. AI is hidden to be biased.

But the MCTS could find the best strategy to win the game.

Despite sometime writing a simulation code is not easy, but it is a good way to understand the algorithm.

C_code_practise

Fri, 18 Apr 2025 00:00:00 GMT

今天是2025年4月18日，今天练习一道medium。

你正在维护一个项目，该项目有 n 个方法，编号从 0 到 n - 1。

给你两个整数 n 和 k，以及一个二维整数数组 invocations，其中 invocations[i] = [ai, bi] 表示方法 ai 调用了方法 bi。

已知如果方法 k 存在一个已知的 bug。那么方法 k 以及它直接或间接调用的任何方法都被视为 可疑方法 ，我们需要从项目中移除这些方法。

只有当一组方法没有被这组之外的任何方法调用时，这组方法才能被移除。

返回一个数组，包含移除所有 可疑方法 后剩下的所有方法。你可以以任意顺序返回答案。如果无法移除 所有 可疑方法，则 不 移除任何方法。


#include <vector>
class Solution {
public:
    vector<int> remainingMethods(int n, int k, vector<vector<int>>& invocations) {

关于C++的vector，学习一下它和python的list的不同，C++的vector是提前规定类型的，而python的list是动态的。而操作来说，vector有push_back(),pop_back()

得到一个n functions 需要维护 with label 0 to n-1. input是一个n,k,一个[ai,bi]的二维数组。有点像一个linked list, 但是是二维的。给了一个例子n=4,k=1,invocations=[[0,1],[1,2],[3,2]] output:[0,1,2,3] 有点像esbmc,尽管需要free memory, 但是由于被后一个函数调用，所以目前不能free.

我们首先要标记哪些函数是可疑的。

class Solution {
public:
    vector<int> remainingMethods(int n, int k, vector<vector<int>>& invocations) {
        // 构建邻接表表示调用关系
        vector<vector<int>> graph(n);
        for (const auto& e : invocations) {
            graph[e[0]].push_back(e[1]);
        }
        
        // 使用DFS标记所有从k可达的节点为可疑
        vector<bool> isSuspicious(n, false);
        function<void(int)> dfs = [&](int node) {
            isSuspicious[node] = true;
            for (int next : graph[node]) {
                if (!isSuspicious[next]) {
                    dfs(next);
                }
            }
        };
        
        dfs(k);
        
        // 检查是否有非可疑方法调用可疑方法
        for (const auto& e : invocations) {
            if (!isSuspicious[e[0]] && isSuspicious[e[1]]) {
                // 如果有非可疑方法调用可疑方法，返回所有方法
                vector<int> allMethods(n);
                for (int i = 0; i < n; i++) {
                    allMethods[i] = i;
                }
                return allMethods;
            }
        }
        
        // 否则，返回所有非可疑方法
        vector<int> result;
        for (int i = 0; i < n; i++) {
            if (!isSuspicious[i]) {
                result.push_back(i);
            }
        }
        return result;
    }
};

这样的题目，特点是包含方法调用关系或者依赖关系，就应该想到图论。效率相当低，看了别人的解题思路是使用DFS进行标记，而不是两次遍历。

TODO：

学习一下C++的DFS和BFS
学习一下C++的图论

S3study

Thu, 17 Apr 2025 00:00:00 GMT

Security Issues and Explanations

English Name	Chinese Explanation	C	Java
Null Pointer Exception (NPE)	空指针异常	✅	✅
Memory Leak	内存泄漏	✅（频繁）	✅（容易被忽视）
OutOfMemoryError (OOM)	内存溢出	✅（可导致 crash）	✅（抛异常）
StackOverflowError	栈溢出	✅	✅
Buffer Overflow	缓冲区溢出	✅（致命）	❌（数组越界抛异常）
Use-After-Free	使用已释放内存	✅	❌（GC 管理）
Double Free	重复释放内存	✅	❌
Race Condition	多线程竞争	✅	✅
Deadlock	死锁	✅	✅
Dangling Pointer	悬挂指针	✅	❌
Denial of Service (DoS)	拒绝服务	✅（资源耗尽）	✅（如 OOM, 死循环）
Insecure Deserialization	不安全反序列化	❌（较少用）	✅（Java 原生有风险）
Sensitive Data Exposure	敏感信息泄露	✅	✅
Hardcoded Secrets	硬编码密钥	✅	✅
Path Traversal	路径穿越攻击	✅	✅
SQL Injection	SQL 注入	✅	✅（拼接 SQL）
XSS (Cross-Site Scripting)	跨站脚本攻击	✅（Web场景）	✅（Java Web中）
CSRF	跨站请求伪造	✅（Web场景）	✅（Spring 等 Web 框架）
Privilege Escalation	权限提升	✅	✅
Broken Access Control	访问控制失败	✅	✅
Remote Code Execution (RCE)	远程代码执行	✅（比如系统调用）	✅（如反序列化漏洞）
Directory Listing	目录浏览漏洞	✅	✅

Class 02

在C中: Fixd size data strucutre 有 single subscripted arrays, double subscripted arrays and structs

Dynamic data structures 比如 linked list, tree, graph, etc.

这一类被称为self referential structures,因为他们用一种类似指针的方式来引用自己。而这种会被null pointer exception 或者out of memory 攻击。

#include <stdio.h>
#include <stdlib.h>

struct node {
    int data;
    struct node *next;
};

int main() {
    // allocates memory for three nodes
    struct node *node1 = (struct node *)malloc(sizeof(struct node));
    struct node *node2 = (struct node *)malloc(sizeof(struct node));
    struct node *node3 = (struct node *)malloc(sizeof(struct node));
    
    node1->data = 15;
    node2->data = 10;
    node3->data = 5;
    
    // link nodes
    node1->next = node2;
    node2->next = node3;
    node3->next = NULL;
    
    // Print the linked list
    printf("node1 data: %d\n", node1->data);
    printf("node2 data: %d\n", node1->next->data);
    printf("node3 data: %d\n", node1->next->next->data);
    

    printf("Accessing freed memory: %d\n", node1->data);     
    
    printf("Accessing freed memory of node3: %d\n", node1->next->next->data); // 可能会崩溃
    
    // Deallocates memory allocated by malloc
    free(node1);
    free(node2);
    free(node3);

    return 0;
}

可运行的 C 代码示例

在这个示例中，我们有三个节点：node1、node2 和 node3。假设 node1 的 next 是 node2 的地址，而 node2 的地址被释放了，那么 node1 的 next 就变成了一个无效的地址。

比如这种，假设 node1 的 next 目前是 node2 的地址，但是 node2 的地址被释放了，那么 node1 的 next 就变成了一个无效的地址。如果这个链不止在这停止，那么后续整个链表都会变成无效的地址。

#include <stdio.h>
#include <stdlib.h>

struct node {
    int data;
    struct node *next;
};

int main() {
    // allocates memory for three nodes
    struct node *node1 = (struct node *)malloc(sizeof(struct node));
    struct node *node2 = (struct node *)malloc(sizeof(struct node));
    struct node *node3 = (struct node *)malloc(sizeof(struct node));
    
    node1->data = 15;
    node2->data = 10;
    node3->data = 5;
    
    // link nodes
    node1->next = node2;
    node2->next = node3;
    node3->next = NULL;
    
    // Print the linked list
    printf("node1 data: %d\n", node1->data);
    printf("node2 data: %d\n", node1->next->data);
    printf("node3 data: %d\n", node1->next->next->data);
    
    // Deallocates memory allocated by malloc

    free(node2);
    
    // Attempt to access freed memory (Dangling Pointer)
    // 这将导致未定义行为，可能会崩溃或输出垃圾值
    printf("Accessing freed memory: %d\n", node1->data); // 可能会崩溃或输出垃圾值
    
    // 进一步演示攻击造成的崩溃
    printf("Accessing freed memory of node3: %d\n", node1->next->next->data); // 可能会崩溃
    
    return 0;
}

Program terminated with signal: SIGSEGV

说明了，如果一个指针指向的内存被释放了，那么这个指针就会变成一个悬挂指针，如果继续使用这个指针，就会导致未定义行为，可能会崩溃或输出垃圾值。

这样的错误应该在java中被避免。因为java有垃圾回收机制，将值设置为null,并不会造成ptr被释放。

以下是java的代码示例：

class Node {
    int data;
    Node next;

    Node(int data) {
        this.data = data;
        this.next = null;
    }
}

class LinkedList {
    public static void main(String[] args) {
        // allocates memory for three nodes
        Node node1 = new Node(15);
        Node node2 = new Node(10);
        Node node3 = new Node(5);
        
        // link nodes
        node1.next = node2;
        node2.next = node3;
        node2 = null; // 释放 node2，node1.next 现在指向 null
        
        // Print the linked list
        System.out.println("node1 data: " + node1.data);
        System.out.println("node2 data: " + node1.next.data); 
        System.out.println("node2 data: " + node2);
        System.out.println("node3 data: " + node1.next.next.data); 
    }
}

node1 data: 15
node2 data: 10
node2 data: null
node3 data: 5

ESBMC_bug_fixing

Wed, 16 Apr 2025 00:00:00 GMT

Here is used to record what I have done and how the UI looks like.

Currently, the AI fixing part is not working well. first the result is not catched as my expected. I can get the result print but failed to show it in the frontend.

Besides that, the repair is quite bad. It tried to fix the bug but just adding a lot of meaningless if statements. To specific, you know, that is not how bug fixing works.

ESBMC_bug_fixing

I have just fixed the bug in the repair suggestion part but the performance is just too bad. DeepSeek and Claude are just trying to add a If statement to change to code to make it work instead of actually fixing the state transition.

Performance is bad. But how to evaluate the fixing? If only based on the assertion properties, it may look like the fixing is correct. Holy Moly!

Hard to continue. I need to talk with my supervisors.

Reading_Club_1

Sat, 11 Jan 2025 00:00:00 GMT

@[TOC](# 立场：理解大模型需要的不仅仅是统计概括)

Position: Understanding LLMs Requires More Than Statistical Generalization

这是我们LLM reading club（一周一次或者两周一次）准备读的内容，我英文比较差，先读完然后再去聊，这周开始，我会尽可能准备好并上传到CSDN，如果正好有人也想读这些文章，那么大家正好一起看一下。我的特点是数学基础不够好，所以在门槛难度上，大家可以放心阅读。我会尽可能用简单的方法给大家分享的。我看不懂或者忘记的地方也会给一点原文没有的案例的，还有如果有说错的地方也可指导我一下，感谢。

当然文章现在阅读很简单，大家问大模型就可以了，我也使用了大模型来解释我不清楚的点，不过起码可以把这个内容当作中文翻译用一用。

这是一篇来自马普智能系统研究所，剑桥大学，UCL，ETH, etc. 的一篇文章主要是理解大规模语言模型（LLMs）需要超越传统的统计泛化理论的文章。链接是 https://arxiv.org/pdf/2405.01964

背景

关于泛化能力的探讨：自回归的（Autoregressive (AR) language）（指的是通过next token prediction的结果进行下一次prediction，比如GPT，PaLM之类的模型）在内容理解和基于数据的fine tuning都表现的很好。过去有的研究说是因为interpolation regime（插值范式）。

tips：插值范式指的是训练模型的状态，神经网络在训练集上训练误差会达到零或者接近零，即模型完全拟合训练数据。插值意思是模型学会了通过训练样本点，并将数据的所有模式完全捕获（我：这可能也导致了大家有一个理念：模型有更多参数，就可以去拟合任何任务）

传统逻辑提出，大模型在大量样本上进行训练，并且基于训练，达到了很低的loss，因此它们在统计任务上有好的表现。但是“统计任务”这词本身无法统述整个下游任务。

尽管不同模型（M1 和 M2）在训练生成 $a^n$$b^n$ 语言上实现了相同的最小测试损失，但在分布外（OOD）提示下，它们的行为可能不同，这种规则外推能力取决于transformer模型的归纳偏置能力，而非训练测试损失而获得。

核心观点

因此本文提出了我们应该从**饱和范式（saturation regime）**的角度去理解大模型：
- 在训练过程中，模型不仅在训练集上达到了全局最小损失，也在测试集上达到了最小测试损失。
- 即模型已经在目前分布内的数据已经在统计学上达到了最优损失。
问题的提出：最优测试损失不足以解释模型表现
- 即使在饱和范式下，不同的模型仍然会在分布外的测试数据上有较大差异
- 这种情况说明：最低的测试损失不足以区分模型在复杂任务（比如out-of-distribution (OOD)的任务）上的表现差异。（我：让我想到code generation，大家一开始都在用OpenAI的HumanEval，但是等到即便HumanEval的表现非常出色的时候，处理实际github任务的表现依然堪忧，当然现在已经不错了）

可能的研究方向，模型的归纳偏置，任务的迁移能力，上下文学习能力。

这篇文章的贡献：

- 强调了使用统计概括来理解研究达到饱和范式LLM的局限性
- 用三个案例来证明上面说的局限性
- 提出研究饱和范式下LLM的三种可能的方向

文章阅读

OK，来到文章部分。

第一部分是background。

关于Statistical generalization（统计泛化），关键的知识点是，统计泛化是衡量模型是否能够转移到相同分部的未见测试数据上，这个与大家入门ML课程应该是一样的。这个概念本身很难解释现在DL的成功。除此以外，统计泛化的一个重要优点是其“黑箱性”，统计泛化不需要特定知识（比如pattern based 的方法的就是反例了，pattern写起来就很累嘛），适用范围广，只依赖于损失函数的定义，不需要考虑具体任务的定性。其他不是很重要，先跳过。

关于Interpolation regime,前面tips里面聊过了。

关于Identifiability of Probabilistic Models，可识别性，是指统计模型是否可以从观测到的数据中唯一的恢复模型。举例，比如clustering的任务中，有白点和黑点分类，那白点和黑点的分组，分组本身是唯一的，因为他们的特征是清晰的，但是如果是一个像素点，他有灰度，那么它的分类可能是模糊的，那么就不具备可识别性了，当然任务简单，其实加入简单的pattern 比如灰度>half就是黑，就能解决，但是很多任务的评价维度其实很大，当定义损失函数也会很难（跑题了，当我没说😅）。

来到正文

AR（自回归）模型的可识别性，自回归模型指

(对于固定的 $L ∈ N$) 的条件分布 ${p(xi |x1:i−1);L ≥ i ≥ 1}$ , 这些分布根据链式规则可以得出 ${p(x1:i);L ≥ i ≥ 1}$ 在整个sequence上.

即这个模型的每个元素的概率都依赖它之前的元素，并且基于链式规则，由条件逐步构建出完整序列的概率分布。

不可识别性：

功能不可识别性：基于上述的公式，我们可以发现，基于统计的AR models除非前面的prefixes是固定的，不然是无法从结果来判断功能是否是一致的。这是因为如果这些前缀如果未在数据上出现，那么模型在这些前缀上的行为可能因为不同的设计或者归因而无法从统计来确定唯一性（比如我目前项目中使用的中间态，如果忽略中间态的正确性，去判断最后结果的一致性，可能在跨数据集的任务上表现有差距，不过我还没试过）。
ε−不可识别性（Mac按一下fn键然后输入 eps），这是本文提出的新概念，ε是个阈值，表示模型在KL 散度（Kullback-Leibler divergence）意义上的接近程度，如果模型$p$与模型$q$在KL散度小于ε，说明两个模型统计意义上非常相似。ε不可识别性之处，尽管接近，但是模型在统计意义上的接近可能在实际功能上会有显著的不同。即使存在唯一的全局最优点，模型的相似最优点之间仍然可能存在较大差异，以至于会影响实际的能力。
参数（parameters）不可识别性，这个比较容易理解了，不同的参数组合可以实现同一个离散分布的功能，也就是说，它们在零样本测试（zero-shot）任务中的表现可能是相同的。但由于这些模型的内部参数不同，它们在迁移学习（transfer learning）和微调（fine-tuning）过程中，可能表现出很大的差异。

例子1:非可识别性与规则外推（Rule Extrapolation）

给定一个自回归模型$q$让其拟合基于上下文无关语法（probabilistic context-free grammar，PCFG）$p$的句子生成任务，句子形式为$a^nb^n$,其中n是随机的。这种语法本身的支持是非常有限的比如abaa的出现概率就为0，这样的任务就是上文提到的OOD prompt。

案例：

如果出现 $KL [p(x{1:k})||q(x_{1:k})] = 0$那么我们可以说模型q在统计意义上完全拟合了任务分布$p$。即模型的分布与目标分布完全一致，模型在训练数据和测试数据上达到了最大似然估计（Maximum Likelihood Estimation, MLE）. 如果忘了看一下一个似然函数的公式。表示的是在给定参数的条件，观测到的数据的联合概率： $$ L(\theta) = p(x_1, x_2, \dots, x_n \mid \theta) = \prod_{i=1}^n p(x_i \mid \theta) $$ 最大似然估计使用argmax，来找到参数$\theta$使得似然函数最大化： $$ \hat{\theta} = \arg\max_{\theta}L(\theta) $$ 在这种情况下，模型达到了对观测数据的最佳模拟。

回到本文的论述，可以发现即便$KL [p(x{1:k})||q(x_{1:k})] = 0$，当$p(x_{1:l})=0$，即在支持域之外时，模型的行为的定义是不明确的。这意味着任何property of 模型$q$在基于OOD prompts都是非唯一确定的（non-identificable）。其中，规则外推（rule extrapolation）问题涉及两个规则：

R1: $a$和$b$的数量必须匹配
R2: b后面不能出现$a$

本文感兴趣的点是，模型是否在未显示表示规则的情况下，遵循规则R1.

实例分析：

在这样的任务上，使用MLE的方法训练了基于解码器的Transformer模型，目标是生成$a^nb^n$的PCFG的sequence。上文提到的R1并未被显示要求。模型在OOD prompts（比如abaa）的表现是评估的重点。本文训练的transformer模型在长度为8的OOD提示上，可以以43.7%的准确率符合R1的序列。然后研究添加了两种附加模型，通过引入有监督损失对规则R1施加不同的影响：

对抗模型：使用了污染数据误导模型对R1对学习。
Oracle模型，通过额外的监督数据显示加强模型对R1的学习。

结果：尽管三种模型都在数据集上实现了相同的最小测试损失但是在规则外推的表现差距明显。这显示出模型即使没有明确的监督，仍然倾向于推断某些规律性。

例子2: ε不可识别性与情景学习（in-context learning (ICL)）

在实际的任务中，训练集中可能OOD prompt是一个非常小概率的情况，而不是完全不出现，目前有一个分布$q$，即模型学到的分布，p是训练数据的分布。而ICL是模型需要学习的规则，我们可以看到，有些规则可能是部分出现在$p$中的。

其中KL散度表示模型的$q$分布是否完全表示了$p$的情况，ε表示一个大于0的很小的差异。根据图片，$q$仍然能表示一些在$p$分布以外的ICL内容。

ICL的定义：模型能够基于提示中提供的input-output pair(e.g. ($x_i,y_i$)),在不更新参数的情况下直接从提示中学习任务并正确预测输入$x_{test}$的输出$y_{test}$

研究设计

目标分布$p$: 作者构造了一组HMM的混合生成，作为与训练分布，提示中的input-output pairs和测试输入都是同一个概念$\theta$下的HMM
对比分布$q_n$:
- 修改$p$的部分概率分布，仅仅在prompt structure相关的序列上进行调整。
- 其他部分，$q_n$和$p$完全保持一致
调整方法
- 针对提示完成的概率$(y \mid S_n, x_\text{test})$,将最高概率和次高概率的值稍做交换，以确保： $$\arg\max_y q_n(y \mid S_n, x_\text{test}) \neq \arg\max_y p(y \mid S_n, x_\text{test}) $$

这种操作对KL散度的影响极小，调整后的KL始终满足<ε.

这样的调整证明了尽管保持KL散度在一个极小范围内变化，却可以摧毁ICL，也意味着，不同的模型或者分布即便在统计意义上接近，它们的ICL能力可能是完全不同的。

PS: 原文在附录里加入了很多数学论证，我跳过了，希望大家可以去看下证明逻辑有没有问题。

例子3: parameters的不可识别性

这次的实验，研究者将一个小型transformer嵌入到一个更大transformer中，并确保功能等价性。如果参数可感知，那么两者在预训练的损失完全相同的情况下，他们在OOD的下游任务上的表现应该相同。

然而，结果是在微调后，二者的表现有显著差异，大transformer能高出10%的下游任务准确率。

这表明模型的参数话和架构会显著影响微调和迁移学习的能力，这些能力可能和模型的归纳偏差的能力有关。

饱和阶段应该测量什么

既然loss和分布内的任务无法区分模型的正确率，那么什么是饱和阶段的研究方向，本文提出了三个。

泛化测量

应该研究

组合泛化：理解模型是否能够通过已知的组件组合推导出新知识。
系统泛化：是否能在理解复杂任务中系统性推理。
符号泛化：是否能处理形式语言（比如PCFG Probabilistic Context-Free Grammar）。

可迁移性

目前的迁移测量指标不足以可靠评估微调后的性能应该设计新的度量方式来评估微调和迁移能力。

归纳偏置

应该研究哪些归纳偏置能够通过参数空间或者函数空间来诱导模型的特定属性。归纳偏置可以帮助解释LLMs' OOD泛化和迁移的能力，超越仅靠着训练损失获得的性能。

讨论，未来的研究方向

更好的泛化测量指标
- 需要开发更好的测量指标，比如组合泛化能力的测试，如通过理解'one', 'cat', 'black'，测试模型能否推导出'one cat', 'black cat' 的组合泛化能力
- 拓广统计泛化的框架，将经典的Bayes泛化理论推广到更复杂的比如生成模型，元学习和处理相互关联的非i.i.d数据上去。
计算语言建模
- 理解语言结构中的统计规律，以及这些规律如何影响模型的泛化和推理能力
归纳偏置的研究
- 研究饱和阶段，不同参数表现出的不同的下游性能。
- 选择合适的归纳偏置是否能捕获特定的下游任务属性，从而克服模型的不可识别性的限制。
- 研究哪些归纳偏置可以引导模型有更好的OOD能力和迁移能力。

评价

一篇还可以的文章，但是主要是针对他对现有方法的批评，对未来的评价体系的归纳主要还是靠猜想，并没有提出新的方法，比如如何去推广到新的任务，或者用什么具体的归纳偏置的分析方式。不过本文提供了很多数学的证明，严谨性应该没问题，大家可以去看看。

Update 讨论：

今天（1.14）有一个reading club的成员讨论，对于OOD的数据进行test，然后再比较，难到不也是使用统计概括吗。 Edoardo：本文可能值得只是in distribution部分的统计不足以概括一个LLM吧，比如同样两个英文-中文翻译器，都没有见过spanish，但是可能表现是不同的，可能一个可以在higher level对语言的structure有更好的理解，而另外一个是记住了pattern。

这次meeting，大家都去旅游了，大家都没提前读，没有什么其他有价值的，就读了读论文。

Markdown Extended Features

Wed, 01 May 2024 00:00:00 GMT

GitHub Repository Cards

You can add dynamic cards that link to GitHub repositories, on page load, the repository information is pulled from the GitHub API.

::github{repo="Fabrizz/MMM-OnSpotify"}

Create a GitHub repository card with the code ::github{repo="<owner>/<repo>"}.

::github{repo="saicaca/fuwari"}

Admonitions

Following types of admonitions are supported: note tip important warning caution

:::note Highlights information that users should take into account, even when skimming. :::

:::tip Optional information to help a user be more successful. :::

:::important Crucial information necessary for users to succeed. :::

:::warning Critical content demanding immediate user attention due to potential risks. :::

:::caution Negative potential consequences of an action. :::

Basic Syntax

:::note
Highlights information that users should take into account, even when skimming.
:::

:::tip
Optional information to help a user be more successful.
:::

Custom Titles

The title of the admonition can be customized.

:::note[MY CUSTOM TITLE] This is a note with a custom title. :::

:::note[MY CUSTOM TITLE]
This is a note with a custom title.
:::

GitHub Syntax

[!TIP] The GitHub syntax is also supported.

> [!NOTE]
> The GitHub syntax is also supported.

> [!TIP]
> The GitHub syntax is also supported.

Simple Guides for Fuwari

Mon, 01 Apr 2024 00:00:00 GMT

Cover image source: Source

This blog template is built with Astro. For the things that are not mentioned in this guide, you may find the answers in the Astro Docs.

Front-matter of Posts

---
title: My First Blog Post
published: 2023-09-09
description: This is the first post of my new Astro blog.
image: ./cover.jpg
tags: [Foo, Bar]
category: Front-end
draft: false
---

Attribute	Description
`title`	The title of the post.
`published`	The date the post was published.
`description`	A short description of the post. Displayed on index page.
`image`	The cover image path of the post.<br/>1. Start with `http://` or `https://`: Use web image<br/>2. Start with `/`: For image in `public` dir<br/>3. With none of the prefixes: Relative to the markdown file
`tags`	The tags of the post.
`category`	The category of the post.
`draft`	If this post is still a draft, which won't be displayed.

Where to Place the Post Files

Your post files should be placed in src/content/posts/ directory. You can also create sub-directories to better organize your posts and assets.

src/content/posts/
├── post-1.md
└── post-2/
    ├── cover.png
    └── index.md

Markdown Example

Sun, 01 Oct 2023 00:00:00 GMT

An h1 header

Paragraphs are separated by a blank line.

2nd paragraph. Italic, bold, and monospace. Itemized lists look like:

this one
that one
the other one

Note that --- not considering the asterisk --- the actual text content starts at 4-columns in.

Block quotes are written like so.

They can span multiple paragraphs, if you like.

Use 3 dashes for an em-dash. Use 2 dashes for ranges (ex., "it's all in chapters 12--14"). Three dots ... will be converted to an ellipsis. Unicode is supported. ☺

An h2 header

Here's a numbered list:

first item
second item
third item

Note again how the actual text starts at 4 columns in (4 characters from the left side). Here's a code sample:

# Let me re-iterate ...
for i in 1 .. 10 { do-something(i) }

As you probably guessed, indented 4 spaces. By the way, instead of indenting the block, you can use delimited blocks, if you like:

define foobar() {
    print "Welcome to flavor country!";
}

(which makes copying & pasting easier). You can optionally mark the delimited block for Pandoc to syntax highlight it:

import time
# Quick, count to ten!
for i in range(10):
    # (but not *too* quick)
    time.sleep(0.5)
    print i

An h3 header

Now a nested list:

First, get these ingredients:
- carrots
- celery
- lentils
Boil some water.

Dump everything in the pot and follow this algorithm:

 find wooden spoon
 uncover pot
 stir
 cover pot
 balance wooden spoon precariously on pot handle
 wait 10 minutes
 goto first step (or shut off burner when done)

Do not bump wooden spoon or it will fall.

Notice again how text always lines up on 4-space indents (including that last line which continues item 3 above).

Here's a link to a website, to a local doc, and to a section heading in the current doc. Here's a footnote [^1].

[^1]: Footnote text goes here.

Tables can look like this:

size material color

9 leather brown 10 hemp canvas natural 11 glass transparent

Table: Shoes, their sizes, and what they're made of

(The above is the caption for the table.) Pandoc also supports multi-line tables:

keyword text

red Sunsets, apples, and other red or reddish things.

green Leaves, grass, frogs and other things it's not easy being.

A horizontal rule follows.

Here's a definition list:

apples : Good for making applesauce. oranges : Citrus! tomatoes : There's no "e" in tomatoe.

Again, text is indented 4 spaces. (Put a blank line between each term/definition pair to spread things out more.)

Here's a "line block":

| Line one | Line too | Line tree

and images can be specified like so:

Inline math equations go in like so: $\omega = d\phi / dt$. Display math should get its own line and be put in in double-dollarsigns:

$$I = \int \rho R^{2} dV$$

$$ \begin{equation*} \pi =3.1415926535 ;8979323846;2643383279;5028841971;6939937510;5820974944 ;5923078164;0628620899;8628034825;3421170679;\ldots \end{equation*} $$

And note that you can backslash-escape any punctuation characters which you wish to be displayed literally, ex.: `foo`, *bar*, etc.

Include Video in the Posts

Tue, 01 Aug 2023 00:00:00 GMT

Just copy the embed code from YouTube or other platforms, and paste it in the markdown file.

---
title: Include Video in the Post
published: 2023-10-19
// ...
---

<iframe width="100%" height="468" src="https://www.youtube.com/embed/5gIf0_xpFPI?si=N1WTorLKL0uwLsU_" title="YouTube video player" frameborder="0" allowfullscreen></iframe>

YouTube

Bilibili