3

我正在尝试编写一个与 dlmalloc 相当的内存分配器,它是 glibc 中使用的 malloc。dlmalloc 是具有块拆分功能的最佳分配器,它保留了最近使用的块池,然后再次将块合并为大块。我正在编写的分配器首先适合样式。

我的问题有两个:(1)与 glibc malloc 相比,我的代码的测试时间非常不规则;(2)有时我的代码的平均运行时间会长 3 到 4 倍;(2) 没什么大不了的,但我想了解为什么 glibc malloc 不会受到同样的影响。在这篇文章中进一步展示了(1)中描述的 malloc 和我的代码之间的行为示例。有时,一批 1000 次测试的平均时间会比 malloc 高得多(上面的问题(2)),有时平均值是相同的。但是,对我的代码进行的一批测试的测试时间总是非常不规则(上面的问题(1));这意味着在一批测试中有时间跳跃到平均值的 20 倍,并且这些跳跃穿插在其他规则(接近平均)时间中。glibc malloc 不这样做。

我正在处理的代码如下。

====================================

/* represent an allocated/unallocated  block of memory */
struct Block {

    /* previous allocated or unallocated block needed for consolidation but not used in allocation */
    Block* prev;
    /* 1 if allocated and 0 if not */
    unsigned int tagh;
   /* previous unallocated block */
   Block* prev_free;
   /* next unallocated block  */
   Block* next_free;
   /* size of current block */
   unsigned int size;
};

#define CACHE_SZ 120000000

/* array to be managed by allocator */
char arr[CACHE_SZ] __attribute__((aligned(4)));

/* initialize the contiguous memory located at arr for allocator */
void init_cache(){
/* setup list head node that does not change */
   Block* a = (Block*)  arr;
  a->prev = 0; 
  a->tagh = 1;
  a->prev_free = 0;
  a->size = 0;

/* setup the usable data block */
  Block* b = (Block*) (arr + sizeof(Block));
  b->prev = a; 
  b->tagh = 0;
  b->prev_free = a;
  b->size = CACHE_SZ - 3*sizeof(Block);
  a->next_free = b;

/* setup list tail node that does not change */
  Block* e = (Block*)((char*)arr + CACHE_SZ - sizeof(Block)); 
  e->prev = b;
  e->tagh = 1;
  e->prev_free = b;
  e->next_free = 0;
  e->size = 0;
  b->next_free = e;
}

char* alloc(unsigned int size){
  register Block* current = ((Block*) arr)->next_free; 
  register Block* new_block;

/* search for a first-fit block */

   while(current != 0){
       if( current->size >= size + sizeof(Block)) goto good;
       current = current->next_free;
   }

/* what to do if no decent size block found */
   if( current == 0) {
       return 0;
   }

/* good block found */
good:
/* if block size is exact return it */
   if( current->size == size){
       if(current->next_free != 0) current->next_free->prev_free = current->prev_free;
       if(current->prev_free != 0) current->prev_free->next_free = current->next_free;
       return (char* ) current + sizeof(Block);
   }

/* otherwise split the block */

   current->size -= size + sizeof(Block); 

    new_block = (Block*)( (char*)current + sizeof(Block) + current->size);
    new_block->size = size;
    new_block->prev = current;
    new_block->tagh = 1;
   ((Block*)((char*) new_block + sizeof(Block) + new_block->size ))->prev = new_block;

   return (char* ) new_block + sizeof(Block);
}

main(int argc, char** argv){
    init_cache();
    int count = 0;

/* the count considers the size of the cache arr */
    while(count < 4883){

/* the following line tests malloc; the quantity(1024*24) ensures word alignment */
   //char * volatile p = (char *) malloc(1024*24);
/* the following line tests above code in exactly the same way */
    char * volatile p = alloc(1024*24);
        count++;

    }
}

======================================

我简单地编译了上面的代码:

g++ -O9 alloc.c

并运行一个简单的测试,该测试总是会拆分块并且永远不会返回确切大小的块:

bash$ for((i=0; i<1000; i++)); do (time ./a.out) 2>&1|grep real; 完毕

我的代码和 glibc malloc 的测试示例输出如下:

我的代码:

real    0m0.023s
real    0m0.109s    <----- irregular jump >
real    0m0.024s
real    0m0.086s
real    0m0.022s
real    0m0.104s    <----- again irregular jump >
real    0m0.023s
real    0m0.023s
real    0m0.098s
real    0m0.023s
real    0m0.097s
real    0m0.024s
real    0m0.091s
real    0m0.023s
real    0m0.025s
real    0m0.088s
real    0m0.023s
real    0m0.086s
real    0m0.024s
real    0m0.024s

malloc 代码(良好且经常保持接近 20 毫秒):

real    0m0.025s
real    0m0.024s
real    0m0.024s
real    0m0.026s
real    0m0.024s
real    0m0.026s
real    0m0.025s
real    0m0.026s
real    0m0.026s
real    0m0.025s
real    0m0.025s
real    0m0.024s
real    0m0.024s
real    0m0.024s
real    0m0.025s
real    0m0.026s
real    0m0.025s

请注意,malloc 代码时间更规律。在其他不可预测的时间,我的代码有 0m0.070s 而不是 0m0.020s,因此平均运行时间接近 70ms 而不是 25ms(上面的问题 (2)),但这里没有显示。在这种情况下,我很幸运能够让它运行接近 malloc 的平均值(25ms)

问题是,(1)如何修改我的代码以拥有更多的常规时间,例如 glibc malloc ?和(2)如果可能的话,我怎样才能使它比 glibc malloc 更快,因为我已经读到 dlmalloc 是一个典型的平衡分配器,并不是最快的(只考虑拆分/最佳匹配/首次匹配分配器而不是其他分配器) ?

4

2 回答 2

5

是的,我比较了这两种解决方案,并以几种不同的变体运行它们。我不确定问题出在哪里,但我的猜测是大部分时间都花在“创建一个 1200000000 字节的大型连续平板”上。如果我减小大小,并且仍然执行相同数量的分配,时间就会减少。

另一个指向这一点的证据是system时间是时间的很大一部分real,而user时间几乎没有。

现在,在我的系统上,一旦我以高内存负载运行这些东西几次,它并没有真正上下摆动。这很可能是因为一旦我换掉了堆积在内存中的一堆旧垃圾,系统就会有大量“备用”页面用于我的进程。当内存更加受限时(因为我让系统去做一些其他事情,比如在我试验的“网站”上做一些数据库工作[它是一个真实网站的“沙盒”版本,所以它有数据库中的真实数据,并且可以快速填充内存等],我得到了更多的变化,直到我再次清理了相当多的内存。

但我认为“神秘”的关键在于系统时间是所用时间的很大一部分。同样值得注意的是,当使用malloc大块时,内存实际上并没有被“真正分配”。而且在分配较小的块时,它似乎malloc在某些方面实际上更聪明,并且比“优化”分配更快 - 至少对于更大的内存量。不要问我这到底是怎么工作的。

这里有一些证据:

我将main代码中的更改为:

#define BLOCK_SIZE (CACHE_SZ / 5000)

int main(int argc, char** argv){
    init_cache();
    int count = 0;
    int failed = 0;
    size_t size = 0;

/* the count considers the size of the cache arr */
    while(count < int((CACHE_SZ / BLOCK_SIZE) * 0.96) ){

/* the following line tests malloc; the quantity(1024*24) ensures word alignment */
   //char * volatile p = (char *) malloc(1024*24);
/* the following line tests above code in exactly the same way */
    char * volatile p;
    if (argc > 1) 
        p = (char *)malloc(BLOCK_SIZE);
    else
        p = alloc(BLOCK_SIZE);
    if (p == 0)
    {
        failed++;
        puts("p = NULL\n");
    }
    count++;
    size += BLOCK_SIZE;
    }
    printf("Count = %d, total=%zd, failed=%d\n", count, size, failed);
}

然后改变 CACHE_SZ 并在有或没有使用allocmalloc选项的参数的情况下运行:

因此,缓存大小为 12000000 (12MB):

这些数字是:

real    0m0.008s
user    0m0.001s
sys 0m0.007s
Count = 4800, total=11520000, failed=0

real    0m0.007s
user    0m0.000s
sys 0m0.006s
Count = 4800, total=11520000, failed=0

real    0m0.008s
user    0m0.001s
sys 0m0.006s
Count = 4800, total=11520000, failed=0

real    0m0.014s
user    0m0.003s
sys 0m0.010s

还有一些运行malloc

real    0m0.010s
user    0m0.000s
sys 0m0.009s
Count = 4800, total=11520000, failed=0

real    0m0.017s
user    0m0.001s
sys 0m0.015s
Count = 4800, total=11520000, failed=0

real    0m0.012s
user    0m0.001s
sys 0m0.010s
Count = 4800, total=11520000, failed=0

real    0m0.021s
user    0m0.007s
sys 0m0.013s
Count = 4800, total=11520000, failed=0

real    0m0.010s
user    0m0.001s
sys 0m0.008s
Count = 4800, total=11520000, failed=0

real    0m0.009s
user    0m0.001s
sys 0m0.007s

将缓存大小增加 10 倍会得到以下结果alloc

real    0m0.038s
user    0m0.001s
sys 0m0.036s
Count = 4800, total=115200000, failed=0

real    0m0.040s
user    0m0.001s
sys 0m0.037s
Count = 4800, total=115200000, failed=0

real    0m0.045s
user    0m0.001s
sys 0m0.043s
Count = 4800, total=115200000, failed=0

real    0m0.044s
user    0m0.001s
sys 0m0.043s
Count = 4800, total=115200000, failed=0

real    0m0.046s
user    0m0.001s
sys 0m0.043s
Count = 4800, total=115200000, failed=0

real    0m0.042s
user    0m0.000s
sys 0m0.042s

并与malloc

real    0m0.026s
user    0m0.004s
sys 0m0.021s
Count = 4800, total=115200000, failed=0

real    0m0.027s
user    0m0.002s
sys 0m0.023s
Count = 4800, total=115200000, failed=0

real    0m0.022s
user    0m0.002s
sys 0m0.018s
Count = 4800, total=115200000, failed=0

real    0m0.016s
user    0m0.001s
sys 0m0.015s
Count = 4800, total=115200000, failed=0

real    0m0.027s
user    0m0.002s
sys 0m0.024s
Count = 4800, total=115200000, failed=0

还有另外 10 倍alloc

real    0m1.408s
user    0m0.002s
sys 0m1.395s
Count = 4800, total=1152000000, failed=0

real    0m1.517s
user    0m0.001s
sys 0m1.505s
Count = 4800, total=1152000000, failed=0

real    0m1.478s
user    0m0.000s
sys 0m1.466s
Count = 4800, total=1152000000, failed=0

real    0m1.401s
user    0m0.001s
sys 0m1.389s
Count = 4800, total=1152000000, failed=0

real    0m1.445s
user    0m0.002s
sys 0m1.433s
Count = 4800, total=1152000000, failed=0

real    0m1.468s
user    0m0.000s
sys 0m1.458s
Count = 4800, total=1152000000, failed=0

malloc

real    0m0.020s
user    0m0.002s
sys 0m0.017s
Count = 4800, total=1152000000, failed=0

real    0m0.022s
user    0m0.001s
sys 0m0.020s
Count = 4800, total=1152000000, failed=0

real    0m0.027s
user    0m0.005s
sys 0m0.021s
Count = 4800, total=1152000000, failed=0

real    0m0.029s
user    0m0.002s
sys 0m0.026s
Count = 4800, total=1152000000, failed=0

real    0m0.020s
user    0m0.001s
sys 0m0.019s
Count = 4800, total=1152000000, failed=0

如果我们将代码更改为常数 1000,则和BLOCK_SIZE之间的差异会变得更小。结果如下:allocmallocalloc

 Count = 1080000, total=1080000000, failed=0

real    0m1.183s
user    0m0.028s
sys 0m1.137s
Count = 1080000, total=1080000000, failed=0

real    0m1.179s
user    0m0.017s
sys 0m1.143s
Count = 1080000, total=1080000000, failed=0

real    0m1.196s
user    0m0.026s
sys 0m1.152s
Count = 1080000, total=1080000000, failed=0

real    0m1.197s
user    0m0.023s
sys 0m1.157s
Count = 1080000, total=1080000000, failed=0

real    0m1.188s
user    0m0.021s
sys 0m1.147s

现在malloc

Count = 1080000, total=1080000000, failed=0

real    0m0.582s
user    0m0.063s
sys 0m0.482s
Count = 1080000, total=1080000000, failed=0

real    0m0.586s
user    0m0.062s
sys 0m0.489s
Count = 1080000, total=1080000000, failed=0

real    0m0.582s
user    0m0.059s
sys 0m0.483s
Count = 1080000, total=1080000000, failed=0

real    0m0.590s
user    0m0.064s
sys 0m0.477s
Count = 1080000, total=1080000000, failed=0

real    0m0.586s
user    0m0.075s
sys 0m0.473s
于 2013-09-05T15:43:49.113 回答
5

不要使用“实时”时间:尝试“用户”+“系统”。大量迭代的平均值。问题是双重的:(a)您的进程在处理器上并不孤单,它会根据其他进程的操作而中断,(b)时间测量具有粒度。我不确定今天是什么,但在以前它只是时间片的大小 => 1/100 秒。

于 2013-09-05T15:14:04.977 回答