12

I'm trying to understand how data is interpreted in Caffe. For that I've taken a look at the Minst Tutorial Looking at the input data definition:

layers {
  name: "mnist"
  type: DATA
  data_param {
    source: "mnist_train_lmdb"
    backend: LMDB
    batch_size: 64
    scale: 0.00390625
  }
  top: "data"
  top: "label"
}

I've now looked at the mnist_train_lmdb and taken one of the entries (shown in hex):

0801101C181C229006
00000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000
00000000000000000000000000000000000000000000000000000000
00000000000054B99F973C2400000000000000000000000000000000
000000000000DEFEFEFEFEF1C6C6C6C6C6C6C6C6AA34000000000000
00000000000043724872A3E3FEE1FEFEFEFAE5FEFE8C000000000000
000000000000000000000011420E4343433B15ECFE6A000000000000
00000000000000000000000000000000000053FDD112000000000000
000000000000000000000000000000000016E9FF5300000000000000
000000000000000000000000000000000081FEEE2C00000000000000
000000000000000000000000000000003BF9FE3E0000000000000000
0000000000000000000000000000000085FEBB050000000000000000
00000000000000000000000000000009CDF83A000000000000000000
0000000000000000000000000000007EFEB600000000000000000000
00000000000000000000000000004BFBF03900000000000000000000
0000000000000000000000000013DDFEA60000000000000000000000
00000000000000000000000003CBFEDB230000000000000000000000
00000000000000000000000026FEFE4D000000000000000000000000
00000000000000000000001FE0FE7301000000000000000000000000
000000000000000000000085FEFE3400000000000000000000000000
000000000000000000003DF2FEFE3400000000000000000000000000
0000000000000000000079FEFEDB2800000000000000000000000000
0000000000000000000079FECF120000000000000000000000000000
00000000000000000000000000000000000000000000000000000000
2807

(I've added the line breaks here to be able to see the '7' digit.)

Now my question is, where this format is described? Or put differently where is defined that the first 36 bytes are some sort of header and the last 8 bytes have some label correspondence?

How would I go about constructing my own data? Neither Blob Tutorial nor Layers Definition give much away about required formats. My intention is not to use image data, but time series

Thanks!

4

2 回答 2

15

我意识到协议缓冲区必须在这里发挥作用。因此，我尝试针对caffe.proto中定义的某些类型对其进行反序列化。

基准似乎是完美的契合：

{Caffe.Datum}
    Channels: 1
    Data: {byte[784]}
    Encoded: false
    FloatData: Count = 0
    Height: 28
    Label: 7
    Width: 28

所以答案很简单：它是根据caffe.proto定义的“基准”类型实例的序列化表示

顺便提一句。由于英语不是我的母语，我必须首先意识到“基准”是“数据”的单数形式

在使用您自己的数据时，其结构如下：

数据的常规 blob 维度是数字 N x 通道 K x 高度 H x 宽度 W。 Blob 内存在布局中以行为主，因此最后/最右边的维度变化最快。例如，索引 (n, k, h, w) 处的值物理上位于索引 ((n * K + k) * H + h) * W + w。

请参阅Blob、层和网络：Caffe 模型的剖析以供参考

于 2015-04-10T03:40:49.967 回答
2

我可以试着回答你的第二个问题。由于 Caffe 只接受一系列选定格式的数据，如 lmdb、hdf5 等，因此最好将您的数据转换（或生成 - 如果是合成数据）这些格式。以下链接可以帮助您。如果你在 Python 中遇到问题import hdf5，那么你可以参考这个页面。

在 Python 中创建 LMDB 文件

用 Python 编写 HDF5 文件

HDF5 更多示例

于 2015-06-12T00:22:35.943 回答