1

我从https://domain.firebaseio.com/users/导出了部分数据

{
  "3": {
    "company": "",
    "d_year": "",
    "email": "mario.giambanco@domain.com",
    "facebook": "",
    "fullname": "Mario Test",
    "google": "",
    "igoto": "",
    "image": "",
    "notifications": {
      "-Jx6fpaJHvKPHc8CylPd": {
        "from": "System",
        "image": "/img/system_icon.jpg",
        "msg": "System:",
        "param": "3",
        "posteddate": 1440016723546,
        "type": "system"
      }
    },
    "school": "",
    "school_year": "",
    "tags": {
      "-JxWuEPs183UEwsI-XNb": {
        "title": "Anesthesia"
      },
      "-JxWuZ-ePcx0XqYRmzc6": {
        "title": "Bridges"
      }
    },
    "twitter": ""
  },
  "4": {
    "company": "",
    "d_year": "",
    "email": "mariogiambanco@domain.com",
    "fullname": "mario test",
    "igoto": "",
    "image": "img/a0.jpg",
    "notifications": {
      "-JxAQpWGzY-gOzej7Xis": {
        "from": "System",
        "image": "/img/system_icon.jpg",
        "msg": "System:",
        "param": "4",
        "posteddate": 1440079641420,
        "type": "system"
      }
    },
    "school": "",
    "school_year": ""
  }
}

执行：SELECT * FROM dfs。/Users/me/Desktop/users.json

有效（或者，至少我得到了结果） 在此处输入图像描述

但是如何将列映射为行中的值。从关系数据库世界来看，屏幕截图中的列标题是唯一的 ID (3, 4) - 这些应该是行的一部分，而不是列标题。使用 push({}) 时生成的唯一生成的密钥也是如此

目标当然是做一个 Select Where (select * from data where fullname="Mario Test") 例如

在使用 Drill 搜索 JSON 之前，我应该对它进行某种预处理吗？

4

2 回答 2

2

可能有另一种方法可以做到这一点，但我会说是的，您可能需要稍微转换数据以便使用 Drill 进行查询。

这看起来像是您想要使用 KVGEN 的情况。KVGEN 会为您提供 Chris Matta 所描述的那种列，但是 KVGEN 对列进行操作，在这种情况下，实际上并没有要使用的列：

0: jdbc:drill:zk=local> select t.* from dfs.`/Users/vince/data/stackoverflow/users.json` t;
+---+---+
| 3 | 4 |
+---+---+
| {"company":"","d_year":"","email":"mario.giambanco@domain.com","facebook":"","fullname":"Mario Test","google":"","igoto":"","image":"","notifications":{"-Jx6fpaJHvKPHc8CylPd":{"from":"System","image":"/img/system_icon.jpg","msg":"System:","param":"3","posteddate":1440016723546,"type":"system"}},"school":"","school_year":"","tags":{"-JxWuEPs183UEwsI-XNb":{"title":"Anesthesia"},"-JxWuZ-ePcx0XqYRmzc6":{"title":"Bridges"}},"twitter":""} | {"company":"","d_year":"","email":"mariogiambanco@domain.com","fullname":"mario test","igoto":"","image":"img/a0.jpg","notifications":{"-JxAQpWGzY-gOzej7Xis":{"from":"System","image":"/img/system_icon.jpg","msg":"System:","param":"4","posteddate":1440079641420,"type":"system"}},"school":"","school_year":""} |
+---+---+
1 row selected (0.133 seconds)

由于这些列是动态的并且位于 JSON 对象的“顶级”，因此您不能在此处使用 KVGEN。但是如果你只是稍微转换一下数据，你就可以使用 KVGEN。我使用这个最优秀的工具 jq 的调用将数据转换成 KVGEN 可以使用的格式：

$ jq '.| { "user": . }' < users.json > users_kv.json

这将获取输入，并将 JSON 对象包装在另一个映射中，这将为我们提供我们需要执行以下操作的“静态”列：

0: jdbc:drill:zk=local> select kvgen(t.`user`) from dfs.`/Users/vince/data/stackoverflow/users_kv.json` t;
+--------+
| EXPR$0 |
+--------+
| [{"key":"3","value":{"company":"","d_year":"","email":"mario.giambanco@domain.com","facebook":"","fullname":"Mario Test","google":"","igoto":"","image":"","notifications":{"-Jx6fpaJHvKPHc8CylPd":{"from":"System","image":"/img/system_icon.jpg","msg":"System:","param":"3","posteddate":1440016723546,"type":"system"},"-JxAQpWGzY-gOzej7Xis":{}},"school":"","school_year":"","tags":{"-JxWuEPs183UEwsI-XNb":{"title":"Anesthesia"},"-JxWuZ-ePcx0XqYRmzc6":{"title":"Bridges"}},"twitter":""}},{"key":"4","value":{"company":"","d_year":"","email":"mariogiambanco@domain.com","fullname":"mario test","igoto":"","image":"img/a0.jpg","notifications":{"-Jx6fpaJHvKPHc8CylPd":{},"-JxAQpWGzY-gOzej7Xis":{"from":"System","image":"/img/system_icon.jpg","msg":"System:","param":"4","posteddate":1440079641420,"type":"system"}},"school":"","school_year":"","tags":{"-JxWuEPs183UEwsI-XNb":{},"-JxWuZ-ePcx0XqYRmzc6":{}}}}] |
+--------+
1 row selected (1.774 seconds)

由于我在列中有一个列表，因此仍然不能以您想要的方式进行查询。所以使用扁平化：

0: jdbc:drill:zk=local> select flatten(kvgen(t.`user`)) as `user` from dfs.`/Users/vince/data/stackoverflow/users_kv.json` t;
+------+
| user |
+------+
| {"key":"3","value":{"company":"","d_year":"","email":"mario.giambanco@domain.com","facebook":"","fullname":"Mario Test","google":"","igoto":"","image":"","notifications":{"-Jx6fpaJHvKPHc8CylPd":{"from":"System","image":"/img/system_icon.jpg","msg":"System:","param":"3","posteddate":1440016723546,"type":"system"},"-JxAQpWGzY-gOzej7Xis":{}},"school":"","school_year":"","tags":{"-JxWuEPs183UEwsI-XNb":{"title":"Anesthesia"},"-JxWuZ-ePcx0XqYRmzc6":{"title":"Bridges"}},"twitter":""}} |
| {"key":"4","value":{"company":"","d_year":"","email":"mariogiambanco@domain.com","fullname":"mario test","igoto":"","image":"img/a0.jpg","notifications":{"-Jx6fpaJHvKPHc8CylPd":{},"-JxAQpWGzY-gOzej7Xis":{"from":"System","image":"/img/system_icon.jpg","msg":"System:","param":"4","posteddate":1440079641420,"type":"system"}},"school":"","school_year":"","tags":{"-JxWuEPs183UEwsI-XNb":{},"-JxWuZ-ePcx0XqYRmzc6":{}}}} |
+------+
2 rows selected (0.257 seconds)

两排 - 好多了。现在你已经准备好做你想做的事了（注意子查询以及保留字 user 和 value 周围的反引号：

    0: jdbc:drill:zk=local> select u.`user`.`key` as userid, u.`user`.`value`.fullname as fullname, u.`user`.`value`.email as email from (select flatten(kvgen(t.`user`)) as `user` from dfs.`/Users/vince/data/stackoverflow/users_kv.json` t) u where u.`user`.`value`.fullname = 'Mario Test';
+---------+-------------+-----------------------------+
| userid  |  fullname   |            email            |
+---------+-------------+-----------------------------+
| 3       | Mario Test  | mario.giambanco@domain.com  |
+---------+-------------+-----------------------------+
1 row selected (0.22 seconds)
于 2015-09-04T18:40:32.543 回答
1

如果实际上键“3”和“4”是 id，则键不应该真正持有值。为 Drill 格式化 JSON 格式的更好方法是为这些值使用实际的键（还要注意每个文件可以有多个记录，Drill 可以解析它们）：

{ "id": 3,
  "data": {
       ...
  }
}
{ "id": 4,
   "data": {
        ...
   }
}

这样你就可以做这样的查询：

> select t.`id`, t.`data`.`fullname` as `fullname` from `firebase.json` t;
+-----+-------------+
| id  |  fullname   |
+-----+-------------+
| 3   | Mario Test  |
| 4   | mario test  |
+-----+-------------+
2 rows selected (0.269 seconds)
于 2015-09-04T17:20:50.850 回答