google-bigquery - BigQuery 表的扁平数据并将扁平数据复制到新的 BigQuery 表

Question

我是 BQ 的新手。我有一个表，其中一些列是重复记录的。我正在尝试平展表格，所以它会是一种关系，并将平展数据插入新的 BigQuery 表格。可能吗？我该怎么做？

Answer 1

以下是 BigQuery 标准 SQL

假设您有如下简单的表格

Row id  repeated_record  
--- --  ---------------
1   1   google   
        facebook     
        viant    
2   2   dell     
        hp     

您可以使用以下查询轻松模仿它

#standardSQL
WITH `table-with-repeated-record` AS (
  SELECT 1 AS id, ['google', 'facebook', 'viant'] AS repeated_record UNION ALL
  SELECT 2, ['dell', 'hp']
)
SELECT *
FROM `table-with-repeated-record`  

所以现在，让它变平 - 使用下面的查询

#standardSQL
WITH `table-with-repeated-record` AS (
  SELECT 1 AS id, ['google', 'facebook', 'viant'] AS repeated_record UNION ALL
  SELECT 2, ['dell', 'hp']
)
SELECT id, flatted_data
FROM `table-with-repeated-record`, 
  UNNEST(repeated_record) AS flatted_data   

结果如下

Row id  flatted_data     
--- --  ------------
1   1   google   
2   1   facebook     
3   1   viant    
4   2   dell     
5   2   hp    

下面是另一个例子

#standardSQL
WITH `table-with-repeated-record` AS (
  SELECT 1 AS id, [STRUCT<line INT64, name STRING>(1, 'google'), (2, 'facebook'), (3, 'viant')] AS repeated_record UNION ALL
  SELECT 2, [STRUCT<line INT64, name STRING>(5, 'dell'), (6, 'hp')]
)
SELECT *
FROM `table-with-repeated-record`  

模仿下表

Row id  repeated_record.line    repeated_record.name     
--- --  --------------------    --------------------
1   1   1                       google   
        2                       facebook     
        3                       viant    
2   2   5                       dell     
        6                       hp

将其展平的方法是：

#standardSQL
WITH `table-with-repeated-record` AS (
  SELECT 1 AS id, [STRUCT<line INT64, name STRING>(1, 'google'), (2, 'facebook'), (3, 'viant')] AS repeated_record UNION ALL
  SELECT 2, [STRUCT<line INT64, name STRING>(5, 'dell'), (6, 'hp')]
)
SELECT id, flatted_data.line, flatted_data.name
FROM `table-with-repeated-record`, 
  UNNEST(repeated_record) AS flatted_data   

最后是

Row id  line    name     
--- --  ----    ----
1   1   1       google   
2   1   2       facebook     
3   1   3       viant    
4   2   5       dell     
5   2   6       hp    

您知道如何在不指定 ['google'、'facebook'、'viant'] 等数据的情况下执行此操作吗？表的大小不是恒定的，它会不时发生变化，以及存储在表中的数据，我唯一确定的是列

您应该只在下面使用（没有用作示例的虚拟数据，并且您可以使用查询）

#standardSQL
SELECT id, flatted_data.line, flatted_data.name
FROM `yourProject.yourDataset.yourTable`, 
  UNNEST(repeated_record) AS flatted_data