1

我正在使用 Azure 数据工厂将本地数据传输到 Azure Data Lake 存储。复制数据后，我在上传的文件上运行 U-SQL 脚本以将其转换为新的 .csv 文件。如果从 Visual Studio 或直接从 Data Lake Analytics 运行，我的 U-SQL 作业运行良好。

但是如果在 Azure 数据工厂中添加和活动。复制数据后，此 U-SQL 脚本活动立即失败。我尝试了很多方法，但无法解决问题。它给了我以下错误。

在此处输入图像描述

我的 U-SQL 活动的 JSON 定义是 

{
    "name": "Transform Data",
    "description": "This will transform work space data.",
    "type": "DataLakeAnalyticsU-SQL",
    "dependsOn": [
        {
            "activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
            "dependencyConditions": [
                "Completed"
            ]
        }
    ],
    "policy": {
        "timeout": "7.00:00:00",
        "retry": 0,
        "retryIntervalInSeconds": 30,
        "secureOutput": false
    },
    "typeProperties": {
        "scriptPath": "Scripts/Script.usql",
        "scriptLinkedService": {
            "referenceName": "Destination_DataLakeStore_lc0",
            "type": "LinkedServiceReference"
        }
    },
    "linkedServiceName": {
        "referenceName": "AzureDataLakeAnalyticsForDF",
        "type": "LinkedServiceReference"
    }
}

整个管道的 JSON 是

{
    "name": "CopyPipeline_d26",
    "properties": {
        "activities": [
            {
                "name": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
                "type": "Copy",
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false
                },
                "typeProperties": {
                    "source": {
                        "type": "FileSystemSource",
                        "recursive": false
                    },
                    "sink": {
                        "type": "AzureDataLakeStoreSink",
                        "copyBehavior": "MergeFiles"
                    },
                    "enableStaging": false,
                    "cloudDataMovementUnits": 0,
                    "enableSkipIncompatibleRow": true
                },
                "inputs": [
                    {
                        "referenceName": "workplace_groups_info_2018_03_19_09_32_csv_i_lc0",
                        "type": "DatasetReference"
                    }
                ],
                "outputs": [
                    {
                        "referenceName": "workplace_groups_info_2018_03_19_09_32_csv_o_lc0",
                        "type": "DatasetReference"
                    }
                ]
            },
            {
                "name": "Transform Data",
                "description": "This will transform work space data.",
                "type": "DataLakeAnalyticsU-SQL",
                "dependsOn": [
                    {
                        "activity": "Copy_workplace_groups_info_2018_03_19_09_32_csv",
                        "dependencyConditions": [
                            "Completed"
                        ]
                    }
                ],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false
                },
                "typeProperties": {
                    "scriptPath": "Scripts/Script.usql",
                    "scriptLinkedService": {
                        "referenceName": "Destination_DataLakeStore_lc0",
                        "type": "LinkedServiceReference"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "AzureDataLakeAnalyticsForDF",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "parameters": {
            "windowStart": {
                "type": "String"
            },
            "windowEnd": {
                "type": "String"
            }
        }
    }
}
4

1 回答 1

2

我通过使用 AppService 创建运行时解决了这个问题。我按照以下步骤操作。

  1. 我在 Active Directory 中创建了一个 WebApp。
  2. 也将 Azure Data Lake 权限分配给该 WebApp。
  3. 在该应用程序中创建一个公钥并记下它。它永远不会再出现。
  4. 记下该 WebApp 的应用程序 ID。
  5. 打开 Azure Data Lake Analytics 并将参与者角色分配给 Active Directory 中创建的 WebApp。
  6. 在创建运行时时，使用应用程序 ID 作为服务主体 ID和公钥作为WebApp 的服务主体密钥。

它工作正常。:)

于 2018-04-02T19:42:16.513 回答