26

我在一个循环中插入 100 个文件。对于此测试,我已禁用退避并重试,因此如果插入失败并出现 403,我将忽略它并继续下一个文件。在 100 个文件中,我得到 63 403 个速率限制异常。

但是,在检查 Drive 时,在这 63 次失败中,有 3 次实际上成功了,即。该文件是在驱动器上创建的。如果我完成了通常的退避并重试,我最终会得到重复的插入。这证实了我在启用退避重试时看到的行为,即。从我的 100 个文件测试中,我一直看到 3-4 个重复插入。

看起来 API 端点服务器和 Drive 存储服务器之间存在异步连接,这会导致不确定的结果,尤其是在大容量写入时。

由于这意味着我不能依靠“403 速率限制”来限制我的插入,我需要知道什么是安全的插入速率,以免触发这些计时错误。

运行下面的代码,给出...

Summary...
File insert attempts (a)       = 100
rate limit errors (b)          = 31
expected number of files (a-b) = 69
Actual number of files         = 73 

代码...

package com.cnw.test.servlets;

import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

import javax.servlet.http.HttpServlet;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

import com.google.api.client.auth.oauth2.Credential;
import com.google.api.client.googleapis.json.GoogleJsonError;
import com.google.api.client.googleapis.json.GoogleJsonResponseException;
import com.google.api.client.http.javanet.NetHttpTransport;
import com.google.api.client.json.jackson.JacksonFactory;
import com.google.api.services.drive.Drive;
import com.google.api.services.drive.model.ChildList;
import com.google.api.services.drive.model.File;
import com.google.api.services.drive.model.File.Labels;
import com.google.api.services.drive.model.ParentReference;

import couk.cleverthinking.cnw.oauth.CredentialMediatorB;
import couk.cleverthinking.cnw.oauth.CredentialMediatorB.InvalidClientSecretsException;

@SuppressWarnings("serial")
    /**
     * 
     * AppEngine servlet to demonstrate that Drive IS performing an insert despite throwing a 403 rate limit exception.
     * 
     * All it does is create a folder, then loop to create x files. Any 403 rate limit exceptions are counted.
     * At the end, compare the expected number of file (attempted - 403) vs. the actual.
     * In a run of 100 files, I consistently see between 1 and 3 more files than expected, ie. despite throwing a 403 rate limit,
     * Drive *sometimes* creates the file anyway.
     * 
     * To run this, you will need to ...
     * 1) enter an APPNAME above
     * 2) enter a google user id above
     * 3) Have a valid stored credential for that user
     * 
     * (2) and (3) can be replaced by a manually constructed Credential 
     * 
     * Your test must generate rate limit errors, so if you have a very slow connection, you might need to run 2 or 3 in parallel. 
     * I run the test on a medium speed connection and I see 403 rate limits after 30 or so inserts.
     * Creating 100 files consistently exposes the problem.
     * 
     */
public class Hack extends HttpServlet {

    private final String APPNAME = "MyApp";  // ENTER YOUR APP NAME
    private final String GOOGLE_USER_ID_TO_FETCH_CREDENTIAL = "11222222222222222222222"; //ENTER YOUR GOOGLE USER ID
    @Override
    public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
        /*
         *  set up the counters
         */
        // I run this as a servlet, so I get the number of files from the request URL
        int numFiles = Integer.parseInt(request.getParameter("numfiles"));
        int fileCount = 0;
        int ratelimitCount = 0;

        /*
         * Load the Credential
         */
        CredentialMediatorB cmb = null;
        try {
            cmb = new CredentialMediatorB(request);
        } catch (InvalidClientSecretsException e) {
            e.printStackTrace();
        }
        // this fetches a stored credential, you might choose to construct one manually
        Credential credential = cmb.getStoredCredential(GOOGLE_USER_ID_TO_FETCH_CREDENTIAL);

        /*
         * Use the credential to create a drive service
         */
        Drive driveService = new Drive.Builder(new NetHttpTransport(), new JacksonFactory(), credential).setApplicationName(APPNAME).build();

        /* 
         * make a parent folder to make it easier to count the files and delete them after the test
         */
        File folderParent = new File();
        folderParent.setTitle("403parentfolder-" + numFiles);
        folderParent.setMimeType("application/vnd.google-apps.folder");
        folderParent.setParents(Arrays.asList(new ParentReference().setId("root")));
        folderParent.setLabels(new Labels().setHidden(false));
        driveService.files().list().execute();
        folderParent = driveService.files().insert(folderParent).execute();
        System.out.println("folder made with id = " + folderParent.getId());

        /*
         * store the parent folder id in a parent array for use by each child file
         */
        List<ParentReference> parents = new ArrayList<ParentReference>();
        parents.add(new ParentReference().setId(folderParent.getId()));

        /*
         * loop for each file
         */
        for (fileCount = 0; fileCount < numFiles; fileCount++) {
            /*
             * make a File object for the insert
             */
            File file = new File();
            file.setTitle("testfile-" + (fileCount+1));
            file.setParents(parents);
            file.setDescription("description");
            file.setMimeType("text/html");

            try {
                System.out.println("making file "+fileCount + " of "+numFiles);
                // call the drive service insert execute method 
                driveService.files().insert(file).setConvert(false).execute();
            } catch (GoogleJsonResponseException e) {
                GoogleJsonError error = e.getDetails();
                // look for rate errors and count them. Normally one would expo-backoff here, but this is to demonstrate that despite
                // the 403, the file DID get created
                if (error.getCode() == 403 && error.getMessage().toLowerCase().contains("rate limit")) {
                    System.out.println("rate limit exception on file " + fileCount + " of "+numFiles);
                    // increment a count of rate limit errors
                    ratelimitCount++;
                } else {
                    // just in case there is a different exception thrown
                    System.out.println("[DbSA465] Error message: " + error.getCode() + " " + error.getMessage());
                }
            }
        }

        /* 
         * all done. get the children of the folder to see how many files were actually created
         */
        ChildList children = driveService.children().list(folderParent.getId()).execute();

        /*
         * and the winner is ...
         */
        System.out.println("\nSummary...");
        System.out.println("File insert attempts (a)       = " + numFiles);
        System.out.println("rate limit errors (b)          = " + ratelimitCount);
        System.out.println("expected number of files (a-b) = " + (numFiles - ratelimitCount));
        System.out.println("Actual number of files         = " + children.getItems().size() + " NB. There is a limit of 100 children in a single page, so if you're expecting more than 100, need to follow nextPageToken");
    }
}
4

1 回答 1

0

我假设您正在尝试进行并行下载...

这可能不是您正在寻找的答案,但这是我在与 google drive api 交互时所经历的。我使用 C#,所以它有点不同,但也许它会有所帮助。

我必须设置特定数量的线程才能一次运行。如果我让我的程序一次将所有 100 个条目作为单独的线程运行,我也会遇到速率限制错误。

我一点也不了解,但是在我的 C# 程序中,我运行了 3 个线程(用户可定义,默认为 3)

opts = new ParallelOptions { MaxDegreeOfParallelism = 3 };
var checkforfinished = 
Parallel.ForEach(lstBackupUsers.Items.Cast<ListViewItem>(), opts, name => {
{ // my logic code here }

我进行了快速搜索,发现 Java 8(不确定您是否正在使用)支持 Parallel().forEach(),也许这会对您有所帮助。我为此找到的资源位于:http ://radar.oreilly.com/2015/02/java-8-streams-api-and-parallelism.html

希望这会有所帮助,轮流尝试帮助其他人,因为人们帮助了我!

于 2016-11-21T20:07:29.113 回答