1

我正在处理一个遗留项目，其中 MP4 文件从一个 React 应用程序一个一个地上传到一个带有 EC2 实例axios的 NestJS API 。busboy然后，将文件上传到 S3 存储桶。

当 AWS S3 库尝试加载文件时，有时会引发明显的“随机”错误：

 warn
 module: videoservice
 method: saveTostream
 message: Error on Upload Success callback
 { [Error: ENOENT: no such file or directory, open 'file.mp4'] errno: -2, code: 'ENOENT', syscall: 'open', path: 'file.mp4' }

 (node:26886) UnhandledPromiseRejectionWarning: Error: ENOENT: no such file or directory, open 'file.mp4'
 (node:26886) UnhandledPromiseRejectionwarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catc h(). (rejection id: 20)

以下是片段：

API 上传端点

  @Post('upload/:id')
  @ApiOperation({ ... })
  @ApiResponse({ status: 201, type: Video })
  @ApiConsumes('multipart/form-data')
  @ApiImplicitFile({ name: 'file', required: true })
  @ApiCreatedResponse({ type: UploadResponseDto })
  async upload(
    @Decoded() decoded: any,
    @Param('id') videoId: number,
    @Query('fileSize') fileSize :number,
    @Req() req: Request,
    @Res() res: Response,
    @Body() uploadDto: UploadDto,
  ): Promise<any> {
    try {
      const busboy = new Busboy({
        headers: req.headers,
      });

      const data = new Map();
      const writtenFiles = [];
      let upload;

      busboy.on('field', (fieldname, val) => {
        data.set(fieldname, val);
      });

      busboy.on(
        'file',
        async (fieldname, file, filename, encoding, mimetype) => {
          try {
            data.set('filename', filename);
            data.set('mimetype', mimetype);
            data.set('encoding', encoding);

            const filepath = path.join(fieldname + path.extname(filename));
            upload = filepath;

            const writeStream = fs.createWriteStream(filepath);
            file.pipe(writeStream);

            const promise = new Promise((resolve, reject) => {
              file.on('end', () => {
                writeStream.end();
              });
              writeStream.on('finish', resolve);
              writeStream.on('error', reject);
            });
    
            writtenFiles.push(promise);
          } catch (err) {
            this.logger.error(log('busboy on file', err.message));
            return res.status(HttpStatus.INTERNAL_SERVER_ERROR).send({
              statusCode: HttpStatus.INTERNAL_SERVER_ERROR,
              message: err.message,
            });
          }
        },
      );

      busboy.on('error', err => {
        this.logger.warn(log('busboy on error', err.message));
      });

      busboy.on('finish', async () => {
        await Promise.all(writtenFiles);
        res.status(HttpStatus.CREATED).send('OK');

        await this.bucketService.saveStream( // Next snippet
          fs.createReadStream(upload),
          data.get('filename'),
          data.get('mimetype'),
          data.get('fileSize'),
          +data.get('programId'),
          +data.get('videoId')
        );
        return;
        fs.unlinkSync(upload);
      });

      req.pipe(busboy);
    } catch (err) {
      this.logger.error(log('catch', err.message));
      return res.status(HttpStatus.INTERNAL_SERVER_ERROR).send({
        statusCode: HttpStatus.INTERNAL_SERVER_ERROR,
        message: err.message,
      });
    }
  }

BucketServicesaveStream方法

public async saveStream(
    body: any
    filename: string,
    mimeType: string,
    fileSize: number,
    programId: number,
    videoId: number
  ): Promise<any> {
    try {
      const callback = async (err: any, data: any) => {
        if (err) {
          this.logger.warn(log('Error on Upload Success callback', err));
          throw err; // Here is where the error is raised
        }
      };

      return this.s3
        .upload(
          this.setParams(body, filename, mimeType, programId, videoId),
          (err, data) => callback(err, data),
        )
    } catch (err) {
      this.logger.error(log('Error on S3 Upload', err));
      throw err;
    }
  }

  private setParams(
    file: any,
    filename: string,
    mimeType: string,
    programId: number,
    videoId: number
  ): any {
    return {
      ...awsBucketConfig,
      Key: `${AWS_UPLOAD_DIR}${programId}/${videoId}/${Date.now()}${path.extname(
        filename,
      )}`,
      Body: file,
      ContentType: mimeType,
    };
  }

在某个时刻，我想这可能是因为 EC2 上的临时文件的名称总是相同的：file.mp4，当同时上传两个文件时，第一个在完成时删除文件（fs.unlinkSync(upload);在端点上）留下另一个没有它的正在进行的过程，所以当尝试上传它时，这个过程将找不到它。但这不是真的，因为我进行了测试，确保文件被一一上传。但是，我还通过更改控制器来确保名称始终不同：

const filepath = path.join(fieldname + path.extname(filename));

经过

const filepath = path.join(Math.floor(Math.random() * 10000) + path.extname(filename));

但错误仍在发生。正在发生的另一件奇怪的事情是，在我的机器上，我可以看到 ( ls) 文件同时上传，但在 EC2 中却没有。

事实：

  • EC2：t2.xlarge（约免费 4GB）

  • 操作系统：Ubuntu 18

  • 节点版本：10.21.0

  • 平均文件大小：2GB

  • 依赖项

     "archiver": "^3.1.1",
 "async-busboy": "^0.7.0",
 "aws-sdk": "^2.553.0",
 "axios": "^0.19.0",
 "body-parser": "^1.19.0",
 "busboy": "^0.3.1",
4

0 回答 0