关于我
 

xjpvictor's Blog
小老鼠,上灯台,两只耳朵竖起来

Dropbox api 上传大文件


各种折腾·dropboxlinuxscript

本文发表于超过180天以前,部分内容可能已经失效

dropbox的api前段时间大概是更新了,发现了一个新的endpoint,https://www.dropbox.com/developers/reference/api#chunked-upload 可以分段上传文件,这样就能突破以前的150MB的限制,上传大文件。以前的脚本在这里。修改一下,加上

API_CHUNKED_UPLOAD_URL='https://api-content.dropbox.com/1/chunked_upload'
API_COMMIT_CHUNKED_UPLOAD_URL='https://api-content.dropbox.com/1/commit_chunked_upload/dropbox'
let max=150000000
if [ $filesize -gt $max ];then
  let chunk=15000000
  CHUNKFILE=/dev/shm/vps-upload-chunkfile
  DD="dd if=$1 bs=1MB iflag=count_bytes of=$CHUNKFILE"
  $DD count=$chunk
  upload_id=`curl -s -H "Content-Length: $chunk" -T $CHUNKFILE "$API_CHUNKED_UPLOAD_URL?oauth_consumer_key=$APP_KEY&oauth_token=$OAUTH_ACCESS_TOKEN&oauth_signature_method=PLAINTEXT&oauth_signature=$APP_SECRET%26$OAUTH_ACCESS_TOKEN_SECRET&oauth_timestamp=$TIME_STAMP&oauth_nonce=$RANDOM" | grep -oP "\"upload_id\": \"[^\"]+\"" | awk -F'"' '{print $(NF-1)}'`
  if [ -n "$upload_id" ];then
    let offset=chunk
    while [ `echo $offset+$chunk|bc` -lt $filesize ];do
      $DD count=$chunk skip=`echo $offset/1000/1000|bc`
      curl -s -H "Content-Length: $chunk" -T $CHUNKFILE "$API_CHUNKED_UPLOAD_URL?upload_id=$upload_id&offset=$offset&oauth_consumer_key=$APP_KEY&oauth_token=$OAUTH_ACCESS_TOKEN&oauth_signature_method=PLAINTEXT&oauth_signature=$APP_SECRET%26$OAUTH_ACCESS_TOKEN_SECRET&oauth_timestamp=$TIME_STAMP&oauth_nonce=$RANDOM" -o /dev/null
      let offset=offset+chunk
    done
    $DD skip=`echo $offset/1000/1000|bc`
    curl -s -H "Content-Length: `echo $filesize-$offset|bc`" -T $CHUNKFILE "$API_CHUNKED_UPLOAD_URL?upload_id=$upload_id&offset=$offset&oauth_consumer_key=$APP_KEY&oauth_token=$OAUTH_ACCESS_TOKEN&oauth_signature_method=PLAINTEXT&oauth_signature=$APP_SECRET%26$OAUTH_ACCESS_TOKEN_SECRET&oauth_timestamp=$TIME_STAMP&oauth_nonce=$RANDOM" -o /dev/null
    curl -s -X POST "$API_COMMIT_CHUNKED_UPLOAD_URL/$DIR/$filename?upload_id=$upload_id&oauth_consumer_key=$APP_KEY&oauth_token=$OAUTH_ACCESS_TOKEN&oauth_signature_method=PLAINTEXT&oauth_signature=$APP_SECRET%26$OAUTH_ACCESS_TOKEN_SECRET&oauth_timestamp=$TIME_STAMP&oauth_nonce=$RANDOM" -o /dev/null
    rm $CHUNKFILE
  else
    echo "Something goes wrong"
  fi
else
  curl -s -H "Content-Length: $filesize" -T "$1" "$API_FILES_PUT_URL/$DIR/$filename?oauth_consumer_key=$APP_KEY&oauth_token=$OAUTH_ACCESS_TOKEN&oauth_signature_method=PLAINTEXT&oauth_signature=$APP_SECRET%26$OAUTH_ACCESS_TOKEN_SECRET&oauth_timestamp=$TIME_STAMP&oauth_nonce=$RANDOM" -o /dev/null
fi

简单来说就是判断文件大小,大于150MB就分段上传,每次上传15MB。上传完第一部分以后dropbox会返回一个id,后面上传的时候要告诉dropbox上传的是这个id所属的文件,然后全部上传完以后再告诉dropbox已经上传完了。读取文件的时候是以1MB为读取的块的大小,临时文件放在/dev/shm里,如果内存有空间就是在内存里,不然就在swap里,上传完后会删除。试了一个250MB的,工作正常。

本文 "Dropbox api 上传大文件" 由 K. Huang 首先发表于 xjpvictor's Blog 并以 CC BY-NC 4.0 许可证发布 © 2012
转载注明引用来源 https://blog.xjpvictor.info/2012/09/dropbox-api-large-file/

赞助我

2条评论

  1. 怎么用? 有完整下载吗?

    回复

评论

你的邮箱地址不会被公开。必填项以 * 标出

无意义或不相关评论将被删除

取消 上传图片 或拖拽上传

* Some files will not be uploaded. Only png, gif, jpg are allowed and maximum 10 images, each file less than 10M.

你可以添加 10 个 png/gif/jpg 图片 (每个文件小于 10M)

允许使用以下html标签:<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

你可以上传文件,粘贴代码或长文至 Drop.it.r

更多相似文章