python - Streaming file transfer from Google Storage to Dropbox -

in short, i'm looking see if it's possible perform chunked transfer file in google storage bucket dropbox. in mind, seems should able download chunk memory (say, 100mb) , send dropbox via file-like object. able both steps separately, cannot figure out if gluing 2 possible. file sizes typically >2gb, > 5gb. recognize cannot md5 checking (at least @ file-level), etc. if in manner.

ideally (but not requirement) done in python, it's file-transfer service i'm creating part of django web application.

what have far:

for downloading (exception handling, etc. removed simplicity), use google-resumable-media library:

import google.auth import google.auth.transport.requests tr_requests google.resumable_media.requests import chunkeddownload  ro_scope = 'https://www.googleapis.com/auth/devstorage.read_only' credentials, _ = google.auth.default(scopes=(ro_scope,)) transport = tr_requests.authorizedsession(credentials)  bucket = '<bucket_name>' blob_name = '<file_path>' url = 'https://storage.googleapis.com/%s/%s' % (bucket, blob_name)  chunk_size = 100*1024*1024  open('foo.gz', 'wb') f:     download = chunkeddownload(url, chunk_size, f)     while not download.finished:         response = download.consume_next_chunk(transport) 

for upload dropbox using python sdk (similar uploading file):

import os import dropbox chunk_size = 100*1024*1024 local_file = 'foo.gz' file_size = os.path.getsize(local_file) token='<token>' client = dropbox.dropbox.dropbox(token) open(local_file) file_obj:         session_start_result = client.files_upload_session_start(file_obj.read(chunk_size))         cursor=dropbox.files.uploadsessioncursor(session_start_result.session_id, offset=file_obj.tell())         commit=dropbox.files.commitinfo(path='/test_upload.gz')         while file_obj.tell() < file_size:             if (file_size-file_obj.tell()) <= chunk_size:                     client.files_upload_session_finish(file_obj.read(chunk_size), cursor, commit)             else:                     client.files_upload_session_append_v2(file_obj.read(chunk_size), cursor)                     cursor.offset = file_obj.tell() 

as referenced earlier, there way pull 100mb chunk google storage (into file-like object in memory), use file_obj.read(chunk_size) push chunk dropbox, , "empty"/"reset" file-like object? i've tried using io.bytesio (instead of "regular" file), , seemingly works until throws memoryerror exception (if watch ram usage, progressively consumes ram).

thanks!