o ckF[<@sddlZddlZddlZddlZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl m Z ddlmZddlmZ Gdd d eZdd d ZGd ddeZdS)N)configstorage_uri_for_key)AWSAuthConnection)ResumableDownloadException)ResumableTransferDisposition)KeyFile)Keyc@s eZdZdZddZddZdS)ByteTranslatingCallbackHandlerz Proxy class that translates progress callbacks made by boto.s3.Key.get_file(), taking into account that we're resuming a download. cCs||_||_dSN proxied_cbdownload_start_point)selfr r rD/usr/lib/python3/dist-packages/boto/s3/resumable_download_handler.py__init__?s z'ByteTranslatingCallbackHandler.__init__cCs||j||dSr r )rtotal_bytes_uploaded total_sizerrrcallCs z#ByteTranslatingCallbackHandler.callN)__name__ __module__ __qualname____doc__rrrrrrr 9s r FcCsPt|tr |s |jS|s|}|dtj|}|s&||tj|S)zH Returns size of file, optionally leaving fp positioned at EOF. r) isinstancergetkeysizetellseekosSEEK_ENDSEEK_SET)fpposition_to_eofcur_pos cur_file_sizerrrget_cur_file_sizeHs r%c@s^eZdZdZdZejeej ej fZ dddZ ddZ dd Zd d Zd d Z  dddZdS)ResumableDownloadHandlerz* Handler for resumable downloads. NcCs(||_||_d|_|r|d|_dS)a Constructor. Instantiate once for each downloaded file. :type tracker_file_name: string :param tracker_file_name: optional file name to save tracking info about this download. If supplied and the current process fails the download, it can be retried in a new process. If called with an existing file containing an unexpired timestamp, we'll resume the transfer for this file; else we'll start a new resumable download. :type num_retries: int :param num_retries: the number of times we'll re-try a resumable download making no progress. (Count resets every time we get progress, so download can span many more than this number of retries.) N)tracker_file_name num_retriesetag_value_for_current_download_load_tracker_file_etagr )rr(r)rrrrbs  z!ResumableDownloadHandler.__init__c Csd}zMzt|jd}|d|_t|j|jkr!td|jWn"tyD}z|j t j kr:td|j|j fWYd}~nd}~wwW|rN| dSdS|rW| ww)Nr zJCouldn't read etag in tracker file (%s). Restarting download from scratch.zJCouldn't read URI tracker file (%s): %s. Restarting download from scratch.) openr(readlinerstripr*len MIN_ETAG_LENprintIOErrorerrnoENOENTstrerrorclose)rferrrr+~s,     z0ResumableDownloadHandler._load_tracker_file_etagc Cs|jd|_|js dSd}z4zt|jd}|d|jWnty7}z td|j|jft j d}~wwW|rA| dSdS|rJ| ww)N"'wz%s zCouldn't write tracker file (%s): %s. This can happenif you're using an incorrectly configured download tool (e.g., gsutil configured to save tracker files to an unwritable directory)) etagstripr*r(r.writer4rr7rABORTr8)rkeyr9r:rrr_save_tracker_infos,    z+ResumableDownloadHandler._save_tracker_infocCs,|jrtj|jrt|jdSdSdSr )r(rpathexistsunlink)rrrr_remove_tracker_files  z-ResumableDownloadHandler._remove_tracker_filec Cs<t|dd} | ra|jra|j|jdkra| |jkr+td|j| tt||jft j | |jkr=|j j j dkr;tddS|j j j dkrHtd|}d | |jdf|d <t|| j}| |_n|j j j dkrltd ||d |_|d t|tr|j||||||d |d n |j||||||d d|dS)zr Attempts a resumable download. Raises ResumableDownloadException if any problems occur. T)r"r;zp%s is larger (%d) than %s (%d). Deleting tracker file, so if you re-try this download it will start from scratchzDownload complete.NzResuming download.z bytes=%d-%dRangez Starting new resumable download.roverride_num_retries hash_algsrJ)r%r*r=r>rrnamestrrrr@bucket connectiondebugr3copyr rr rBtruncaterGSKeyget_fileflush) rrAr!headerscbnum_cbtorrent version_idrKr$rrr_attempt_resumable_downloadsJ        z4ResumableDownloadHandler._attempt_resumable_download Fc  Cs |jjj} |s i}|jdurtddd|_d} t|} z||||||||||| dkr6t dWdS|j y} z<| dkrMt d | t | t rw| jtjkrwt |trk|j||||||d|d n |j||||||dd WYd} ~ nId} ~ wty} z9| jtjkr| dkrt d | j| jtjkr| dkrt d | j|| dkrt d| jWYd} ~ nd} ~ wwt|| krd} n| d7} | |jkrtdtjz|Wn tjyYnwd| } | dkrt d| | ft| q)a Retrieves a file from a Key :type key: :class:`boto.s3.key.Key` or subclass :param key: The Key object from which upload is to be downloaded :type fp: file :param fp: File pointer into which data should be downloaded :type headers: string :param: headers to send when retrieving the files :type cb: function :param cb: (optional) a callback function that will be called to report progress on the download. The callback should accept two integer parameters, the first representing the number of bytes that have been successfully transmitted from the storage service and the second representing the total number of bytes that need to be transmitted. :type num_cb: int :param num_cb: (optional) If a callback is specified with the cb parameter this parameter determines the granularity of the callback by defining the maximum number of times the callback will be called during the file transfer. :type torrent: bool :param torrent: Flag for whether to get a torrent for the file :type version_id: string :param version_id: The version ID (optional) :type hash_algs: dictionary :param hash_algs: (optional) Dictionary of hash algorithms and corresponding hashing class that implements update() and digest(). Defaults to {'md5': hashlib/md5.md5}. Raises ResumableDownloadException if a problem occurs during the transfer. NBotor)rTrGzResumable download complete.zCaught exception (%s)rIrLz4Caught non-retryable ResumableDownloadException (%s)zXCaught non-retryable ResumableDownloadException (%s); aborting and removing tracker filez3Caught ResumableDownloadException (%s) - will retryzeToo many resumable download attempts failed without progress. You might try this download again laterzWGot retryable failure (%d progress-less in a row). Sleeping %d seconds before re-trying)rOrPrQr)rgetintr%r\rFr3RETRYABLE_EXCEPTIONS__repr__rr4r5EPIPErTrUr dispositionrABORT_CUR_PROCESSmessager@r8httplibIncompleteReadtimesleep)rrAr!rWrXrYrZr[rKrQprogress_less_iterationshad_file_bytes_before_attemptr:sleep_time_secsrrrrUs *      z!ResumableDownloadHandler.get_file)NN)Nr]FNN)rrrrr2rh HTTPExceptionr4socketerrorgaierrorrbrr+rBrFr\rUrrrrr&Xs  0r&)F)r5rhrrerprjbotorrboto.connectionrboto.exceptionrrboto.s3.keyfiler boto.gs.keyrrTobjectr r%r&rrrrs"