o ckF[z@sddlZddlZddlZddlZddlZddlZddlZddlZddlm Z ddl m Z m Z ddl mZddlmZddlmZddlmZddlmZ Gd d d eZdS) N)md5)config UserAgent)AWSAuthConnection)InvalidUriError)ResumableTransferDisposition)ResumableUploadException)KeyFilec@seZdZdZejeejej fZ dZ d*ddZ ddZ dd Zd d Zd d ZddZddZd+ddZddZddZd,ddZddZddZdd Zd!d"Z $d-d%d&Zd.d(d)ZdS)/ResumableUploadHandleri )rNcCs.||_||_d|_d|_|r|d|_dS)a  Constructor. Instantiate once for each uploaded file. :type tracker_file_name: string :param tracker_file_name: optional file name to save tracker URI. If supplied and the current process fails the upload, it can be retried in a new process. If called with an existing file containing a valid tracker URI, we'll resume the upload from this URI; else we'll start a new resumable upload (and write the URI to this tracker file). :type num_retries: int :param num_retries: the number of times we'll re-try a resumable upload making no progress. (Count resets every time we get progress, so upload can span many more than this number of retries.) rN)tracker_file_name num_retriesserver_has_bytes tracker_uri_load_tracker_uri_from_fileupload_start_point)selfr r rB/usr/lib/python3/dist-packages/boto/gs/resumable_upload_handler.py__init__@s zResumableUploadHandler.__init__c Csd}z[zt|jd}|}||Wn<ty8}z|jtjkr.td|j|j fWYd}~nd}~wt yR}ztd||jfWYd}~nd}~wwW|r\| dSdS|re| ww)NrzHCouldn't read URI tracker file (%s): %s. Restarting upload from scratch.zXInvalid tracker URI (%s) found in URI tracker file (%s). Restarting upload from scratch.) openr readlinestrip_set_tracker_uriIOErrorerrnoENOENTprintstrerrorrclose)rfurierrrr[s.      z2ResumableUploadHandler._load_tracker_uri_from_filec Cs|jsdSd}z+tt|jtjtjBdd}||jWdWdS1s+wYWdStyJ}z t d|j|j ft j d}~ww)zM Saves URI to tracker file if one was passed to constructor. NiwzCouldn't write URI tracker file (%s): %s. This can happenif you're using an incorrectly configured upload tool (e.g., gsutil configured to save tracker files to an unwritable directory)) r osfdopenrO_WRONLYO_CREATwriterrrrrABORT)rr!r#rrr_save_tracker_uri_to_filess*  & z0ResumableUploadHandler._save_tracker_uri_to_filecCsTt|}|jdvs|jstd|||_|j|_d|j|jf|_ d|_ dS)z Called when we start a new resumable upload or get a new tracker URI for the upload. Saves URI and resets upload state. Raises InvalidUriError if URI is syntactically invalid. )httphttpszInvalid tracker URI (%s)z%s?%srN) urlparseschemelowernetlocrrtracker_uri_hostpathquerytracker_uri_pathr)rr" parse_resultrrrrs   z'ResumableUploadHandler._set_tracker_uricCs|jS)zX Returns upload tracker URI, or None if the upload has not yet started. )rrrrrget_tracker_urisz&ResumableUploadHandler.get_tracker_uricCs6d}|jr||jvr|j|j|t|dSdS)zt Returns the upload ID for the resumable upload, or None if the upload has not yet started. z ?upload_id=N)rindexlen)rdelimrrr get_upload_ids z$ResumableUploadHandler.get_upload_idcCs,|jrtj|jrt|jdSdSdSN)r r%r3existsunlinkr7rrr_remove_tracker_files  z+ResumableUploadHandler._remove_tracker_file*cCs d||fS)Nz bytes %s/%sr)r range_spec length_specrrr_build_content_range_headers z2ResumableUploadHandler._build_content_range_headercCs8i}|d||d<d|d<tj|d|j|j||jdS)a~ Queries server to find out state of given upload. Note that this method really just makes special case use of the fact that the upload server always returns the current start/end state whenever a PUT doesn't complete. Returns HTTP response from sending request. Raises ResumableUploadException if problem querying server. rA Content-Range0Content-LengthPUTr3 auth_pathheadershost)rDr make_requestr5r2)rconn file_length put_headersrrr_query_server_states z*ResumableUploadHandler._query_server_statec Cs|||}|jdkrd|dfS|jdkrtd|jtjd}|d}|rAtd|}|r@t| d}t| d }d }n|j S|sRtd t | tj|j dkr_td ||f||fS) a+ Queries server to find out what bytes it currently has. Returns (server_start, server_end), where the values are inclusive. For example, (0, 2) would mean that the server has bytes 0, 1, *and* 2. Raises ResumableUploadException if problem querying server. ri4z1Got non-308 response (%s) from server state queryFrangezbytes=(\d+)-(\d+)Tz6Couldn't parse upload server state query response (%s)zServer has: Range: %d - %d.)rQstatusrr START_OVER getheaderresearchlonggroupSERVER_HAS_NOTHINGstr getheadersdebugr) rrNrOrespgot_valid_responserBm server_start server_endrrr_query_server_poss<       z(ResumableUploadHandler._query_server_posc Cs|jj}|jdkr tdd|_i}|D]}|dkr"tdtj||||<qd||j j <| d|jj |j |}| }|jdvrLtd |jtj|jd kr_|jd kr_td |jtj|d }|sntd|tj|||dS)zn Starts a new resumable upload. Raises ResumableUploadException if any errors occur. rSzStarting new resumable upload.rzcontent-lengthz5Attempt to specify Content-Length header (disallowed)startPOST)zEGot status %d from attempt to start resumable upload. Will wait/retryrRz>Got status %d from attempt to start resumable upload. AbortingLocationzINo resumable tracker URI found in resumable initiation POST response (%s)N)bucket connectionr`rrr0rrr*providerresumable_upload_headerrMnamereadrVWAIT_BEFORE_RETRYrXrr+) rkeyrKrN post_headerskrabodyrrrr_start_new_resumable_uploadsR        z2ResumableUploadHandler._start_new_resumable_uploadc Cs ||j} |r&|dkr||j|d} n |dkrd} nd} d} ||||s+i} n|} |rL||kr<|d|} n |d||df|} | | d<t||| d<tj|d |jd | |jd }| d |j | D] }| || |qj| | d| r|| |jD] }|j|| q|t| 7}|r| d7} | | ks| dkr|||d} ||j} | s| |j|r|||||krtd ||ftj|}| |j|jd kr|d|d|dfS|jdvrtj}ntj}td|j|jf|)z Makes one attempt to upload file bytes, using an existing resumable upload connection. Returns (etag, generation, metageneration) from server upon success. Raises ResumableUploadException if any problems occur. rUrr rAz%d-%drSrErGrHNrIzxs z3ResumableUploadHandler.send_file..Botor Tc3s"|] }|j|fVqdSr=)rr{rr7rrrs  rSzResumable upload complete.zCaught exception (%s))&r isinstancer getkeysizerr%SEEK_ENDtellrmrnr`rdictrr rgetintrrrr generationmetagenerationdigest local_hashesr@rrRETRYABLE_EXCEPTIONS__repr__rrEPIPEr rrr)rrtrrKrrrCTrOr`rryrr#r)rrr send_file;sh%           z ResumableUploadHandler.send_file)NN)rArAr=)Tr)NrN)__name__ __module__ __qualname__rzhttplib HTTPExceptionrrrgaierrorrr]rrr+rr8r<r@rDrQrfrxrrrrrrrrrrr 6s0    4:^^ r )rrr%rrYrrr.hashlibrbotorrboto.connectionrboto.exceptionrrrboto.s3.keyfiler objectr rrrrs"