o `&@sddlmZddlmZeddlmZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZddlmZddlmZddlmZddlTdd lmZd d lmZd d lmZd Ze jdvrtddlmZede jnddlZGddde jZ GdddeZddZ!dS))division)standard_library)rangeN)config)log)progress)*) FileChunkIO) BotoBackend)get_connectionz2.1.1)darwinlinux2)dummyzAMultiprocessing is not supported on %s, will use threads instead.cs(eZdZdZfddZddZZS)ConsumerThreadz A background thread that collects all written bytes from all the pool workers, and reports it to the progress module. Wakes up every second to check for termination cs0tt|d|_d|_i|_||_||_dS)NTF)superr__init__daemonfinishrqueuetotal)selfrr __class__@/usr/lib/python3/dist-packages/duplicity/backends/_boto_multi.pyr>s  zConsumerThread.__init__c Csd}|js@z|j|d}|d|j|d<d}Wn!tjy:}ztt|j|jd}WYd}~nd}~ww|jrdSdS)NTr rF) rrgetrEmptyreport_transfersumvaluesr)rwaitargserrrrunFs zConsumerThread.run)__name__ __module__ __qualname____doc__rr$ __classcell__rrrrr8s rc@s2eZdZdZddZddZddZd d d ZdS) r a Backend for Amazon's Simple Storage System, (aka Amazon S3), though the use of the boto module, (http://code.google.com/p/boto/). To make use of this backend you must set aws_access_key_id and aws_secret_access_key in your ~/.boto or /etc/boto.cfg with your Amazon Web Services key id and secret respectively. Alternatively you can export the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY. cCs6t||zddl}Wntyw|dS)Nr)BotoSingleBackendrboto ImportError _setup_pool)r parsed_urlr+rrrr_s    zBotoBackend.__init__cCsftj}|s tjdd}t|ddr#td|j|j d|_td|t j |d|_dS)NF)logical_poolz8A process pool already exists. Destroying previous pool.z;Setting multipart boto backend process pool to %d processes) processes) rs3_multipart_max_procspsutil cpu_countgetattrrDebugr0 terminatejoinmultiprocessingPool)rnumber_of_procsrrrr-gs     zBotoBackend._setup_poolcCs,t|td|j|jdS)Nz Closing pool)r*_closerr6r0r7r8)rrrrr<vs   zBotoBackend._closeNc Csddl}tj}|tjkrtdtj|ftj}tj|}||kr&d}n ||}||r2|d7}t d||f|j j |j |tj d}d} tjr\t} | } t| |} | g} t|D]#} ||j}|j|j||j|j|| |tj| g }| |jt|qbt dt | | rz)| dj!tj"d| d#r| d$r| d=nt d|%Wntj&Wntj&yt d t | |%Ynw| st d tjrd | _'| (t | dkst |)|kr|*t+d |,S) Nrz+Minimum chunk size is %d, but %d specified.r zUploading %d bytes in %d chunks) encrypt_keyz2Waiting for the pool to finish processing %s tasks)timeoutz6Part upload not successful, aborting multipart upload.z^%s tasks did not finish by the specified timeout,aborting multipart upload and resetting pool.z.Done waiting for the pool to finish processingTz!Multipart upload failed. Aborted.)-r+rs3_multipart_chunk_sizes3_multipart_minimum_chunk_sizerWarnospathgetsizer6bucketinitiate_multipart_uploadkey s3_use_sserr9ManagerQueuerstartr storage_uri boto_uri_strschemer. bucket_nameid num_retriesappendr0 apply_asyncmultipart_upload_workerlenr!s3_multipart_max_timeoutready successfulr- TimeoutErrorrr8 get_all_parts cancel_uploadBackendExceptioncomplete_upload)rfilenamerGheadersr+ chunk_sizebyteschunksmprmanagerconsumertasksnrLparamsrrrupload|sr         zBotoBackend.upload)N)r%r&r'r(rr-r<rirrrrr Ss  r c s4fdd f dd|S)z Worker method for uploading a file chunk to S3 using multipart upload. Note that the file chunk is read into memory, so it's important to keep this number reasonably small. cs<tj}td|||fdur|gdSdS)Nz%s: Uploaded %s/%s bytes)r9current_processnamerr6put)uploadedr worker_name)offsetrrr_upload_callbacks z1multipart_upload_worker.._upload_callbackc stj}td|dfzt }|}|D]o}|jkrt ddT}t }z|j |dt ddddWnt jyc}z t|jWYd}~nd}~wwt }td j|dt dt||d Wdn1swYnq |d}d}~Wn9ty} z-t|rtd |d|df|dWYd} ~ Std |df| d} ~ wwtd |dfdS)Nz%s: Uploading chunk %dr r)rorai)cbnum_cbz>{name}: Uploaded chunk {chunk} at roughly {speed} bytes/second)rkchunkspeedz8%s: Upload of chunk %d failed. Retrying %d more times...z*%s: Upload of chunk %d failed. Aborting...z%s: Upload of chunk %d complete)r9rjrkrr6r lookuplist_multipart_uploadsrPr timeupload_part_from_filemaxsocketgaierrorrAstrerrorformatabsclose Exception traceback print_exc) rQrnconnrErcfdrKexendr#) _uploadrprOrar^ multipart_idror.rNrLrrrs\        z(multipart_upload_worker.._uploadr) rNr.rLrOrr^rorarQrr) rrprOrar^rror.rrNrLrrTs'rT)" __future__rfuturerinstall_aliasesbuiltinsrrBr3rr}sys threadingrzr duplicityrrrduplicity.errorsduplicity.filechunkior _boto_singler r*r BOTO_MIN_VERSIONplatformr9rr6ThreadrrTrrrrs6            r