o baQ6@sdZddlmZddlmZddlmZddlmZmZddlZ ddl Z ddl Z ddl Z ddl Z ddlmZddlmZdd lmZdadaGd d d e jZGd d d eZddZGddde jZdS)a Functions to compute progress of compress & upload files The heuristics try to infer the ratio between the amount of data collected by the deltas and the total size of the changing files. It also infers the compression and encryption ration of the raw deltas before sending them to the backend. With the inferred ratios, the heuristics estimate the percentage of completion and the time left to transfer all the (yet unknown) amount of data to send. This is a forecast based on gathered evidence. )absolute_import)division)object)datetime timedeltaN)config)log)utilcs^eZdZdZeddZddZgdffdd Zd d Zd d Z d dZ fddZ Z S)Snapshotz A convenience class for storing snapshots in a space/timing efficient manner Stores up to 10 consecutive progress snapshots, one for each volume c Cst}tjdurHtjjdkrHztdtjjd}t|}| W|St yG}zt d t|t jjt}WYd}~|Sd}~ww|S)z9 De-serializes cached data it if present Nz %s/progressrzBWarning, cannot read stored progress info from previous backup: {})r rrestart start_volopenarchive_dir_pathnamepickleloadclose ExceptionrWarnformatr uexc WarningCode cannot_stat)snapshot progressfder4/usr/lib/python3/dist-packages/duplicity/progress.py unmarshall<s   zSnapshot.unmarshallcCs*tdtjjd}t|||dS)z, Serializes object to cache s %s/progresszwb+N)rrrrrdumpr)selfrrrrmarshallOs  zSnapshot.marshall cstt|||d|_dSNr)superr __init__last_vol)r"iterablemaxlen __class__rrr'Ws zSnapshot.__init__cCs:t|}|dkr dS|tdt|||jd|dS)Nrr )lenmaxminr()r"volumenitemsrrr get_snapshot[s&zSnapshot.get_snapshotcCs||||_dSN)appendr()r"r1 snapshot_datarrr push_snapshotas  zSnapshot.push_snapshotcCs|Sr4)popleftr"rrr pop_snapshoteszSnapshot.pop_snapshotcstt|d|_dSr%)r&r clearr(r9r+rrr;hs zSnapshot.clear) __name__ __module__ __qualname____doc__ staticmethodr r#r'r3r7r:r; __classcell__rrr+rr 6s r c@sLeZdZddZddZddZddZd d Zd d Zd dZ ddZ dS)ProgressTrackercCs|d|_d|_d|_d|_d|_d|_d|_d|_d|_d|_ d|_ d|_ t |_ d|_t|_d|_d|_d|_d|_dS)Nrr-F) total_statsnsteps start_timechange_mean_ratiochange_r_estimationprogress_estimationtime_estimationtotal_bytecountlast_total_bytecountlast_bytecountstall_last_time last_timer elapsed_sumspeedsys_collectionsdeque transfersis_fullcurrent_estimationprev_estimation prev_datar9rrrr'os&  zProgressTracker.__init__cCs,|jdur|j||j|jdSdS)z Snapshots the current progress status for each volume into the disk cache If backup is interrupted, next restart will deserialize the data and try start progress from the snapshot N)rWr7rHr#r"r1rrrsnapshot_progresss z!ProgressTracker.snapshot_progresscCs |jduS)z Returns true if the progress computation is on and duplicity has not yet started the first dry-run pass to collect some information N)rCr9rrrhas_collected_evidences z&ProgressTracker.has_collected_evidencec Csbtjr|s dSt}|jdur||_|jdur ||j}nt}||_|jdur.||_||jj t ddtj krRt d|j|j|j||jj |jddS|jd7_ ddlm}|jj|jj}|jj|jj}|dksz|jjdkr|dS|j}|jrt|jt||_nt|jt|jj}||j}|j|t|j7_|j|||j7_t t!|jt|j} t|t||jd | d |j|jd | |j|_ |jd kr t|t||jd | d |j|jd | |j|_ |jd kr|jt|t||_ t d t"|j#d |j#|jd |_ |j$|7_$d } |jdkrGd |j|j} t%| t|j$&|_|j|kr\||_ |&dkrv|j'(t|j|j)t|&|j|_)t*|j'dkr|j'+d |_|j'D] } d| d|j|_qt d|j|j|j||jj |jddS)zV Aproximative and evolving method of computing the progress of upload NgY@Tr r)diffdirgq= ףp?g?gQ?r-Gz?g333333?gffffff?F),rprogressrZrnowrErNrrMsecondsr/ progress_raterTransferProgressrHrIrJrPrD duplicityr]stats NewFileSizeChangedFileSizerC RawDeltaSizerTfloatrUrFrGmathsqrtfabsr0rVrOint total_secondsrSr5rKr.r8) r" current_timeelapsedr]changes total_changeslast_progress_estimation change_ratio change_delta change_sigma projectionxrrrlog_upload_progresss                   z#ProgressTracker.log_upload_progresscCsBt||jd}|jt|7_||_|dkrt|_dSdS)z Annotate the number of bytes that have been added/changed since last time this function was called. bytecount param will show the number of bytes since the start of the current volume and for the current volume rN)r/rLrJrnrrarM)r" bytecountchangingrrrannotate_written_bytess z&ProgressTracker.annotate_written_bytescCs||_||_dS)z Stores the collected statistics from a first-pass dry-run, to use this information later so as to estimate progress N)rCrT)r"rfrTrrr set_evidence(s zProgressTracker.set_evidencecCs0t|_|j||_tdt|jd|_dS)Nr-r^)r r rWr3rVr/r0rHrXrrrset_start_volume0s z ProgressTracker.set_start_volumecCst|jjS)zT Elapsed seconds since the first call to log_upload_progress method )rrarErbr9rrrtotal_elapsed_seconds5sz%ProgressTracker.total_elapsed_secondsN) r<r=r>r'rYrZrzr}r~rrrrrrrBms   rBcCs&tdurtdurt|dSdSdS)z Method to call tracker.annotate_written_bytes from outside the class, and to offer the "function(long, long)" signature which is handy to pass as callback N)progress_threadtrackerr})r{ totalbytesrrrreport_transfer<srcs(eZdZdZfddZddZZS)LogProgressThreadz_ Background thread that reports progress to the log, every --progress-rate seconds cs"tt||dd|_dS)NTF)r&rr' setDaemonfinishedr9r+rrr'Ms  zLogProgressThread.__init__cCsHtjstjrtr |js"tttj |jr dSdSdSdSdSr4) rdry_runr`rrZrrztimesleeprcr9rrrrunRs  zLogProgressThread.run)r<r=r>r?r'rrArrr+rrHs r)r? __future__rrbuiltinsrrr collectionsrQrkr threadingrrerrr rrrRr rBrThreadrrrrrs(     7P