o ~_j@sdZddlmZddlmZeddlmZddlmZddlm Z ddlm Z ddlm Z dd l Z dd l Z dd lmZdd lmZdd lmZdd lTdd lTddlmZd ad ZGdddeZddZddZddZddZd=ddZd=ddZd>ddZd=dd Z d!d"Z!d#d$Z"d%d&Z#d'd(Z$d)d*Z%Gd+d,d,e Z&Gd-d.d.e Z'Gd/d0d0e Z(Gd1d2d2e Z)Gd3d4d4e)Z*Gd5d6d6e)Z+Gd7d8d8e)Z,d9d:Z-d;d<Z.d S)?a Functions for producing signatures and deltas of directories Note that the main processes of this module have two parts. In the first, the signature or delta is constructed of a ROPath iterator. In the second, the ROPath iterator is put into tar block form. )division)standard_library)map)next)str)range)objectN) statistics)util)config)*)progressc@s eZdZdS)DiffDirExceptionN)__name__ __module__ __qualname__rr3/usr/lib/python3/dist-packages/duplicity/diffdir.pyr6srcCst|S)z) Alias for SigTarBlockIter below )SigTarBlockIter path_iterrrrDirSig:srcCst|tdS)a Return a tarblock full backup of items in path_iter A full backup is just a diff starting from nothing (it may be less elegant than using a standard tar file, but we can be sure that it will be easy to split up the tar and make the volumes the same sizes). )DirDeltaioStringIOrrrrDirFullAs rcCst|td|S)zN Return full backup like above, but also write signature to sig_outfp r)DirDelta_WriteSigrr)r sig_outfprrrDirFull_WriteSigMrcCs^tat|trtdd|D}nt|}t||}tj s'tj r+t j s+t |St|S)z Produce tarblock diff given dirsig_fileobj_list and pathiter dirsig_fileobj_list should either be a tar fileobj or a list of those, sorted so the most recent is last. cSg|]}t|qSrsigtar2path_iter.0xrrr ^zDirDelta..)r StatsDeltaProcessstats isinstancelistcombine_path_itersr#get_delta_iterr dry_runr trackerhas_collected_evidenceDummyBlockIterDeltaTarBlockIter)rdirsig_fileobj_listsig_iter delta_iterrrrrTs   rcCsJ|r|}n |r|}nJdttdt|t|fdS)zA Called by get_delta_iter, report error in getting delta rz)Both new and sig are None for some reasonError %s getting delta for %sN)get_relative_pathlogWarn_r uexcfsdecode)excnew_pathsig_pathsig_tar index_stringrrrdelta_iter_error_handleris   rCcs^|sJr ||j|}ttd||ffdd}|rV|rV|rV|jdkrVd|_|d}t |d}rLt ||| }| t ||nLd|_rutjjdkrmd td _nd d _|srtrtj| 7_nt |d}rt ||| }| ||||jj|j_|S) zj Return new delta_path which, when read, writes sig to sig_fileobj, if sigTarFile is not None zGetting delta of %s and %scsTt|_tjjdkrdtd_ndd_ t |dS)zG Callback activated when FileWithSignature read to end  signature// signature/N) lensizesys version_infomajorr r=joinnameaddfilerBytesIO) sig_stringindex sigTarFiletirrcallbacks  z get_delta_path..callback signaturediffrbsnapshotrD snapshot/rF snapshot/) get_tarinforS get_ropathr9Debugr;isregdifftypeopenFileWithReadCounterFileWithSignaturegetsize setfileobjlibrsync DeltaFilerJrKrLr r=rMrNrOr*SourceFileSize copy_attribsstatst_size)r?r@rT delta_pathrV old_sigfpnewfprrRrget_delta_pathxsH       rpcCs|jdkr'|r|r||ttdt|tjj t |dS|r0|r0| |ttdt|tjj t |dS)zI Look at delta path and log delta. Add stats if new_path is set rZzA %szM %sN) ra add_new_filer9Infor;r r=r8InfoCode diff_file_newescapeadd_changed_filediff_file_changed)rmr?r*rrrlog_delta_paths"         rxc cst||}|rtd|}nd}|D]\}}ttd|o#t|j|o*t|jf|r3|js|r| r|jdkrt tdt | tj jt| |r|t|j}tjjdkrndt|j|_n dd |j|_||t|t|jVq|r||krttt|||f}|rt||t|Vqtjd 7_qt |qt!|r|!dSdS) a1 Generate delta iter from new Path iter and sig Path iter. For each delta path of regular file type, path.difftype with be set to "snapshot", "diff". sig_iter will probably iterate ROPaths instead of Paths. If sig_fileobj is not None, will also write signatures to sig_fileobj. wNzComparing %s and %srzD %srDdeleted/sdeleted/rF)" collate2itersr make_tarfiler9r_r;uindexrStypeexistsrrr=r8rsdiff_file_deletedruROPathr]rJrKrLrNrMrOr*add_deleted_filerobustcheck_common_errorrCrprxErrorsadd_unchanged_fileclose) new_iterr5 sig_fileobjcollatedrTr?r@rUrmrrrr.sP              r.c cstd|}d|_|D]n}t|}dD]}||r-|t|d|dd}}nqtd|ftjj dkrFt t | d}nt | d}|dsW|dd}t |}||_|d ksf|d krw|||rw||||Vq |dS) zK Convert signature tar file object open for reading into path iter rr{)rEr[rzNzBad tarinfo name %srDrFrWrZ)r r}debugget_tarinfo_name startswithrHrrJrKrLtuplefsencodesplitrrainit_from_tarinfor`rf extractfiler) sigtarobjtftarinfotinameprefixrNrarSropathrrrr#s0       r#ccsd\}} |s-zt|}Wnty)|rd|fV|D]}d|fVqYdSw|j}|sTzt|}WntyP|rC|dfV|D]}|dfVqEYdSw|j}||kr`|dfVd}n||krn||fVd\}}nd|fVd}q)a4 Collate two iterators. The elements yielded by each iterator must be have an index variable, and this function returns pairs (elem1, elem2), (elem1, None), or (None, elem2) two elements in a pair will have the same index, and earlier indicies are yielded later than later indicies. NNr{Nr StopIterationrS)riter1riter2relem1relem2index1index2rrrr|sD             r|c#s|ddfddfdd}ddttttD}|r<||dd V|||s)dSdS) a Produce new iterator by combining the iterators in path_iter_list This new iter will iterate every path that is in path_iter_list in order of increasing index. If multiple iterators in path_iter_list yield paths with the same index, combine_path_iters will discard all paths but the one yielded by the last path_iter. This is used to combine signature iters, as the output will be a full up-to-date signature iter. Ncs2zt|}Wn tyYdSw|j||fS)zI Represent the next element as a triple, to help sorting Nr) iter_indexpath)path_iter_listrr get_tripleVs   z&combine_path_iters..get_triplecsr|dd}d}|t|kr7||}|d|kr-|d}|r)|||<|d7}n||=ndS|t|ksdSdS)zK Update all elements with path_index same as first element rr{N)rH) triple_list path_indexr old_triple new_triple)rrrrefresh_triple_list`s     z/combine_path_iters..refresh_triple_listcSsg|]}|r|qSrrr$rrrr'rr(z&combine_path_iters..r)reverserr,rrHsort)rrrr)rrrr-Gs    r-cCsVtat|trt|}nt|}t|||}tj s#tj r't j s't |St|S)a Like DirDelta but also write signature into sig_fileobj Like DirDelta, sig_infp_list can be a tar fileobj or a sorted list of those. A signature will only be written to newsig_outfp if it is different from (the combined) sig_infp_list. )r r)r*r+r,get_combined_path_iterr#r.r r/r r0r1r2r3)r sig_infp_list newsig_outfp sig_path_iterr6rrrrys   rcCstdd|DS)zI Return path iter combining signatures in list of open sig files cSr!rr"r$rrrr'r(z*get_combined_path_iter..)r-)rrrrrr rc@s*eZdZdZddZd ddZddZd S) rczH File-like object which also computes amount read as it is read cC ||_dS)zFileWithReadCounter initializerN)infile)selfrrrr__init__s zFileWithReadCounter.__init__rc Cszz|j|}Wn'ty/}zd}ttdt|t|jj fWYd}~nd}~wwt r;t j t |7_ |S)Nr7) rreadIOErrorr9r:r;r r<r=rNr*rirH)rlengthbufexrrrrs zFileWithReadCounter.readcCs |jSN)rrrrrrrs zFileWithReadCounter.closeNr)rrr__doc__rrrrrrrrcs   rcc@s.eZdZdZdZddZd ddZdd Zd S) rdzF File-like object which also computes signature as it is read icGs.|||_|_tt||_d|_||_dS)au FileTee initializer The object will act like infile, but whenever it is read it add infile's data to a SigGenerator object. When the file has been read to the end the callback will be called with the calculated signature, and any extra_args if given. filelen is used to calculate the block size of the signature. N)rrVrg SigGeneratorget_block_sizesig_genactivated_callback extra_args)rrrVfilelenrrrrrs  zFileWithSignature.__init__rcCs|j|}|j||Sr)rrrupdate)rrrrrrrs  zFileWithSignature.readcCsJ|js ||jr ||js d|_|j|jg|jR|jS)Nr{) rr blocksizerVrgetsigrrrrrrrrs   zFileWithSignature.closeNr)rrrrrrrrrrrrrds   rdc@eZdZdZddZdS)TarBlockz5 Contain information to add next file to tar cCs||_||_dS)z8 TarBlock initializer - just store data N)rSdata)rrSrrrrrs zTarBlock.__init__N)rrrrrrrrrr rc@sreZdZdZddZdddZddZd d Zd d Zd dZ ddZ ddZ ddZ ddZ ddZddZdS) TarBlockIterz A bit like an iterator, yield tar blocks given input iterator Unlike an iterator, however, control over the maximum size of a tarblock is available by passing an argument to next(). Also the get_footer() is available. cCs@||_d|_d|_d|_d|_d|_d|_d|_d|_d|_ dS)z* TarBlockIter initializer rFN) input_iteroffsetprocess_waitingprocess_next_vol_numberprevious_indexprevious_block remember_nextremember_valueremember_block queued_data)rrrrrrs zTarBlockIter.__init__rcCs\t||_|jdtjd}t|jtj\}}|dkr"dtj|}nd}t|d|||fS)z< Make tarblock out of tarinfo and file data replace)errorsencodingrrs%s%s%s) rHrItobufr fsencodingdivmodtarfile BLOCKSIZEr)rrSr file_dataheadersblocks remainder filler_datarrrtarinfo2tarblocks zTarBlockIter.tarinfo2tarblockcCs|jrJtdS)z? Turn next value of input_iter into a TarBlock NrXXX)rvalrrrprocesss zTarBlockIter.processcCs|jsJtdS)z Get more tarblocks If processing val above would produce more than one TarBlock, get the rest of them by calling process_continue. Nrrrrrprocess_continueds zTarBlockIter.process_continuedcCs|jdur |j}d|_|S|jr|}n|t|j}|j}|jt|j 7_|j |_ ||_ |j r>|j |_||_d|_ |S)z5 Return next block and update offset NF)rrrrrrrrrHrrSrrrrr)rresult block_numberrrr__next__s  zTarBlockIter.__next__cCsdS)Nirrrrr get_read_size+szTarBlockIter.get_read_sizecC |j|jfS)zM Return index of last tarblock, or None if no previous index )rrrrrrget_previous_index2 zTarBlockIter.get_previous_indexcCr)zW Next time next() is called, we will return data instead of processing N)r)rrrrrqueue_index_data8s zTarBlockIter.queue_index_datacCsd|_d|_d|_dS)zL When called, remember the index of the next block iterated TN)rrrrrrrremember_next_index>s z TarBlockIter.remember_next_indexcCr)zD Retrieve index remembered with remember_next_index )rrrrrr recall_indexFrzTarBlockIter.recall_indexcCs&t|jtj\}}d|_dtj|S)zA Return closing string for tarfile, reset offset rr)rrr RECORDSIZE)rrrrrr get_footerLszTarBlockIter.get_footercCs|Srrrrrr__iter__TszTarBlockIter.__iter__N)r)rrrrrrrrrrrrrrrrrrrrrs    rc@r)r2z0 TarBlockIter that does no file reading cCsd|}|j}|jr |js|||Str,tjd7_tj|7_t dtj|||S)z7 Get a fake tarblock from delta_ropath r{N) r]rSrfileobjrr* SourceFilesrirer9Progress)r delta_ropathrUrSrrrr\s   zDummyBlockIter.processNrrrrrrrrrr2Xrr2c@r)rzK TarBlockIter that yields blocks of a signature tar from path_iter cCs|}|r;t|dt|}|}|dd |j |_ t j jdkr3t|j |_ ||j ||Sdd |j |_ t j jdkrQt|j |_ ||j |S)z@ Return associated signature TarBlock from path rYrGrFrDr\)r]r`rgSigFilerbrrerrrMrSrNrJrKrLr r=r)rrrUsfpsigbufrrrrts    zSigTarBlockIter.processNrrrrrrprrc@s(eZdZdZddZddZddZdS) r3z TarBlockIter that yields parts of a deltatar file Unlike SigTarBlockIter, the argument to __init__ is a delta_path_iter, so the delta information has already been calculated. c Csdd}|}|j}|jr|js,|js||dn |jdks!J||d|||S|d}||\}}trCtj t |7_ |rf|jdkrP||dn|jdkr[||dnJd||||Sd |j|j f}|d |_ ||_ ||_ ||_d |_d |_||||S) z2 Get a tarblock from delta_ropath cSs,|jdkr |d|_dSd||jf|_dS)z(Add prefix to the name of a tarinfo file./z%s/%sN)rN)rrrrr add_prefixs z-DeltaTarBlockIter.process..add_prefixdeletedrZrYrXrzUnknown difftypezmultivol_%s/%sz/1r{r)r]rSrrrarrbget_data_blockr* RawDeltaSizerHrNprocess_prefix process_fpprocess_ropathrr) rrrrUrSfpr last_block full_namerrrrs8          zDeltaTarBlockIter.processcCs>|}||}t||kr|rtd|dfS|dfS)zH Return pair (next data block, boolean last data block) zError closing fileTF)rrrHrr)rr  read_sizerrrrrs  z DeltaTarBlockIter.get_data_blockcCs|jsJ|j}||j}}d|j|jf|_||j\}}t r,t j t |7_ |r>d|_d|_d|_d|_d|_n|jd7_| |||S)zA Return next volume in multivol diff or snapshot z%s/%dNr{) rr r]rSr rrNrr r*r rHr)rrrUrSrrrrrrs z#DeltaTarBlockIter.process_continuedN)rrrrrrrrrrrr3s / r3cCst|tr t|jd}nt|tdjfrt|d}n|}|D]}||jq||| r4Jt|tr?| dSdS)z< Write block_iter to filename, path, or file object wbrN) r+PathrbrNr __class__writerrrsetdata) block_iterout_objr blockrrrwrite_block_iters     rcCs(|dkrdSt|dd}t|tjS)z Return a reasonable block size to use on files of length file_len If the block size is too big, deltas will be bigger than is necessary. If the block size is too small, making deltas and patching can take a really long time. ii)intminr max_blocksize)file_lenfile_blocksizerrrrs rrr)/r __future__rfuturerinstall_aliasesbuiltinsrrrrrrrJ duplicityr r r duplicity.pathduplicity.lazyr r*r0 ExceptionrrrrrrCrprxr.r#r|r-rrrcrdrrr2rr3rrrrrrsT               6 8!,2% {Z