o /a@s2dZddlmZddlmZddlmZddlmZddlmZddlZddlm Z dd lm Z dd lm Z dd lm Z dd lm Z dd lmZddlmZdd lm Z ddlmZejdkrdeefZnefZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZdS)z6Classes and functions on collections of backup volumes)str)zip)map)range)objectN)log) file_naming)path)util)dup_time)config)manifest)GPGError)c@s eZdZdS)CollectionsErrorN)__name__ __module__ __qualname__rr;/usr/lib/python3/dist-packages/duplicity/dup_collections.pyr1src@seZdZdZddZddZd(ddZd d Zd d Zd dZ ddZ ddZ ddZ d)ddZ ddZddZddZddZd d!Zd"d#Zd$d%Zd&d'ZdS)* BackupSetzE Backup set - the backup information produced by one session cCsL||_d|_i|_d|_d|_d|_d|_d|_d|_d|_ g|_ ||_ dS)zN Initialize new backup set, only backend is required at first FN) backendinfo_setvolume_name_dictremote_manifest_namelocal_manifest_pathtime start_timeend_timepartial encrypted files_changedaction)selfrr"rrr__init__9s zBackupSet.__init__cC|jS)z8 Assume complete if found manifest file )rr#rrr is_completeJzBackupSet.is_completeNcCs|st|}|r|jdks|jdksdS|js||n2|j|jkr&dS|j|jkr.dS|j|jks:|j|jkrz$BackupSet.delete..r;cSrBrrCrDrrrrGrHN) get_filenamesreverserdelete ExceptionrDebug_r"r r<r=rr+rrrr>r release_lockfile)r#rfnr@lfnr4rrrrKs0        zBackupSet.deletecCs@g}|jr ||j|t|jddttj |S)z0 For now just list files in set z[%s], ) rr>extendlistrvaluesjoinrr r1)r#filelistrrr__str__s  zBackupSet.__str__cCst|jp|jS)z@ Return time string suitable for log statements )r timetoprettyrrr&rrr get_timestrzBackupSet.get_timestrTcCs|js|jsttdtjj|jsJd|r|nd}|jr&|}|r;|jr;|r;||kr;ttdtjj |sM|jrC|}n ttdtjj | dS)zA Make sure remote manifest is equal to local one z6Fatal Error: No manifests found for most recent backupzif only one, should be remoteNzFatal Error: Remote manifest does not match local one. Either the remote backup set or the local archive directory has been corrupted.z;Fatal Error: Neither remote nor local manifest is readable.) rrr FatalErrorrN ErrorCode no_manifestsget_remote_manifestget_local_manifestmismatched_manifestsunreadable_manifests check_dirinfo)r# check_remoteremote_manifestlocal_manifestrrrcheck_manifestss(     zBackupSet.check_manifestscCs@|jsJ|j}ttd|jjt|ft |S)zG Return manifest object by reading local manifest file z!Processing local manifest %s (%s)) rget_datarInforNnamelenr Manifest from_string)r#manifest_bufferrrrr`s    zBackupSet.get_local_manifestc Cs|jsJz |j|j}Wn%ty3}zttdt|jt |fWYd}~dSd}~wwt tdt|jt |ft |S)zG Return manifest by reading remote manifest on backend z)Error processing remote manifest (%s): %sNz"Processing remote manifest %s (%s))rrrhrrErrorrNr r1uexcrirkr rlrm)r#rnmessagerrrr_s   zBackupSet.get_remote_manifestcCs|jr|S|S)zK Return manifest object, showing preference for local copy )rr`r_r&rrrr7szBackupSet.get_manifestcs^jsJtj}|fdd|D}jr-tj}|r-|js-| j|S)zJ Return sorted list of (remote) filenames of files in set csg|]}j|qSr)r)rExr&rrrG rHz+BackupSet.get_filenames..) rrTrkeyssortrrr+rr>)r#volume_num_listvolume_filenamesr4rr&rrIs    zBackupSet.get_filenamescCs |jr|jS|jr |jSJd)zH Return time if full backup, or end_time if incremental rz'Neither self.time nor self.end_time set)rrr&rrrget_time+s zBackupSet.get_timecCr%r6)r!r&rrrr85szBackupSet.get_files_changedcCstt|jS)z9 Return the number of volumes in the set )rkrTrrsr&rrr__len__8r[zBackupSet.__len__cCs@|j|jko|j|jko|j|jko|j|jkot|t|kS)zB Return whether this backup set is equal to other )r,rrrrk)r#otherrrr__eq__>s    zBackupSet.__eq__r6)T)rrr__doc__r$r'r5r-r:r/rKrXrZrgr`r_r7rIrwr8rxrzrrrrr5s( 1     rc@steZdZdZddZddZddZdd d Zd d Zd dZ ddZ ddZ dddZ ddZ ddZddZdS) BackupChainz BackupChain - a number of linked BackupSets A BackupChain always starts with a full backup set and continues with incremental ones. cCs"||_d|_g|_d\|_|_dS)zI Initialize new chain, only backend is required at first NNN)rfullset incset_listrr)r#rrrrr$PszBackupChain.__init__cCs:|jst|ts J||_|jsJ|j|j|_|_dS)z% Add full backup set N)r~ isinstancerrrr)r#r~rrrset_fullYs zBackupChain.set_fullcCs|j|jkr |j|n7|jr/|j|jdjkr/|j|jdjkr/ttd||jd<nttdt|jt|jfdS|j|_ttdt|jt|jf|js`JdS)zL Add incset to self. Return False if incset does not match z'Preferring Backupset over previous one!z;Ignoring incremental Backupset (start_time: %s; needed: %s)Fz;Added incremental Backupset (start_time: %s / end_time: %s)T) rrrr>rrirNr rY)r#incsetrrradd_incbs*         zBackupChain.add_incFcCsJtt|jdddD] }|j|q |jr!|s#|jdSdSdS)z< Delete all sets in chain, in reverse order rN)rrkrrKr~)r# keep_fullirrrrKzs  zBackupChain.deletecs fdd|jD}|jg|S)zI Return a list of sets in chain earlier or equal to time csg|] }|jkr|qSrrrEsrrrrGz0BackupChain.get_sets_at_time..rr~)r#r older_incsetsrrrget_sets_at_times zBackupChain.get_sets_at_timecCs|jr|jdS|jS)z0 Return last BackupSet in chain rrr&rrrget_lasts zBackupChain.get_lastcCr%)zF Return first BackupSet in chain (ie the full backup) )r~r&rrr get_firstr(zBackupChain.get_firstcCsdt|jt|jfS)zf Return a short one-line description of the chain, suitable for log messages. z [%s]-[%s])r rYrrr&rrr short_descs  zBackupChain.short_descc Csdg}|D])}|jrd}|j}nd}|j}|jrd}nd}|d||t|t||fq|S)z> Return summary, suitable for printing to log r)r*encnoencz %s%s %s %d %s) get_all_setsrrr r>r timetostringrk)r#prefixlrbtyperrrrr to_log_infos $zBackupChain.to_log_infoc Csd}dtdt|jtdt|jtdt|jdftd|f|tdtd td fg}|D]#}|j rGtd }|j }ntd }|j}| ||t|t|fq:| dd |S)zD Return string representation, for testing purposes z%20s %30s %15s-------------------------zChain start time: zChain end time: z#Number of contained backup sets: %drz%Total number of contained volumes: %dType of backup set:Time:z Num volumes:Full Incremental ) rNr rYrrrkrget_num_volumesrrr>rV)r# set_schemarrrrrrrrXs*    zBackupChain.__str__cCs"d}|D]}|t|7}q|S)zA Return the total number of volumes in the chain r)rrk)r#nrrrrrs zBackupChain.get_num_volumescCs|jr |jg|jS|jS)z9 Return list of all backup sets in chain )r~rr&rrrrszBackupChain.get_all_setsNF)r)rrrr{r$rrrKrrrrrrXrrrrrrr|Is      r|c@sXeZdZdZddZddZddZdd Zdd d Zdd dZ dddZ dddZ d S)SignatureChainz~ A number of linked SignatureSets Analog to BackupChain - start with a full-sig, and continue with new-sigs. cCs>|r |d|_|_nd||_|_d|_g|_d\|_|_dS)a Return new SignatureChain. local should be true iff the signature chain resides in config.archive_dir_path and false if the chain is in config.backend. @param local: True if sig chain in config.archive_dir_path @type local: Boolean @param location: Where the sig chain is located @type location: config.archive_dir_path or config.backend Nr})r<rfullsiginclistrr)r#locallocationrrrr$s zSignatureChain.__init__cCsL|jrtd}ntd}g}|jr||j||jd|d|fS)z> Local or Remote and List of files in the set rremotez%s: [%s]rR)r<rNrr>rSrrV)r#placerWrrrrXs   zSignatureChain.__str__cCs*|D]}t|tvrJd||fqdS)z? Check to make sure times are in whole seconds rzTime %s in %s wrong typeN)r, integer_types)r# time_listrrrr check_times s  zSignatureChain.check_timescCs|jrdSdS)zQ Return true if represents a signature chain in archive_dir_path TF)r<r&rrrislocalszSignatureChain.islocalNcCs|st|}|s dS|jr0|jdkrdS|j|jkrdS|j|||jg|j|_dS|jdkr7dS||_||j |j g|j |j |_|_dS)zM Add new sig filename to current chain. Return true if fits Nnew-sigrfull-sig) rr+rr,rrrr>rrr2rrrr5s&     zSignatureChain.add_filenamecs>jsJjrfddnjjfdd|DS)zt Return ordered list of signature fileobjs opened for reading, optionally at a certain time cstjj|f}|dS)z:Open filename in archive_dir_path, return filtered fileobjrb)r DupPathr<rj filtered_open)r3sig_dpr&rrfilename_to_fileobj>s z8SignatureChain.get_fileobjs..filename_to_fileobjcsg|]}|qSrrrD)rrrrGDz/SignatureChain.get_fileobjs..)rr<rget_fileobj_readrI)r#rr)rr#r get_fileobjs7s zSignatureChain.get_fileobjsFcCs|jr+tt|jdddD] }|j|j|q|s)|j|jdSdS|js0J|jdd}||sC||j|j|dS)z3 Remove all files in signature set rrN) r<rrkrr>rKrrrJ)r#rr inclist_copyrrrrKFs  zSignatureChain.deletecs>|jr|jg}ng}|j}rfdd|D}|||S)zP Return ordered list of filenames in set, up to a provided time cs g|] }t|jkr|qSr)rr+r)rErrrrrGcs z0SignatureChain.get_filenames..)rrrS)r#rrrrrrrIXs  zSignatureChain.get_filenamesr6r) rrrr{r$rXrrr5rrKrIrrrrrs  rc@seZdZdZddZddZddZd2d d Zd d Zd dZ ddZ ddZ d3ddZ ddZ ddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0d1ZdS)4CollectionsStatusz: Hold information about available chains and sets cCsL||_||_||_d|_d|_d|_d|_g|_g|_d|_ d|_ d|_ dS)z7 Make new object. Does not set values N) rr<r"matched_chain_pairall_backup_chainsother_backup_chainsall_sig_chainslocal_orphaned_sig_namesremote_orphaned_sig_namesorphaned_backup_setsincomplete_backup_sets values_set)r#rr<r"rrrr$ms zCollectionsStatus.__init__cCsd|jjjfd|jfg}tt|jD]}|d|f||j|d7}q|j r<|d||j dd7}|dt|j f|dt|j f|S) zP Return summary of the collection, suitable for printing to log z backend %szarchive-dir %szchain-no-sig %d zchain-completerzorphaned-sets-num %dzincomplete-sets-num %d) r __class__rr<rrkrr>rrrrr#rrrrrrs  zCollectionsStatus.to_log_infocCshtddtd|jjjftd|jr|jjndfg}|dtddt|j t|j t t|j D]!}|td |d t|j f|t |j ||d q6|j ro|dtd |t |j d n|td |j s||jr|tddt|j t|j f|tddt|jt|jf|tdn|tdd|S)z9 Return string summary of the collection zCollection Statusz-----------------zConnecting with backend: %szArchive dir: %sNonerz Found %d secondary backup chain.z!Found %d secondary backup chains.zSecondary chain %d of %d:rrz9Found primary backup chain with matching signature chain:z-No backup chains with active signatures foundz/Also found %d backup set not part of any chain,z0Also found %d backup sets not part of any chain,zand %d incomplete backup set.zand %d incomplete backup sets.zEThese may be deleted by running duplicity with the "cleanup" command.z,No orphaned or incomplete backup sets found.)rNrrrr<uc_namer>ngettextrkrrrrrrrVrrrrrXsR       zCollectionsStatus.__str__rc Csd|_|j}ttddt|t||jdkr"|j }ng}ttddt|t|g}|D]}t |}|rH|j rH| |q7|||\}|_|_||}||_t|t|jksjJd|d\}|_|jd |d \} |_||| ||||S) a Set values from archive_dir_path and backend. Returns self for convenience. If sig_chain_warning is set to None, do not warn about unnecessary sig chains. This is because there may naturally be some unecessary ones after a full backup. rz%d file exists on backendz%d files exist on backendr;z%d file exists in cachez%d files exist in cachez7get_sorted_chains() did something more than re-orderingTF)rW)rrrTrrMrrkr"r<r=rr+rr>get_backup_chainsrrget_sorted_chainsrget_signature_chainsrrset_matched_chain_pairwarn) r#sig_chain_warningbackend_filename_listr@partialsrAr4 backup_chainslocal_sig_chainsremote_sig_chainsrrr set_valuessP           zCollectionsStatus.set_valuescCs|o||}||_|dd|_d|_|rl|rl|d}tt|dddD]E}||j|jkr1n.t|dkr^||j|djkr^t t d|j |j d|j dd|_ nq&|jdurk|||f|_|jrz|j|jddSdS)a9 Set self.matched_chain_pair and self.other_sig/backup_chains The latest matched_chain_pair will be set. If there are both remote and local signature chains capable of matching the latest backup chain, use the local sig chain (it does not need to be downloaded). NrrzGWarning, discarding last backup set, because of missing signature file.)rrrrrrkrrrWarnrNrr>rremove)r# sig_chainsrlatest_backup_chainrrrrrs*  z(CollectionsStatus.set_matched_chain_paircCs|jsJ|jr#ttddt|jddttj |jtj j |j rAttddt|j ddttj |j tj j |j rS|rS|jsSttdtj j|jr`ttdtj j|jrttdd t|jddtt|jtj jd Sd S) zN Log various error messages if find incomplete/orphaned files z;Warning, found the following local orphaned signature file:z.cst|}D]}|||rttdt|fdSqttdt|ftj j }|||rA |dSttdt|dS)zO Try adding filename to existing sets, or make new one zFile %s is part of known setz4File %s is not part of a known set; creating new setz+Ignoring file (rejected by backup set) '%s'N) rr+r5rrMrNr r1rrr"r>)r3r4setnew_set)r#setsrr add_to_sets[s   z8CollectionsStatus.get_backup_chains..add_to_setscs|jdkr!tj}|||ttd|dS|jdks(JD]}| |rCttd| |fdSq*ttd| f|dS)zL Try adding set to existing chains, or make new one r)zFound backup chain %sr*z%Added set %s to pre-existing chain %szFound orphaned set %sN) r,r|rrr>rrMrNrrrZ)r new_chainchain)chains orphaned_setsr#rr add_to_chainsrs     z:CollectionsStatus.get_backup_chains..add_to_chains)rrMrNget_sorted_sets)r# filename_listrrFincomplete_setsrrr)rrr#rrrMs     z#CollectionsStatus.get_backup_chainscCsvgg}}|D]"}|s||q|jdkr!||j|fq||j|fq|jddddd|D|fS)zM Sort set list by end time, return (sorted list, incomplete) r)cSs|dS)Nrrrrrrrsz3CollectionsStatus.get_sorted_sets..keycSg|]}|dqSrrrEprrrrGrz5CollectionsStatus.get_sorted_sets..)r'r>r,rrrt)r#set_listtime_set_pairsrrrrrrs   z!CollectionsStatus.get_sorted_setsNc sfdd}fdd}gg}}|D])}t|}|r@|jdkr6|} | ||s0J|| q|jdkr@||qg} |jddd |D]} |D] } | | rZnqQ| | qM|| fS) z Find chains in archive_dir_path (if local is true) or backend Use filelist if given, otherwise regenerate. Return value is pair (list of chains, list of signature paths not in any chains). cs2durSrjdkrjSgSjS)Nr;)r"r<r=rrTrrWrr#rr get_filelists   z.get_filelistcsrtdjStdjS)z: Return new empty signature chain TF)rr<rr)rr#rrget_new_sigchains  z@CollectionsStatus.get_signature_chains..get_new_sigchainrrcSstt|jSr6)intrr+rrrrrrsz8CollectionsStatus.get_signature_chains..r)rr+r,r5r>rt) r#rrWrrrnew_sig_filenamesr3r4rorphaned_filenames sig_filenamerrrrrs.        z&CollectionsStatus.get_signature_chainscCsi}|D]}|j|vr||j|q|g||j<qt|}|g}|D]>}||}t|dkr=||dq)t|dksEJ|djrY||d||dq)||d||dq)|S)zL Return chains sorted by end_time. If tie, local goes last rrr)rr>rTrsrtrkr)r# chain_listendtime_chain_dictrsorted_end_timessorted_chain_listrrrrrs&    z#CollectionsStatus.get_sorted_chainscst|jstdfdd|jD}t|dkrtdt|dkr%|dSfdd|jD}|r5|dS|jdS) z Return backup chain covering specified time Tries to find the backup chain covering the given time. If there is none, return the earliest chain before, and failing that, the earliest chain. zNo backup chains foundc,g|]}|jkr|jkrnn|qSrrrrEcrrrrGz>CollectionsStatus.get_backup_chain_at_time..rzTwo chains cover the given timercg|] }|jkr|qSrrrrrrrGrr)rrrk)r#rcovering_chains old_chainsrrrget_backup_chain_at_times   z*CollectionsStatus.get_backup_chain_at_timecs|jstdfdd|jD}|r|dSfdd|jD}|r'|dS|jd}|jkrFttdt|jtjj t |j|S)z Return signature chain covering specified time Tries to find the signature chain covering the given time. If there is none, return the earliest chain before, and failing that, the earliest chain. zNo signature chains foundcrrrrrrrrG rzACollectionsStatus.get_signature_chain_at_time..rcr rrrrrrrGrrz]No signature chain for the requested time. Using oldest available chain, starting at time %s.) rrrrrrNr rYrno_sig_for_timer)r#rr r oldestrrrget_signature_chain_at_times"     z-CollectionsStatus.get_signature_chain_at_timecCsh|jsJg}g}|j|j}|D]}|jr||q||q||j7}||j7}||fS)z Return list of the names of extraneous duplicity files A duplicity file is considered extraneous if it is recognizable as a duplicity file, but isn't part of some complete backup set, or current signature chain. )rrrrrSrIrr)r#local_filenamesremote_filenamesext_containers set_or_chainrrrget_extraneous s    z CollectionsStatus.get_extraneouscCs$dd|D}|dd|DS)z@Return new list containing same elems of setlist, sorted by timecSsg|]}||fqSrrwrrrrrG7sz/CollectionsStatus.sort_sets..cSrrrrrrrrG9r)rt)r#setlistpairsrrr sort_sets5szCollectionsStatus.sort_setscCF|jsJg}|jD]}|j|kr |jr||jdur ||q |S)a Returns a list of backup chains older than the given time t All of the times will be associated with an intact chain. Furthermore, none of the times will be of a chain which a newer set may depend on. For instance, if set A is a full set older than t, and set B is an incremental based on A which is newer than t, then the time of set A will not be returned. r)rrrrr>r#tr rrrrget_chains_older_than;    z'CollectionsStatus.get_chains_older_thancCr)a Returns a list of signature chains older than the given time t All of the times will be associated with an intact chain. Furthermore, none of the times will be of a chain which a newer set may depend on. For instance, if set A is a full set older than t, and set B is an incremental based on A which is newer than t, then the time of set A will not be returned. r)rrrrr>rrrrget_signature_chains_older_thanOrz1CollectionsStatus.get_signature_chains_older_thancC |dS)zY Return the time of the last full backup, or 0 if there is none. r)get_nth_last_full_backup_timer&rrrget_last_full_backup_timec z+CollectionsStatus.get_last_full_backup_timecCs ||}|dur dS|jS)z` Return the time of the nth to last full backup, or 0 if there is none. Nr)get_nth_last_backup_chainrr)r#rrrrrr js  z/CollectionsStatus.get_nth_last_full_backup_timecCr)zr Return the last full backup of the collection, or None if there is no full backup chain. r)r#r&rrrget_last_backup_chainur"z'CollectionsStatus.get_last_backup_chaincCsZ|jsJ|dks Jt|j|krdS|jdd}|jddd|||dS)a! Return the nth-to-last full backup of the collection, or None if there is less than n backup chains. NOTE: n = 1 -> time of latest available chain (n = 0 is not a valid input). Thus the second-to-last is obtained with n=2 rather than n=1. rNcSs |jSr6)rrrrrrrs z=CollectionsStatus.get_nth_last_backup_chain..rr)rrkrrtrJ)r#rsortedrrrr#|s  z+CollectionsStatus.get_nth_last_backup_chaincCs,g}||D] }||q||S)a Returns a list of backup sets older than the given time t All of the times will be associated with an intact chain. Furthermore, none of the times will be of a set which a newer set may depend on. For instance, if set A is a full set older than t, and set B is an incremental based on A which is newer than t, then the time of set A will not be returned. )rrSrr)r#rold_setsrrrrget_older_thans  z CollectionsStatus.get_older_thancsV|jsJfdd|jD}g}|D]}fdd|D}||q||S)a  Returns list of old backup sets required by new sets This function is similar to the previous one, but it only returns the times of sets which are old but part of the chains where the newer end of the chain is newer than t. csg|] }|jkr|qSrrrrrrrGrz=CollectionsStatus.get_older_than_required..csg|] }|kr|qSrrrr(rrrGs)rrrrSr)r#r new_chains result_setsrr&rr(rget_older_than_requireds   z)CollectionsStatus.get_older_than_requiredc Cs|}d|vrd|ddd}|jsdS|jd}g}g}t|}|D]$}dd|D}||vrK||||}|||dq't|t t ||S) z= Returns time line of specified file changed r"z\x20rrcSrrr)rEfileinforrrrGrz=CollectionsStatus.get_file_changed_record..r) replacerrr fsencoder8r>indexFileChangedStatusrTr) r#filepathmodified_filepathall_backup_setspecified_file_backup_setspecified_file_backup_typebsrWr0rrrget_file_changed_records"   z)CollectionsStatus.get_file_changed_recordrr6)rrrr{r$rrXrrrrrrrr rrrrrr!r r$r#r'r+r8rrrrris2 -5&)< 3  rc@seZdZddZddZdS)r1cCs||_||_dSr6)r2 fileinfo_list)r#r2r9rrrr$s zFileChangedStatus.__init__c Csd}dtd|jtdt|j|tdtdtdfg}|jD]'}|d}|d }|jr4td }ntd }|||t|| fq"|dd |S) Nz%20s %30s %20srzFile: %szTotal number of backup: %drrzType of file change:rrrrr) rNr2rkr9rr>r rYrwtitlerV)r#rrr backup_type backup_setr,rrrrXs   $  zFileChangedStatus.__str__N)rrrr$rXrrrrr1s r1)r{builtinsrrrrrsys duplicityrrr r r r r duplicity.gpgr version_inforrrLrrr|rrr1rrrrsB                 f