o `W@sddlmZddlmZddlmZddlZddlZddlZddlZddlm Z ddlm Z ddlm Z ddlm Z dd l mZmZmZdd lT Gd d d eZdS) )next)str)objectN)config)diffdir)log)util) GlobbingErrorFilePrefixErrorselect_fn_from_glob)*c@seZdZdZedejejBZddZ ddZ ddZ d d Z d d Z d dZddZddZddZd*ddZddZddZddZddZd d!Zd"d#Zd$d%Zd&d'Zd(d)ZdS)+SelectayIterate appropriate Paths in given directory This class acts as an iterator on account of its next() method. Basically, it just goes through all the files in a directory in order (depth-first) and subjects each file to a bunch of tests (selection functions) in order. The first test that includes or excludes the file means that the file gets included (iterated) or excluded. The default is include, so with no tests we would just iterate all the files in the directory in order. The one complication to this is that sometimes we don't know whether or not to include a directory until we examine its contents. For instance, if we want to include all the **.py files. If /home/ben/foo.py exists, we should also include /home and /home/ben, but if these directories contain no **.py files, they shouldn't be included. For this reason, a test may not include or exclude a directory, but merely "scan" it. If later a file in the directory gets included, so does the directory. As mentioned above, each test takes the form of a selection function. The selection function takes a path, and returns: None - means the test has nothing to say about the related file 0 - the file is excluded by the test 1 - the file is included 2 - the test says the file (must be directory) should be scanned Also, a selection function f has a variable f.exclude which should be true iff f could potentially exclude some file. This is used to signal an error if the last function only includes, which would be redundant and presumably isn't what the user intends. z(.*[*?[]|ignorecase\:)cCs0t|ts Jt|g|_||_|jj|_dS)z/Initializer, called with Path of root directoryN) isinstancePathrselection_functionsrootpathuc_nameprefix)selfpathr5/usr/lib/python3/dist-packages/duplicity/selection.py__init__TszSelect.__init__cCs|SNrrrrr__iter__[szSelect.__iter__cCs t|jSr)riterrrrr__next__^s zSelect.__next__cCs|j||j|_|S)z)Initialize generator, prepare to iterate.)rsetdataIteraterrrrrset_iteras zSelect.set_iterc#sDddfdd}|jsttd|jdSttd|j|V|s.dS||g}g}|rz t|d\}}WntyT| |rR| Yq5w|d kr|rs|D]}t td|jd |Vq]|dd=ttd|j|V|r| ||n|d kr| || |||s7dSdS) aReturn iterator yielding paths in path This function looks a bit more complicated than it needs to be because it avoids extra recursion (and no extra function calls for non-directory files) while still doing the "directory scanning" bit. c Sstj|j|}z9t|tj}t|r,tt dt |tj j t |WdStt dt |tjjt |WdSty^tt dt |tjjt |YdSw)NzSkipping socket %szError initializing file %s'Error accessing possibly locked file %s)osrjoinnamestatST_MODES_ISSOCKrInfo_rfsdecodeInfoCodeskipping_socketescapeWarn WarningCodecannot_iterateOSError cannot_stat)excrfilenamefullpathmoderrr error_handlerrs(   z%Select.Iterate..error_handlerc3sddlm}||D]]}|tj||f}|ri|}|jdvrRt |j tj sR|dks4|dkrRt td|jt jjt|j tjrQtjjd7_q |dkr\|dfVq |dkri|ri|dfVq dS)aGenerate relevant files in directory path Returns (path, num) where num == 0 means path should be generated normally, num == 1 means the path is a directory and should be included iff something inside is included. r)robust)regdirr!N) duplicityr8listpathcheck_common_errorrappendr typer"accessr$R_OKrr.r)rr/ cannot_readrr-rstatsErrorsisdir)rr8r4new_pathsr7rrrdiryields6        z Select.Iterate..diryieldz*Warning: base %s doesn't exist, continuingNz Selecting %srr;) rArr.r)rDebugrGr StopIterationpopLogr@)rrrKdiryield_stackdelayed_path_stacksubpathval delayed_pathrrJrrhsL "      zSelect.IteratecCstd|j|jstddSd}|jD]#}||}tdt||jf|dkr0d}q|dks8|dkr:nq|rC|dkrCd}|d urId}|dkrTtd |S|dkr_td |S|dkseJtd |S) zARun through the selection functions and return dominant val 0/1/2zSelection: examining path %sz8Selection: + no selection functions found. Includingr;Fz,Selection: result: %4s from function: %sr<TrNzSelection: - excluding filezSelection: + including filez/Selection: ? scanning directory for matches)rrNrrrr$)rr scan_pendingsfresultrrrr s8       z Select.Selectc Csd}z|D]\}}|dkr|||dq|dkr'|j||dddq|dkr5|j|ddq|dks=|dkrS|||d|D]}||qF|d 7}q|d kr`||dq|d krn|||dq|d kr{|||q|d kr|||d q|dks|dkr|||d |D]}||q|d 7}q|dkr|||d qJd|Wnty}z | |WYd}~nd}~ww|t |ksJ| dS)aCCreate selection functions based on list of tuples The tuples are created when the initial commandline arguments are read. They have the form (option string, additional argument) except for the filelist tuples, which should be (option-string, (additional argument, filelist_fp)). rz --excludez--exclude-if-presentT) add_to_startz--exclude-device-filesz--exclude-filelistz--exclude-globbing-filelistr;z--exclude-other-filesystemsz--exclude-regexpz--exclude-older-thanz --includez--include-filelistz--include-globbing-filelistz--include-regexpzBad selection option %sN) add_selection_func glob_get_sfpresent_get_sfdevfiles_get_sffilelist_globbing_get_sfsother_filesystems_get_sf regexp_get_sfexclude_older_get_sfr parse_catch_errorlenparse_last_excludes)r argtuples filelistsfilelists_indexoptargrXerrr ParseArgssF        zSelect.ParseArgscCsTt|trttd||jftjjdSt|tr)ttd|tjj dS)zDeal with selection error exczFatal Error: The file specification %s cannot match any files in the base directory %s Useful file specifications begin with the base directory or some pattern (such as '**') which matches the base directory.z*Fatal Error while processing expression %sN) rr r FatalErrorr)r ErrorCodefile_prefix_errorr globbing_error)rr3rrrrcs    zSelect.parse_catch_errorcCs@|jr|jdjsttd|jdjftjjdSdSdS)z;Exit with error if last selection function isn't an excluderLzLast selection expression: %s only specifies that files be included. Because the default is to include all files, the expression is redundant. Exiting because this probably isn't what you meant.N)rexcluderrmr)r$rnredundant_inclusionrrrrre.s  zSelect.parse_last_excludesNcCs&|r |jd|dS|j|dS)z6Add another selection function at the end or beginningrN)rinsertr@)rsel_funcrZrrrr[;szSelect.add_selection_funccCs|}|s d|fS|drd|fS|}|dr#d}|dd}n |dr0d}|dd}|dr:|dsD|d rJ|d rJ|dd }||fS) aZ Sanitises lines of both normal and globbing filelists, returning (line, include) and line=None if blank/comment The aim is to parse filelists in a consistent way, prior to the interpretation of globbing statements. The function removes whitespace, comment lines and processes modifiers (leading +/-) and quotes. N#z+ r;r<z- r'"rL)strip startswithendswith)rlineinclude_defaultincluderrrfilelist_sanitise_lineCs     ( zSelect.filelist_sanitise_lineccsvttd|tjrdpd}z|dWnY||D]}|||\}}|s1q$| ||Vq$dS)a-Return list of selection functions by reading fileobj filelist_fp should be an open file object inc_default is true iff this is an include list list_name is just the name of the list, used for logging See the man page on --[include/exclude]-globbing-filelist zReading globbing filelist %s rN) rNoticer)rnull_separatorseekreadsplitr~r\)r filelist_fp inc_default list_name separatorr{r}rrrr_`s z Select.filelist_globbing_get_sfscs>dks dks J|jfdd} |_d|_|S)z=Return selection function matching files on other filesystemsrr;cs|r |kr SdSr)exists getdevlocrr} root_devlocrrrt}sz1Select.other_filesystems_get_sf..sel_funczMatch other filesystems)rrrqr$)rr}rtrrrr`ws  zSelect.other_filesystems_get_sfcsjdks dks Jzt|Wnty"ttd|wfdd} |_d||_|S)z0Return selection function given by regexp_stringrr;z%Error compiling regular expression %scs|jrSdSr)searchrrr}regexprrrts z&Select.regexp_get_sf..sel_funczRegular expression: %s)recompile Exceptionrr.r)rqr$)r regexp_stringr}rtrrrras  zSelect.regexp_get_sfcCsdd}d|_d|_|S)z4Return a selection function to exclude all dev filescSs|rdSdSNr)isdevrrrrrtsz(Select.devfiles_get_sf..sel_funcr;zExclude device filesrqr$)rrtrrrr^szSelect.devfiles_get_sfcsfdks dks Jt|tsJ|dkrfdd}n||} |_dr+dp,d|f|_|S) z.Return selection function given by glob stringrr;z**csSrrrr}rrsz$Select.glob_get_sf..zCommand-line %s glob: %sr}rq)rrglob_get_normal_sfrqr$)rglob_strr}rtrrrr\s zSelect.glob_get_sfcsn|dks |dks Jddlmfdd}|dkr|}ntdtjj| |_d|r/dp0d f|_|S) zEReturn selection function given by existence of a file in a directoryrr;)r?cs0rfdd}|jgrdSdSdS)Ncs>ttdjtjjtjtj rtj j d7_ dS)Nr!r;F) rr.r)rr/rDrr-rrErF)_exc _filenamerrrr7szFSelect.present_get_sf..exclude_sel_func..error_handlerr)rGcontains)rr7r?r4rrexclude_sel_funcs   z/Select.present_get_sf..exclude_sel_funcz<--include-if-present not implemented (would it make sense?).zCommand-line %s filename: %szinclude-if-presentzexclude-if-present)duplicity.robustr?rrmrnnot_implementedrqr$)rr4r}rrtrrrr]s zSelect.present_get_sfcCszt|tsJd|tddd}|dr(|tdd}d}t|dd |j }|s7t |t|||S) aMReturn selection function based on glob_str The basic idea is to turn glob_str into a regular expression, and just use the normal regular expression. There is a complication because the selection function should return '2' (scan) for directories which may contain a file which matches the glob_str. So we break up the glob string into parts, and any file which matches an initial sequence of glob parts gets scanned. Thanks to Donovan Baarda who provided some code which did some things similar to this. zThe glob string ignorez is not unicodeFz ignorecase:NTr;r) rrdecodesysgetfilesystemencodinglowerryrdr rr )rrr} ignore_casefile_prefix_selectionrrrrs  zSelect.glob_get_normal_sfcs"fdd}d|_df|_|S)zFReturn selection function based on files older than modification date c sR|sdSztj|jkrWdSWdSty(}zWYd}~dSd}~wwr)isregr"rgetmtimer$r1)rrkdaterrrts z-Select.exclude_older_get_sf..sel_funcTzSelect older than %sr)rrrtrrrrbs  zSelect.exclude_older_get_sfr)__name__ __module__ __qualname____doc__rrISglob_rerrrr rr rlrcrer[r~r_r`rar^r\r]rrbrrrrr /s,"_%0  ' "r )builtinsrrrr"r%rrr=rrrrduplicity.globmatchr r r duplicity.pathr rrrrs