o Qaf@s(ddlmZddlmZddlmZddlmZddlmZddlmZmZddl m Z m Z m Z m Z dd lmZdd lmZdd lmZmZmZdd lZdd lZdd lZdd lZdd lZdd lZejd kZgdZddZddZddZ ddZ!ddZ"ddZ#d"ddZ$dd ifddZ%d d!Z&d S)#)absolute_import)S3)Config)S3Uri)FileDict) dateS3toUnixdateRFC822toUnix) unicodise deunicodise deunicodise_sreplace_nonprintables)ParameterError) HashCache)debuginfowarningN)r)fetch_local_listfetch_remote_listcompare_filelistsccsz tt|}Wn tyYdSwgg}}|D]#}t|}tjttj||r:t||s9| |q| |q|||fV|D]}tj||}tj t|sct |D]}|Vq]qHdS)z_ Reimplementation of python's os.walk to nicely support unicode in input as in output. N) oslistdirr Exceptionr pathisdirjoinhandle_exclude_include_walk_dirappendislink_os_walk_unicode)topnamesdirsnondirsnamenew_pathxr(./usr/lib/python3/dist-packages/S3/FileLists.pyr !s,       r cCst}tj||d}td|d}|jD]!}|jds%|jds%q||r7d}td|j |nq|rh|j D]*}|jdsL|jdsLq=td||jf||rgd}td |j |nq=|rrtd ||Std ||S) a  Should this root/dirname directory be excluded? (otherwise included by default) Exclude dir matches in the current directory This prevents us from recursing down trees we know we want to ignore return True for including, and False for excluding z CHECK: %rFz\/$z \/\Z(?ms)TEXCL-MATCH: '%s'zINCL-TEST: %s ~ %sINCL-MATCH: '%s'z EXCLUDE: %rPASS: %r) rrrrrexcludepatternendswithsearch debug_excludeinclude debug_include)rootdirnamecfgdexcludedrr(r(r)r:s2       rc cstjt|s Jt|g}t|D]F\}}}ttjt|}|D]4}tj||}ttjt|}tj t|rY||ksM| |tjj rTt d|q%| |q%q|D]} t| D] \}}}|||fVqcq]dS)z Walk filesystem, following symbolic links (but without recursion), on python2.4 and later If a symlink directory loop is detected, emit a warning and skip. E.g.: dir1/dir2/sym-dir -> ../dir2 z+Skipping recursively symlinked directory %sN)rrrr setr r realpathrr startswithsepradd) rwalkdirsdirpathdirnames filenames real_dirpathr6current real_currentwalkdirr(r(r)_fswalk_follow_symlinks]s*   rHccs&t|D] \}}}|||fVqdS)z\ Directory tree generator path (str) is the root of the directory tree to walk N)r )rrArBrCr(r(r)_fswalk_no_symlinksusrIcCstdt}tdd}|D]T}td|d}|jD]}||r1d}td|j|nq|rL|jD]}||rKd}td|j|nq7|r^td|||||<||=qtd |q||fS) NzApplying --exclude/--includeF ignore_case CHECK: %sTr+r,z EXCLUDE: %sr-) rrrkeysr.r1r2r3r4)src_listr7 exclude_listfiler9r:r(r(r)filter_exclude_include~s4         rQc CsNdd}i}|jD]}zxd}|dkrtjtjddd}n.z tjt|dd}Wn"tyK}ztd||j fWYd}~W|rF| q d}~ww|D]-}t | }t jt j||}t t jt|}t t jt|} |||| qNW|r| q |r| wwg} t|D]} || } | | | g| fq| S) NcSs(||vr |g||<dS|||dS)N)r)r8keyvaluer(r(r)_appendsz(_get_filelist_from_file.._append-r:F)modeclosefd)rVzJ--files-from input file %s could not be opened for reading (%s), skipping.) files_fromioopensysstdinfilenor IOErrorrstrerrorcloser striprrnormpathrr6basenamesortedsortr) r7 local_pathrTfilelistfnamefeliner6rcresultrRvaluesr(r(r)_get_filelist_from_filesF       rnFcs8fdd}fdd}fdd}ttjr-tjtjr-jntdg}t dd }d}t |t t t fvrF|g}|dkrMj}|D]"} t| } | j d ks`td | | rl|sltd | || qO|D] } ||| \} }qtt|d krd}t|\}} ||} ||||| | fS)Nc sbt|}d}td|d}|D]}|d7}|ddkr$td||f|dkr)q||d}z ttt|}WntyT}z |jtjkrOWYd}~qd}~ww|| |j |j |j |j |j|j|d||j 7}d jvr|j |j |j |j }|durz||}Wn tyYqw|j |j |j |j ||||j |j ||j q|S) NrzYRunning stat() and reading/calculating MD5 values on %d files, this may take some time...riz[%d/%d]rU full_name)sizemtimedevinodeuidgidsrmd5)lenrr stat_resultstatr OSErrorerrnoENOENTupdatest_sizest_mtimest_devst_inost_uidst_gid sync_checksrwget_md5r^r?record_hardlink) loc_list len_loc_list total_sizecounter relative_filerorvrjrw)cacher7r(r)_fetch_local_list_infosR       z0fetch_local_list.._fetch_local_list_infocstd|dkr2z t}t}Wn ty!d}d}Ynwddd||ddd|d<|dfS|r\|}|}rMtj rMt |}d}n j rUt |}nt |}d}nd}|}|g|gfg}d}|D]r\} } } | ||d } | D]c} tj| | }tjt|stjt|rtd |q}tjt|rj std |q}tj| | }tjjd krd |tjj}jd krt|}|dr|dd}d|i||<q}qo||fS)Nz Compiling list of local files...rUr)rorprqrtrurrrsTFr*rz"Skipping over non regular file: %szSkipping over symbolic link: %s/normalz./ro)rrcrgeteuidgetegidrrrrxrXrnfollow_symlinksrHrIr6replacerisfiler existsrrr>spliturlencoding_moder r=)r local_urirrtru local_baserfrg single_filer5r#filesrel_rootriror)r7is_srcr(r)_get_filelist_localsn              z-fetch_local_list.._get_filelist_localcsjrAtjdkrC|tr|}n|}|D]}|dd}|s&q||d||d|dq| | jdSdSdS)Nrrsrrrqrp) cache_filerxrXmark_all_for_purgePY3rm itervaluesgetunmark_for_purgepurgesave)r local_listlocal_list_val_iterf_inforsr7r(r)_maintain_cache0s  z)fetch_local_list.._maintain_cachez!No cache file found, creating it.FrJrPz.Expecting filename or directory instead of: %sz)Use --recursive to upload a directory: %sr)rrrrrrr loadrrtypelisttupler; recursiverrrrrxrQ)argsrrrrr local_urisrrarguri list_for_urirOrr()rr7rr)rs<- ?          rcsfdddfdd }tg}tdd}t|tttfvr$|g}|dkr+j}|D]}t|}|jdks>td || |q-d } |rn|D]"}||dd \} } | | 7} | D]} | | || <| | | | qZqJn|D]}| } t jd | d d}t|dkr|\}}d|v}|t||d \} } | | 7} | D]} tj| | d| r| | || <qqpttjt|} | std| || |d}|r؈||||| <|d}|r| | || |dd 7} qpt|\}}||| fS)Ncst|}|dsdS|t|dd|dddt|dddz|dd}|d|itd |WdStyJYdSw) Nheaderszcontent-lengthetag"'z last-modified)rprw timestampz s3cmd-attrsrwzretreived md5=%s from headers) r object_inforr~intrar rKeyError)r remote_itemresponserwrr(r)_get_remote_attribsos   z.fetch_remote_list.._get_remote_attribsTc std|td}d}tt}|j|||d}|}}|}|dkrF|ddkrF|d|dd}t d ||f}t |} t d d } d } |d D]} | d |kr}| d ddkr}t t jt| d } |}d} t d d } n| d | d} || }|| rtdqUt| dt| d| dd| d ||ddd| | <d| | dvrˆt || | | | d}| | ||t| d7}| r| |fSqU| |fS)Nz*Retrieving list of remote files for %s ...z\A\s*\Zr)prefixr uri_paramsr*rrrz s3://%s/%sFrJrKeyTz(Empty object name on S3 found, ignoring.Size LastModifiedETagr)rprrw object_keyobject_uri_strbase_urirrrsrUrw)rrecompilerr bucket_listbucketobjectrfindrrxrr rrrcr rmatchrrrra record_md5) remote_urirempty_fname_rers3rrem_base_originalrem_baseremote_uri_original rem_base_lenrem_list break_nowrrRrrw)rrr(r)_get_filelist_remotesZ                z/fetch_remote_list.._get_filelist_remoteFrJrz Expecting S3 URI instead of '%s'r)rz\*|\?r)maxsplitrrrz3Expecting S3 URI with a filename or --recursive: %s)rrrrwrp)T)rrrrrr;rrrrrrrrrrxglobfnmatchr rrrcr rrrQ)rrequire_attribsrrr remote_uris remote_listrrr objectlisttmp_total_sizerRuri_strwildcard_split_resultrrestneed_recursionrrwrOr()rr7rr)rnsj B            rc sTdd}fdd}|dkr|dkrJtdttdd}g}td||||f|D]}td ||vrԈjrOtd |||=|=q3z |||||} Wnttfyvtd |td |||=|=Yq3w| rtd |||=|=q3z| |} Wn tyd} Ynw| dur| j vr | } td| |f| ||| || f||=|=q3 || ||||<||=|=q3z| |} Wn tyd} Ynw | } | durtd| |f| ||| || f||=q3 || q3D]} | |vs| |vr"| =q|||fS)NcSs |rdpdS)Nremotelocalr() is_remoter(r(r)__direction_strs z*compare_filelists..__direction_strc sPd}||vr |vstd|||v|vfdSdjvrId|vrId||vrI|d||dkrItd|||d|dfd}djv}|rp|dkr\d||dvsh|dkrpd|dvrpd}td||r|rz ||}|}Wnttfytd |td |w||krd}td |||f|S) z@Return True if src_list[file] matches dst_list[file], else FalseTzE%s: does not exist in one side or the other: src_list=%s, dst_list=%sFrpz'xfer: %s (size mismatch: src=%s dst=%s)rwrUzdisabled md5 check for %sIGNR: %s (disappeared)%s: file disappeared, ignoring.z&XFER: %s (md5 mismatch: src=%s dst=%s))rrrrr^r{r) rNdst_lst src_remote dst_remoterP attribs_match compare_md5src_md5dst_md5r7dst_listr(r)_compares6 " 0    z#compare_filelists.._compareFzVerifying attributes...rJz)Comparing filelists (direction: %s -> %s)rLzIGNR: %s (used --skip-existing)rrzIGNR: %s (transfer not needed)zDST COPY src: %s -> %szDST COPY dst: %s -> %s)rrrrrM skip_existingr^r{rrby_md5 find_md5_onerr) rNrrrrr update_list copy_pairsr same_filerwdst1rir(rr)rs|'                 r)FN)' __future__rrrrr BaseUtilsrr Utilsr r r r Exceptionsrrloggingrrrrr[rrr|rY version_infor__all__r rrHrIrQrnrrrr(r(r(r)s:        #  &-