o ]Lbz@sddlmZddlZddlZddlZddlZddlmZddlm Z ddl m Z ddl m Z mZmZmZmZmZmZmZmZmZddlmZdd lmZd d Zd d ZdDddZddZddZdEddZ ddZ!ddZ"dFddZ#ddZ$d d!Z%d"d#Z&Gd$d%d%e'Z(dZ)dZ*d&Z+d'Z,d(d)Z-d*d+Z.ej/d,d-Z0d.d/Z1d0d1Z2d2d3Z3d4d5Z4d6d7Z5d8d9Z6ej/d:d;Z7dd?Z9d@dAZ:dBdCZ;dS)G)absolute_importN)_)open) repository) bookmarks cacheutilerror narrowspecphasespycompat requirementsscmutilstoreutil)nodemap) stringutilcCs t|}|tj8}|||S)a determine the final set of requirement for a new stream clone this method combine the "default" requirements that a new repository would use with the constaint we get from the stream clone content. We keep local configuration choice when possible. )setrequirementsmodSTREAM_FIXED_REQUIREMENTSupdate)default_requirementsstreamed_requirementsr r7/usr/lib/python3/dist-packages/mercurial/streamclone.pynew_stream_clone_requirements&s  rcCs|jtj@}|S)zthe set of requirement the new clone will have to support This is used for advertising the stream options and to generate the actual stream content.)r rr)reporequiredformatsrrrr3s rFc Cs |j}|j}d}|jrd|jdgvrd}|r|sdS|r"|s"dSt|r(dS|jr-dS|j}|dur9|d}|s=dSt }|drO| t j d|fS|d}|sa|jj td dSt |d }||j}|r|jj td d t||jj td dS|}d|fS)aWhether it is possible to perform a streaming clone as part of pull. ``bundle2`` will cause the function to consider stream clone through bundle2 and only through bundle2. Returns a tuple of (supported, requirements). ``supported`` is True if streaming clone is supported and False otherwise. ``requirements`` is a set of repo requirements from the remote, or ``None`` if stream clone isn't supported. Fsv2sstreamT)FNNsstream-preferreds streamreqss=warning: stream clone requested but server has them disabled ,sGwarning: stream clone requested but client is missing requirements: %s , sQ(see https://www.mercurial-scm.org/wiki/MissingRequirement for more information) )rremote canusebundle2remotebundle2capsgetlenheadsstreamclonerequestedcapableraddrREVLOGV1_REQUIREMENTuiwarnrsplit supportedjoinsorted) pullopbundle2rr bundle2supportedstreamrequestedr streamreqs missingreqsrrrcanperformstreamclone>sb    !   r6c Csddlm}t|\}}|sdS|j}|j}d}|dr9|}|di}Wdn1s4wY|j t d|}|di}Wdn1sXwY| } zt | } Wntyvtt d| w| dkrtt d| d krtt d | d krtt d | } z tt | d d\} } Wnttfytt d| w|:t||| | t|j||_||j |j|j|j_t|t ||r|j!"|||#WddS1swYdS)zPossibly perform a legacy stream clone operation. Legacy stream clones are performed as part of pull but before all other operations. A legacy stream clone will not be performed if a bundle2 stream clone is supported. r localrepoNs branchmapsstreaming all changes s stream_out'unexpected response from remote server:soperation forbidden by servers$locking the remote repository failedrs%the server sent an unknown error code )$r8r6rr r'commandexecutor callcommandresultr*statusrreadlineint ValueErrorr ResponseErrorAbortmapr, TypeErrorlock consumev1rr resolvestorevfsoptionsfeaturessvfsoptionsrwritereporequirementsrpost_stream_cleanup _branchcachesreplace invalidate) r0r8r-r rr rbranchmapefplresp filecount bytecountrrrmaybeperformlegacystreamclonesh           "rZcCsFtj|jvrdS|jjddddsdSt|}|r!|jddSdS)z5Whether streaming clones are allowed from the server.Fsservers uncompressedT) untrustedsuncompressedallowsecret)rREPO_FEATURE_STREAM_CLONErKr* configboolr hassecret)rsecretrrrallowservergenerations  r`cCs |j|SN)rwalk)rmatcherrrr_walkstreamfiles rdcsgd})jdtD]\}}}|r%||f||7}qtWdn1s4wYtjdt|fjjj fdd}t||fS)aEmit content for version 1 of a streaming clone. This returns a 3-tuple of (file count, byte size, data iterator). The data iterator consists of N entries for each file being transferred. Each file entry starts as a line with the file name and integer size delimited by a null byte. The raw file data follows. Following the raw file data is the next file entry, or EOF. When used on the wire protocol, an additional line indicating protocol success will be prepended to the stream. This function is not responsible for adding it. This function will obtain a repository lock to ensure a consistent view of the store is captured. It therefore may raise LockError. r scanning Ns%d files, %d bytes to transfer c 3sD]I\}}rjd||fdt||fV|ddd }|dkr0||Vn tj||dD]}|Vq7Wdn1sGwYqdS)Nssending %s (%d bytes) s%s%d rbF) auditpathlimit)r*debugr encodedirreadr filechunkiter)namesizerUchunk debugflagentriesrrLrremitrevlogdata"s z"generatev1..emitrevlogdata) rHr*rlrdappend_test_sync_point_walk_1_test_sync_point_walk_2r$rLrt)r total_bytes file_typerprqrvrrsr generatev1s&   r|ccslt|s dVdSz t|\}}}Wntjy!dVYdSwdVd||fV|D]}|Vq.dS)aEmit content for version 1 of streaming clone suitable for the wire. This is the data output from ``generatev1()`` with 2 header lines. The first line indicates overall success. The 2nd contains the file count and byte size of payload. The success line contains "0" for success, "1" for stream generation not allowed, and "2" for error locking the repository (possibly indicating a permissions error for the server process). s1 Ns2 s0 s%d %d )r`r|r LockError)rrXrYitrrrrrgeneratev1wireproto4s rUNcs@dkrtdt}dt|fdd}||fS)a@Emit content for version 1 of a stream clone bundle. The first 4 bytes of the output ("HGS1") denote this as stream clone bundle version 1. The next 2 bytes indicate the compression type. Only "UN" is currently supported. The next 16 bytes are two 64-bit big endian unsigned integers indicating file count and byte count, respectively. The next 2 bytes is a 16-bit big endian unsigned short declaring the length of the requirements string, including a trailing . The following N bytes are the requirements string, which is ASCII containing a comma-delimited list of repo requirements that are needed to support the data. The remaining content is the output of ``generatev1()`` (which may be compressed in the future). Returns a tuple of (requirements, data generator). rs.we do not support the compression argument yetrc3sdVVt\}}}jtd||ftd||VtdtdVdVdks8Jjjtd|td d }|d |D] }|j t|d |VqL| dS) NsHGS1swriting %d bytes for %d files >QQ>Hrrbundlebytestotalunitrstep) r|r*r@rstructpackr$ makeprogressr incrementcomplete)rXrYr~progressrr compressionrrequiresrrgenls&    zgeneratebundlev1..gen)rCrr.r/)rrr rrrrgeneratebundlev1Ps  rc Cs ||jtd|t|f|jjtd|tdd}|dt}| d|j j |j|dyt |D]k}|}z|dd\}} t| } Wnttfydttd |w|jjrv|jd |t| ft|} |j | d d d } tj|| dD]} |jt| d| | qWdn1swYq?Wdn1swY|jd dWdn1swYt|} | dkrd} ||jtdt|| t|| fWddS1swYdS)a Apply the contents from version 1 of a streaming clone file handle. This takes the output from "stream_out" and applies it to the specified repository. Like "stream_out," the status line added by the wire protocol is not handled by this function. !%d files to transfer, %s of data clonerrr) expectedcountrrr9sadding %s (%s) wT)backgroundcloserjrNclearfilecacheMbP?(transferred %s in %.1f seconds (%s/sec) )rHr*r@rrrYrrtimer transactionrLbackgroundclosingr xrangerAr,rBrCrGr rDrtrlr decodedirrorr$writerRr)rrUrXrYrstartirVrprqpathofprrelapsedrrrrIsh         "rIcCs|d}|dkrttd|td|d\}}td|dd}||}|ds9ttd t|d d }|||fS) Nr:rs<only uncompressed stream clone bundles are supported; got %srrrrs@malformed stream clone bundle: requirements not properly encodedr) rnr rErrunpackendswithrrstripr,)rUrrXrY requireslenrr rrrreadbundle1headers(    rcCsjt|r ttdt|\}}}||j}|r'ttddt|t||||t |dS)zApply the content from a stream clone bundle version 1. We assume the 4 byte header has been read and validated and the file handle is at the 2 byte compression identifier. s2cannot apply stream clone bundle on non-empty repo4unable to apply stream clone: unsupported format: %srN) r$r rErrr-r.r/rIrrO)rrUrXrYr r5rrr applybundlev1s  rc@s eZdZdZddZddZdS)streamcloneapplierzClass to manage applying streaming clone bundles. We need to wrap ``applybundlev1()`` in a dedicated type to enable bundle readers to perform bundle type-specific functionality. cCs ||_dSra)_fh)selffhrrr__init__s zstreamcloneapplier.__init__cCs t||jSra)rr)rrrrrapplyrezstreamcloneapplier.applyN)__name__ __module__ __qualname____doc__rrrrrrrs rsccCsg}|s |d|S)z!list snapshot file from the stores phaseroots) publishingrw)rfnamesrrr_walkstreamfullstorefiless rcCs2|\}}}}|tkr |S|||||||fS)z actually copy the snapshot files) _filefullr.)entrycopyvfsmapsrcrpftypedatarrr _filterfull's rc #sJgzfdd}|VWD]}t|qdSD]}t|qw)z(return a function to temporary copy filecs4t\}}t||tj||dd|S)NT)hardlink)r mkstemposcloserwr copyfiles)rfddstfilesrrr5s   zmaketempcopies..copyN)r tryunlink)rtmprrrmaketempcopies/s   rcCs&t|jt|ji}|j|vsJ|S)z$make a (src -> vfs) map for the repo) _srcstorerL _srccachecachevfsvfsvalues)rrrrr_makemapBs rc #st||jvrtd|jjtd|tdd}|dt |fdd|D}dVd}|D]\}}}}|} |Vt t |V|t kr[| |} |} n|tkrjt|d } t | j} d} zJt | V|V| d kr| | f} nt j| | d } | D]}| t |7} |t |7}|||Vq| | krttd | || fW| q;| wWdn1swYWddSWddS1swYdS) zactually emit the stream bundle9repo.vfs must not be added to vfsmap for security reasonsrrrrcsg|]}t|qSr)r).0rTrrrr bsz_emit2..Nrgrirjs=clone could only read %d bytes from %s, but expected %d bytes)rrrr ProgrammingErrorr*rrrrr uvarintencoder$ _fileappendrrfstatst_sizernrorEr)rru totalfilesizertotalbytecountrrprrrrUrqrYchunksrrrrr_emit2Os`        PrcCdSz+a function for synchronisation during testsNrrrrrrxrxcCrrrrrrrryrryc Cs8||jdus Jg}d}d}|s|rt|j||}t||D]\}}} | r>t} |tj@r1t } | t || | f|| 7}q!t |D]}|j |r]||j |j7}| t |t dfqC|rx|j drx||j dj7}| t dt dft|D]}|j|r||j|j7}| t|t dfq}||fS)aemit a seris of files information useful to clone a repo return (entries, totalfilesize) entries is a list of tuple (vfs-key, file-path, file-type, size) - `vfs-key`: is a key to the right vfs to write the file (see _makemap) - `name`: file path of the file to copy (to be feed to the vfss) - `file-type`: do this file need to be copied with the source lock ? - `size`: the size of the file (or None) Nrsobsstore) _currentlock_lockrefr matchrootrdrrFILEFLAGS_VOLATILErrwrrrLexistslstatrr cachetocopyrr) rincludesexcludesincludeobsmarkersrurrcrl_typerprqftrrr_v2_walks8     rcCs|,|jdt||||d\}}t|||}t|}|dus%Jt|Wdn1s3wYt|t|||fS)aEmit content for version 2 of a streaming clone. the data stream consists the following entries: 1) A char representing the file destination (eg: store or cache) 2) A varint containing the length of the filename 3) A varint containing the length of file data 4) N bytes containing the filename (the internal, store-agnostic form) 5) N bytes containing the file data Returns a 3-tuple of (file count, file size, data iterator). rfrrrN) rHr*rlrrnextrxryr$)rrrrrurrfirstrrr generatev2s    rc gs|d}|dd}|.|r(t| dVWdn1s"wYn dVWddSWddS1s>wYdS)Nrr)nested)ctxsthisrestrrrrs  "rc sjtd|t|ft}jjtd|tdd}|dt }j | vr:t ddfdd| D}t|gt|D]Z}t|d } || } t|} t|} t|| } jjrjd | | t| f| | d }tj|| d D]}|jt|d ||qWdn1swYqTWdn1swYjddWdn1swYt|}|dkrd}jtdt|j|t|j|f|WddS1swYdS)zApply the contents from a version 2 streaming clone. Data is read from an object that only needs to provide a ``read(size)`` method. rrrrrrc3s|] }|jVqdSra)rr*)rrrrr szconsumev2..rsadding [%s] %s (%s) rrjrNTrrr)rHr*r@rrrYrrrrrrr rrrrange readexactlyuvarintdecodestreamrtrlrorr$rrRposr)rrUrXfilesizerrrrrrrnamelendatalenrprrrrrrr consumev2sn               $r csddlm}fdd|D}|rttddt|t|||tj |_ | j j j j _ttdS)Nrr7csg|] }|jvr|qSr)r-rrrrrr1sz!applybundlev2..rr)r<r8r rErr.r/r rr rJr*rKrLrMrrNrrO)rrUrXrr r8r5rrr applybundlev2.s$    r c sdgfdd}|D]=\}}}||}||} ||} | |} tj| } tj| s4t| | |tj| | d|dd q dS)NTcsdd<td_dS)NFrscopying)rtopicrrrrr copy_usedHsz_copy_files..copy_usedrF)rno_hardlink_cbcheck_fs_hardlink) r.rrdirnamerrmakedirs register_filecopyfiler) src_vfs_map dst_vfs_maprurrkrrqsrc_vfsdst_vfssrc_pathdst_pathrrrr _copy_filesEs(       rc Cs2dd|jD}dd|jD}||ksJ||_t|}|d}d}tj|r4d}t|dddd \}}t |} t |} |j j t d t ||t d d } d d|D} t| | | | } |rwt|}|d}t||Wdn1swY| | rd}nd}|j |t |||d}|j|Wdn1swY|jdfg}|||D]>\}}z||Wqty}z&|jtjkrt d}||}t|}|||f;}|j |WYd}~qd}~wwWddS1swYdS)zYcopy all content from one local repository to another This is useful for local clonecSh|] }|tjvr|qSrrWORKING_DIR_REQUIREMENTSr rrr g  zlocal_copy..cSrrr r rrrr"lr#s bookmarksrrNTrslinkingsfiles)rrrcss"|] \}}}}|||fVqdSrar)rrrrrqrrrrs zlocal_copy..slinked %d files scopied %d files s localclonesundo.backupfilesserror removing %s: %s ) r rHr bookmarksvfsr.rrrrrr*rrr$rrrrrlrrrrLextend undofilesunlinkOSErrorerrnoENOENTr forcebytestrr+)src_repo dest_reposrc_store_requirementsdest_store_requirements src_book_vfs srcbookmarksbm_countrurr dest_vfs_maprrr dst_book_vfs dstbookmarksmsgtr undo_filesundovfsundofilerTre_msgrrr local_copycsv            '        $r<)Fra)r)< __future__r contextlibr)rri18nrr r interfacesrr<rrr r r r rrrr revlogutilsrutilsrrrr6rZr`rdr|rrrIrrobjectrrrrrrrcontextmanagerrrrrxryrrrr r rr<rrrrsX    0  ]K 8 :K  <)!  E