o ]LbĤ @sLdZddlmZddlZddlZddlZddlmZmZddl m Z ddl m Z ddl mZmZmZmZmZmZmZmZmZmZmZmZmZmZmZmZ m!Z!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*dd l+m,Z,mZm-Z-m.Z.m/Z/m0Z0m1Z2m3Z3m4Z4m5Z5m6Z6m7Z7ej8iZ9e#8e9Z8iZ:e#;e:Z;e;d d d d e;d ddd e;d ddd e;d ddd e;d ddd e;d ddd e;d ddd e;d dejZ=e;d d?e=d e;d d@dAd fe;d dBgdCd e;d dDdEd e;d dFdGd e;d dHdId e;d dJdAd fe;d dKdIgd e;d dLdEd e;d dMdNd e;d dOdId dPZ>ej?Z@e@jABe,jCe6jDZDdQdRZEdSdTZFdUdVZGdWdXZHdYdZZId[d\ZJd]d^ZKd_d`ZLdadbZMdcddZNddedfZOdgdhZPdidjZQddkdlZRddmdnZSdodpZTddqdrZUd aVdsdtZWddudvZXdwdxZYe8dyge dzd6d{d|d}ZZd~dZ[ddZ\ddZ]ddZ^ddZ_ddZ`ddZaddZbe8dddde dfge dd6d{ddZce8dddde dfge dd6d{ddZde8dddde dfddde ddfge dd6d{ddZee8dge dd6d{ddZfe8dge dddZge8dge dddZhe8dge dddZiddZje8dddge de dfdd4d e dfddde dfgejke de8jldddZme8d4ddde dÃdfddde dŃdfddde dǃdfge dȃddʄZndS)aremotefilelog causes Mercurial to lazilly fetch file contents (EXPERIMENTAL) This extension is HIGHLY EXPERIMENTAL. There are NO BACKWARDS COMPATIBILITY GUARANTEES. This means that repositories created with this extension may only be usable with the exact version of this extension/Mercurial that was used. The extension attempts to enforce this in order to prevent repository corruption. remotefilelog works by fetching file contents lazily and storing them in a cache on the client rather than in revlogs. This allows enormous histories to be transferred only partially, making them easier to operate on. Configs: ``packs.maxchainlen`` specifies the maximum delta chain length in pack files ``packs.maxpacksize`` specifies the maximum pack file size ``packs.maxpackfilecount`` specifies the maximum number of packs in the shared cache (trees only for now) ``remotefilelog.backgroundprefetch`` runs prefetch in background when True ``remotefilelog.bgprefetchrevs`` specifies revisions to fetch on commit and update, and on other commands that use them. Different from pullprefetch. ``remotefilelog.gcrepack`` does garbage collection during repack when True ``remotefilelog.nodettl`` specifies maximum TTL of a node in seconds before it is garbage collected ``remotefilelog.repackonhggc`` runs repack on hg gc when True ``remotefilelog.prefetchdays`` specifies the maximum age of a commit in days after which it is no longer prefetched. ``remotefilelog.prefetchdelay`` specifies delay between background prefetches in seconds after operations that change the working copy parent ``remotefilelog.data.gencountlimit`` constraints the minimum number of data pack files required to be considered part of a generation. In particular, minimum number of packs files > gencountlimit. ``remotefilelog.data.generations`` list for specifying the lower bound of each generation of the data pack files. For example, list ['100MB','1MB'] or ['1MB', '100MB'] will lead to three generations: [0, 1MB), [ 1MB, 100MB) and [100MB, infinity). ``remotefilelog.data.maxrepackpacks`` the maximum number of pack files to include in an incremental data repack. ``remotefilelog.data.repackmaxpacksize`` the maximum size of a pack file for it to be considered for an incremental data repack. ``remotefilelog.data.repacksizelimit`` the maximum total size of pack files to include in an incremental data repack. ``remotefilelog.history.gencountlimit`` constraints the minimum number of history pack files required to be considered part of a generation. In particular, minimum number of packs files > gencountlimit. ``remotefilelog.history.generations`` list for specifying the lower bound of each generation of the history pack files. For example, list [ '100MB', '1MB'] or ['1MB', '100MB'] will lead to three generations: [ 0, 1MB), [1MB, 100MB) and [100MB, infinity). ``remotefilelog.history.maxrepackpacks`` the maximum number of pack files to include in an incremental history repack. ``remotefilelog.history.repackmaxpacksize`` the maximum size of a pack file for it to be considered for an incremental history repack. ``remotefilelog.history.repacksizelimit`` the maximum total size of pack files to include in an incremental history repack. ``remotefilelog.backgroundrepack`` automatically consolidate packs in the background ``remotefilelog.cachepath`` path to cache ``remotefilelog.cachegroup`` if set, make cache directory sgid to this group ``remotefilelog.cacheprocess`` binary to invoke for fetching file data ``remotefilelog.debug`` turn on remotefilelog-specific debug output ``remotefilelog.excludepattern`` pattern of files to exclude from pulls ``remotefilelog.includepattern`` pattern of files to include in pulls ``remotefilelog.fetchwarning``: message to print when too many single-file fetches occur ``remotefilelog.getfilesstep`` number of files to request in a single RPC ``remotefilelog.getfilestype`` if set to 'threaded' use threads to fetch files, otherwise use optimistic fetching ``remotefilelog.pullprefetch`` revset for selecting files that should be eagerly downloaded rather than lazily ``remotefilelog.reponame`` name of the repo. If set, used to partition data from other repos in a shared store. ``remotefilelog.server`` if true, enable server-side functionality ``remotefilelog.servercachepath`` path for caching blobs on the server ``remotefilelog.serverexpiration`` number of days to keep cached server blobs ``remotefilelog.validatecache`` if set, check cache entries for corruption before returning blobs ``remotefilelog.validatecachelog`` if set, check cache entries for corruption before returning metadata )absolute_importN)hexwdirrev)_)open) changegroup changelogcommands configitemscontextcopies debugcommandsdispatcherrorexchange extensionshg localrepomatchmerge mergestatepatchpycompat registrarrepairrepoviewrevsetscmutilsmartset streamcloneutil) constantsr fileserverclient remotefilectx remotefilelogremotefilelogserverrepack shallowbundle shallowrepo shallowstore shallowutilshallowverifier remotefilelogsdebugF)defaultsreponames cachepaths cachegroups cacheprocessscacheprocess.includepaths cachelimits1000 GBs fallbackpath)r-s fallbackrepo)r.aliassvalidatecachelogs validatecachesonserversservercachepathsserverexpirationbackgroundrepackbgprefetchrevs pullprefetchbackgroundprefetch prefetchdelayx prefetchdayssstrip.includefilesalls getfilesstepi's getfilestypes optimistics batchsizes fetchwarningsincludepatternsexcludepatterngcrepack repackonhggcrepackschainorphansbysizeT)r. experimentalspackss maxpacksizes maxchainlenisdevelsremotefilelog.bg-waiti'snodettlsdata.gencountlimitsdata.generations)s1GB100MBs1MBsdata.maxrepackpacks2sdata.repackmaxpacksizes4GBsdata.repacksizelimitrAshistory.gencountlimitshistory.generationsshistory.maxrepackpacksshistory.repackmaxpacksizes400MBshistory.repacksizelimitsships-with-hg-corecCstjtjttjdt }|ddddt dfttjdt j ttjdt j ttjd tttjd td d }ttjd |d#dd}td|ttjdttjt_ttdtjttdtjttdtttd tttdtttdttt j!dt"tt#dt$tt%dt&tt'dt(tt)dt*tt j+dt,tt j!dt-tt.dt/ttdt0t#j12d t3tt#d!t4tt5d"t6t5j7t5j8d"<dS)$zUWraps user facing Mercurial commands to swap them out with shallow versions. scloner!r/sshallowNs5create a shallow clone which uses remote file historys debugindexs debugindexdotslogpullc_s8t|r|drttd|||g|Ri|S)Nr;s(--all is not supported in a shallow repo) isenabledgetrAbortr)origuirepoargsoptsrL>/usr/lib/python3/dist-packages/hgext/remotefilelog/__init__.py _manifest&szuisetup.._manifestsmanifestFcSsDd}ztd}Wn tyYnw|r |tj|t_dSdS)Nlfs)rfindKeyError wrapfilelogr%r#_lfsmod)loadedlfsmodrLrLrM _lfsloaded/s   zuisetup.._lfsloadedrOs debugdatas_addchangegroupfilessmakechangegroups makestores applyupdatess_checkunknownfiless _checklookups _findrenamess_computeforwardmissings runcommands_collectbrokencsetssfilectxstrydiffsverifyr-s getrenamedfnsfilelogF)9rwirepeersetupfuncsappendr# peersetupr wrapcommandr table cloneshallowrr debugindex debugindexdotlogpull afterloadeddebugdatashallowr(shallowcg1packerrcgpacker wrapfunctionaddchangegroupfilesmakechangegroupr storewrapperr exchangepullr applyupdatescheckunknownfilesr workingctx checklookupr findrenamesr computeforwardmissingr runcommandr_collectbrokencsets changectxfilectxworkingfilectxrtrydiff_verifyfileprefetchhooksadd_fileprefetchhook getrenamedfnr filelogrevsetfilelogsymbols)rHentryrNrVrLrLrMuisetup s`    rc s|dr1gfdd}ttd|ddfdd}ttd |dd d }ttd |z&|||g|Ri|W|drTD]}t|drS|jqFdSdS|drmD]}t|drl|jq_ww)Nshallowcst|sR|t|j|t|tjr&|jjd|jf|j_|j t j | t|Wdn1sAwYt||g|Ri|S||g|Ri|SNr)rDrY unfiltered setupclientrH isinstancer __class__ __bases__ requirementsryr"SHALLOWREPO_REQUIREMENTlockrwritereporequirementsrj)rGselfrJkwargs)reposrLrM pull_shallowas    z"cloneshallow..pull_shallowrCcs fdd}td|dS)NcsX}tj|vr)i}jrdj|d<jr!dj|d<jdi|S|S)Nincludepatternexcludepatternstream_out_shallow)r) capabilitiesr"NETWORK_CAP_LEGACY_SSH_GETFILESrjoinr _callstream)rGcapsrKremoterIrLrMstream_out_shallows zAcloneshallow..setup_streamout..stream_out_shallows stream_out)rrf)rIrrrLrrMsetup_streamout~s z%cloneshallow..setup_streamoutcs|j|j||SN)rIr)rGop)rrLrM stream_wrapsz!cloneshallow..stream_wrapsmaybeperformlegacystreamcloneFcSs`d|jdgvrdd|jdD|jd<|rdS|||d\}}|dur,|tj||fS)Nv2sstreamcSsg|]}|dkr|qS)rrL).0crLrLrM sz?cloneshallow..canperformstreamclone..)FN)bundle2)remotebundle2capsrEryr"r)rGpullopr supportedrrLrLrMcanperformstreamclones  z+cloneshallow..canperformstreamclonescanperformstreamclones fileservicerW) rErrfrrr safehasattr fileserviceclose) rGrHrIrJrKrrrrrL)rrrMr]]s:          r]cOs8tjj}zddtj_||i|W|tj_S|tj_w)NcSsdS)Nr!rL)xrLrLrMsz"debugdatashallow..)r%__len__)rGrJkwdsoldlenrLrLrMrcs  rccCsr|sdS|ddt|ddt|dd}t|}|r&|r&td|r-t|||r7t||dSdS)Nshookssupdate.prefetchscommit.prefetchr-r1s+Cannot be both a server and shallow client.) local setconfig wcpprefetch configboolrD RuntimeErrorrr& setupserver)rHrIisserverenabledisshallowclientrLrLrM reposetups  rcCs>t|tjsdSt|t|t|t |j |_ dSr) rrlocalrepositoryr& onetimesetuponetimeclientsetupr)wraprepor* wrapstorestore)rHrIrLrLrMrs   rcCs$||||}tj|vrt|}|Sr)r"rr*r)rGrpathvfstypesrLrLrMris   ric Ksjt|r(|}g} |tjgD]\} } } | | t|| fq|j| |||||||fi|Sr) rDmanifest getactions mergestatemod ACTION_GETrYrrprefetch) rGrImresultwctxmctx overwrite wantfiledatarKrfilesfrJmsgrLrLrMrks rkcOst|rRg}||} |D]:\} \} } } | r| | sq| tjtjtjfvr6|| t | | fq| tj krK| d}||t | |fq|j |||||||g|Ri|Sr)rDmaybesparsematchrevfilemaprACTION_CREATEDACTION_DELETED_CHANGEDACTION_CREATED_MERGErYrfilenodeACTION_LOCAL_DIR_RENAME_GETrr)rGrIrrforcerrJrr sparsematchrm actionargsrf2rLrLrMrls$   rlc Cs`|j}t|r*g}|jD]}|D]}||vr"||t||fqq |j|||||Sr)_reporD_parentsrYrrrr)rGrrmtime_boundaryrI prefetchfilesparentrrLrLrMrn s   rnc Osht|r&g}|d}|D]} | |vr|| t|| fq|j||||||g|Ri|S)N.)rDrrYrrr) rGrImatcheraddedremovedrJrrpmfrrLrLrMros  roc Cs||||d}|j}t|rA|}g}||}|r;t} |D]} || r8|| t|| f| | q"| }|j ||S)N)r) rrDrrrsetrYrryrr) rGabrmissingrImbrr sparsemissingrrLrLrMrp&s   rpcOsPd}|r t|r |j}z|||g|Ri|W|r|SS|r'|wwr)rDrr)rGluirIrJrrrLrLrMrq<s   rqcs*trtfdd|D}|||S)Ncsg|] }|s|qSrL) shallowmatchrrrIrLrMrMsz'_collectbrokencsets..)rDlist)rGrIrstriprevrLrrMrrKs rrcCsN|dur ||}t|jr|j|rtj|j||||dS|||||dS)N)fileidrsr})rr})rrDrrr$)rGrrrr}rLrLrMrtRs  rtcCs8t|jr|j|rtj|j|||dS||||dS)N)rmr})r})rDrrr$remoteworkingfilectx)rGrrr}rLrLrMru\s  ruc  Ost|rEg} |} |||D].}|| vr(| ||}|r(| |t|f||vr>| ||}|r>| |t|fq|j| |||||||||| g | Ri| Sr)rDrrrYrrr)rGrIrevsctx1ctx2modifiedrrcopy getfilectxrJrrmf1fnamefnoderLrLrMrves:   rvcCs,|}z t|W|S|wr)rr,verifyrelease)rGrIlevelrrLrLrMrwsrwcsRtrdSdag dfdd }ttjd|fdd}ttjd|dS) NTc sHt|tr||||||||| | f |S|||||||||| | d S)N) _metatuple)rintrY) rGrrawtext transactionlinkp1p2nodeflags cachedeltarpendingfilecommitsrLrMaddrawrevisions6 z*onetimeclientsetup..addrawrevisionsaddrawrevisionc st|}||g|Ri|}t|}||krDD])}|\ }} } } } } }}}}|| }||kr>|| | || | |||| qtdntddDdkrTtddd=|S)Ns4pending multiple integer revisions are not supportedcSsh|]}|dqS)rL)rrrLrLrM z;onetimeclientsetup..changelogadd..r!)lenr rrProgrammingError)rGrrJrrr newlenoldargsr`rttrrrrnflrrlinknoder rLrM changelogadds&   z(onetimeclientsetup..changelogaddsaddNN) clientonetimerrfr%r)rHrrrLr rMrs + rcs2tr tr||Sifdd}|S)Ncs||ivr||Sz/||}|D]}||kr2|}|o+|d||<q|}|o<|dWStjyHYdSw)zlooks up all renames for a file (up to endrev) the first time the file is given. It indexes on the changerev and only parses the manifest if linkrev != changerev. Returns rename info for fn at changerev rev.rN) setdefaultrt ancestorsrrenamedrr LookupError)fnrfctxancestorr rcacherIrLrM getrenameds   z getrenamedfn..getrenamed)rDr usechangesetcentricalgo)rGrIendrevr'rLr%rMr{s  r{c st|s ||||St|td}tj|j||gd|ddtt |sL|D]}||}| } D]}||vrI | nq:q,n,fdd|dD} | D]}|d |}  | | D] }  | qmqYtfdd|DS) a``filelog(pattern)`` Changesets connected to the specified filelog. For performance reasons, ``filelog()`` does not show every changeset that affects the requested file(s). See :hg:`help log` for details. For a slower, more accurate result, use ``file()``. sfilelog requires a patternsrelpathN)r.ctxc3s|] }|r|VqdSrrLr)rrLrM 1sz filelogrevset..csg|]}|vr|qSrLrLrr)rrLrMr8z!filelogrevset..)rDr getstringrmatchmodrrootgetcwdrpatkindrryrrtlinkrevrrbaseset) rGrIsubsetrpatrr*cfilesrrr#actxrL)rrrMr|s4     r|sgcshg gc [REPO...])norepoc Ost}tj|dd}|r||g}|jd}|r ||||g}|D])}zt |i|} || tj| j dd} | rG|| Wq)t j yRYq)w|D]} t || qU|D] } t|| jq_dS)z4garbage collect the client and server filelog cachesT) allowemptysPWDN)rr+ getcachepathryenvironrErYextendrpeerrHr RepoErrorgcclientr&gcserverr) rHrJrK cachepaths systemcache repopathspwdrrepopathrI repocache cachepathrLrLrMgc;s4       rIc s^tj|d}tj|s|td|dSt|d}dd|D}|g}t }d}d}d} |j tddt |d } |D]} | | | d 7} z t tj| } Wn!tyz} z|td | | ftWYd} ~ qCd} ~ wwz t|i| } | j}Wn tjyYqCw|| t|sqCt |d s|td | qC|jdd}|jdd}|r|rz t|d}WqCttj fyYnw|j!|s|j"}fdd}tj#|||d}qC| $t%d}zt|d}|&dd|D|Wt%|nt%|w|dur!|'|dS|s-|tddSdS)Nsrepossno known cache at %s srbcSsh|]}|ddqS)NrLr,rLrLrMrir-zgcclient..Frsanalyzing repositories)unittotalr!swarning: malformed path: %r:%s snames.repo %s is a misconfigured remotefilelog repo r-r=r<Tcst|t|Sr)r# getcachekeyrrrreponamerLrMkeyfnszgcclient..keyfn)rQ lastkeepkeysr@swbcSsg|]}d|qS)%s rLr,rLrLrMrrzgcclient..s$warning: no valid repos in repofile )(osrrexistswarnrr readlinesrr makeprogressrupdater expandpathnormpath TypeError traceback print_excrr>rrr?rYrDrrHr repackmodincrementalrepackIOErrorRepackAlreadyRunningname sharedstorekeepsetcompleteumask writelinesrI)rHrH repospath reposfiler validreposkeepkeys sharedcache filesrepackedcountprogressrer>rI repackonhggcgcrepackrQoldumaskrLrOrMr@as              r@c Ost|s|||g|Ri|S|d}|d}|rX|r |r$d|d<|sX|sXt|d|t|}| }|rO|D]} tj | | sNd}nq?|rX| t d|||g|Ri|S)NfollowrTrrFsEwarning: file log can be slow on large repos - use -f to speed it up )rDrErrr byteskwargsanypatsrrTrisfilewjoinrVr) rGrHrIpatsrKrurrrxfilerLrLrMr`s.    r`cCs$|dd}|dkrd||f}|S)zUpdate revset so that only changesets no older than 'prefetchdays' days are included. The default value is set to 14 days. If 'prefetchdays' is set to zero or negative value then date restriction is not applied. r-r9rs(%s) & date(-%s)) configint)rHrdaysrLrLrM revdatelimits  r~cCs|jdd}|jd}d}t|d'tj|}t||kr3t |dd}Wd|SWd|S1s>wY|S)zCheck that enough time has passed since the last background prefetch. This only relates to prefetches after operations that change the working copy parent. Default delay between background prefetches is 2 minutes. r-r7s lastprefetchFaNT) rHr|vfsrrrTrgetmtimetimeutime)rItimeoutrreadymodtimerLrLrM readytofetchs      rc sbt}|ddt}|rr|sdSjddt|fdd}|dS)zPrefetches in background revisions specified by bgprefetchrevs revset. Does background repack if backgroundrepack flag is set in config. r-r4Nr3cs.tdr jr dSd_jddS)Ns ranprefetchTr')r r ranprefetchbackgroundprefetch)unused_successbgprefetchrevsbgrepackrIrLrManonszwcpprefetch..anon)rDconfigrrHrr~ _afterlock)rHrIrrisreadyrrLrrMrs   rc Os|||g|Ri|}t|r`|dd}|jdd}|jdd}|rW|tdt||g} |d} |rE|j ||d|S|j | | d|rUt j |d d |S|r`t j |d d |S) Nr-r5r3r6sprefetching file contents rr)baseT incremental) rDrrHrstatusrrrevrangerrrr_backgroundrepack) rGrHrIrzrKresultprefetchrevsetr bgprefetchrrrLrLrMras&  racOsR ddd}t|dr||_n t|drt|d||||g|Ri|S)Nc[s.|st}|tj||f|||d|S)N)headscommon bundlecaps)rryr"BUNDLE2_CAPABLITY)rGsourcerrrrrLrLrMlocalgetbundle4s z$exchangepull..localgetbundles _callstreams getbundle)NNN)r r _localreporrf)rGrIrrJrrrLrLrMrj1s  rjc Cst|rJg}|D]9\}}|tks|durq||}|}||}||D]}|r1||r@||vr@||t||fq)q|j |dSdSr) rDrrrrwalkrYrrr) rI revmatchesallfilesrrr*mfrrrLrLrMrzFs rzsdebugremotefilelogds decompresssdecompress the filelog firstshg debugremotefilelog cKtj||fi|Sr)r debugremotefilelogrHrrKrLrLrMrU rsverifyremotefilelogsdecompress the filelogs firsts#hg verifyremotefilelogs cKrr)r verifyremotefilelogrrLrLrMrarrs debugdatapackslongsprint the long hashessnodesdump the contents of nodesNODEshg debugdatapack cOstj|g|Ri|Sr)r debugdatapack)rHpathsrKrLrLrMrms rsdebughistorypackshg debughistorypack cKs t||Sr)r debughistorypackrrLrLrMrzs rs debugkeepsetshg debugkeepsetc sfdd}t|dS)Ncstj|t|Sr)r#rMrcrrNrrLrMrQszdebugkeepset..keyfn)r_re)rHrIrKrQrLrrM debugkeepsets  rsdebugwaitonrepackshg debugwaitonrepackcK t|Sr)r debugwaitonrepackrHrIrKrLrLrMr rsdebugwaitonprefetchshg debugwaitonprefetchcKrr)r debugwaitonprefetchrrLrLrMrrrcCs|ds8ddg}|ddd}|r|d||ddd}|r)|d|d|}t||}|g|d<|d sAd|d <|S) Nrevrsdraft()r-r5s(%s)r4+base)rErrYrr~)rHrKrrrrLrLrMresolveprefetchoptss     rsprefetchrrs prefetch the specified revisionssREVsrun repack after prefetchbrs'rev that is assumed to already be localshg prefetch [OPTIONS] [FILE...]) helpcategorycOsrt|}t|sttdt||}t|| d}| || d||| dr7t j |dddSdS)atprefetch file revisions from the server Prefetchs file revisions for the specified revs and stores them in the local remotefilelog cache. If no rev is specified, the default rev is used which is the union of dot, draft, pullprefetch and bgprefetchrev. File names or patterns can be used to limit which files are downloaded. Return 0 on success. srepo is not shallowrrr>TrN) rrvrDrrFrrrrrErr_r)rHrIrzrKrrLrLrMrs   rs backgroundsrun in a background processs incrementalsdo an incremental repack packsonlys&only repack packs (skip loose objects)shg repack [OPTIONS]c Os|drtj||d|ddddSd|di}z|dr-tj||dWdStj||dWdStjyR}z|jd|WYd}~dSd}~ww) N backgroundr packsonlyF)rrr)optionsrS)rEr_rr` fullrepackrbrHrV)rHrIrzrKrexrLrLrMrepack_s   rrr)o__doc__ __future__rrTrr]mercurial.noderrmercurial.i18nrmercurial.pycompatr mercurialrrr r r r r hgdebugcommandsrrrrrrrr/rrrrrrrrrrrrr r"r#r$r%r&r'r_r(r)r*r+r,commandcmdtable configtable configitemdynamicdefault _defaultlimit testedwithr repoclass_basesupportedryrrDrr]rcrrrirkrlrnrorprqrrrtrurvrwrrr{r|rIr@r`r~rrrarjrzrrrrrrrrwalkoptsCATEGORY_MAINTENANCErrrLrLrLrMsB x  p8   QV       / W( %^#