o ]Lby@s^ddlmZddlZddlZddlmZddlmZddlm Z m Z m Z m Z mZmZmZmZmZddlmZddlmZmZmZmZmZmZed ZGd d d e jZd0ddZ d1ddZ!d1ddZ"ddZ#ddZ$ddZ% d2ddZ&ddZ'ddZ(d d!Z)d"d#Z* d3d$d%Z+d1d&d'Z,Gd(d)d)e-Z.Gd*d+d+e-Z/Gd,d-d-e-Z0d.d/Z1dS)4)absolute_importN)_short) encodingerrorlockmdiffpolicypycompatscmutilutilvfs)procutil) constants contentstoredatapack historypack metadatastore shallowutilosutilc@s eZdZdS)RepackAlreadyRunningN)__name__ __module__ __qualname__rrr?) r*r4rBrCrDrErFrGrHrIrJrrrincrementalrepackmsD      rNcCsV|jj}|jj}|jj}|jj}t|tj}t |j j j tj}|||f|||ffSN) r=r1localdatastoresr9localhistorystoresrr:rr?getlocalpackpathsvfsrbase)r*r1rPr9rQsharedpackpath localpackpathrrrr>s r>cs0tfdd|D}tfdd|D}|S)Nc3s|] }tj|VqdSrO)ospathjoin.0p)rBrr sz_topacks..c3s|]}|VqdSrOrrZ) constructorrrr])list)rBfilesr^pathspacksr)r^rBr_topackssrdc s|jdddkr |Sddgfdd|DD](}tj||}D]}||}|jd|tt|j ft |q(qfd d |DS) zsDeletes packfiles that are bigger than ``packs.maxpacksize``. Returns ``files` with the removed files omitted.packss maxpacksizers .datapacks.dataidxcs@h|]\}}}|jkrtj|dvrtj|dqS)rr)st_sizerWrXsplitext)r[rXftypestat) VALIDEXTSmaxsizerr s z"_deletebigpacks..s#removing oversize packfile %s (%s) cs$g|]}tj|dvr|qSr)rWrXbasename)r[row) oversizedrr s$z#_deletebigpacks..) r$ configbytesrWrXrYdebugr bytecountrirfunlink)r*folderra rootfnamerootpathextrXr)rjrkrpr_deletebigpackss&  rzc Cst|j|tj|dd}t|||}t|t|j|tj}| dd|Dt|t |j|t j } | dd|Dt |t j t j} t|dd| Dt j } | dd|Dt|tj|d|itj| ddi||tj| ddi|d dS) NT)rics |] }t|tjs|VqdSrO) isinstancer datapackstorer[srrrr]s   z%_incrementalrepack..csr{rOr|rhistorypackstorer~rrrr]  css|]\}}}|VqdSrOr)r[fmoderirrrr]csr{rOrr~rrrr]rr2) fullhistoryr4)rmkstickygroupdirr$rlistdirrzrd_computeincrementaldatapackrextend_computeincrementalhistorypackr_allpackfileswithsuffix PACKSUFFIX INDEXSUFFIXr<rr7rr8) r*r5 historystorerBcategoryrLr4ra datapacks historypacksallhistoryfilesallhistorypacksrrrrMsZ         rMcCsR|dd|dd|dd|dd|ddd}t|tjtj}t||S)N remotefilelogsdata.gencountlimitsdata.generationssdata.maxrepackpackssdata.repackmaxpacksizesdata.repacksizelimit gencountlimit generationsmaxrepackpacksrepackmaxpacksizerepacksizelimit) configint configlistrrrrrr_computeincrementalpackr$raopts packfilesrrrrs    rcCsX|dd|dddg|dd|ddd|ddd }t|tjtj}t||S) Nrshistory.gencountlimitshistory.generationss100MBshistory.maxrepackpacksshistory.repackmaxpacksizes400MBshistory.repacksizelimitr)rrrrrrrrrrrrrr%s(  rc Csbg}dd|D}|D]#\}}}||sq |dt| }|||vr&q ||||fq |S)NcSsh|]\}}}|qSrr)r[fnrrirrrrl@sz*_allpackfileswithsuffix..)endswithlenr#) ra packsuffix indexsuffixresultfilesetfilenamerriprefixrrrr>s  rc sttdd|dDdd}|dg}tt|D]}|gqi|D])\}}}|j}||dkr8q)||<t|D]\}} || krQ|||nq@q)g} t|D] \}} t|||dkry| t||dfd d d nqY| d d } | d d } t fdd| D} | |dkr| rt| |dkr| | | | d7} | |dkr| rt| |dks| S)a`Given a set of pack files along with the configuration options, this function computes the list of files that should be packed as part of an incremental repack. It tries to strike a balance between keeping incremental repacks cheap (i.e. packing small things when possible, and rolling the packs up to the big ones over time). css|]}t|VqdSrO)r sizetointr~rrrr]Zrz*_computeincrementalpack..rT)reverserrrcs|SrOr)xsizesrrzz)_computeincrementalpack..)rkeyNc3s|]}|VqdSrOrr[nrrrr]r_rr) r`sortedr#r xrangerrf enumeratersumpop) rarlimits generationsirrrisizelimitgenpacks chosenpacks repacksizerrrrOsR         rc Cst|j|dd}|jdd}|s|}t||||||||d} t|j|?} t|j|} z| | | Wnt j yHt t dwWdn1sSwYWddSWddS1skwYdS)NcSsD|j||d}||}|jdd}t|}|d|kS)zCheck if the file node is older than a limit. Unless a limit is specified in the config the default limit is taken. )fileidrsnodettlr)filectxlinkrevdater$rtime)r*rnoderfiletimettlrrrrisolds   z_runrepack..isoldrsgcrepack)gcrr4s3skipping repack - another repack is already running)rrr$r&repackerrmutabledatapackrmutablehistorypackrunrLockHeldrr) r*datahistoryrBrrr4rgarbagecollectpackerdpackhpackrrrr<s>  "r<cCsh|st}n|}gd}|jddd}|r|d||jddd}|r.|d|d|}d|g}t||}t}d}t|jD]i} ||  |vr]||  } n|ri||  |} n||  } ||  }|| t| turt| D]\} } | d d dur||| | d d qqHt| D] \} } ||| | qqH|S) a Computes a keepset which is not garbage collected. 'keyfn' is a function that maps filename, node to a unique key. 'lastkeepkeys' is an optional argument and if provided the keepset function updates lastkeepkeys with more keys and returns the result. ).sdraft()sparents(draft())rs pullprefetchNs(%s)sbgprefetchrevs+ssort((%s), "topo")r)setr$configr#rYr revrangereversed_listp1rev manifestctx readdeltamanifestdiffaddtypedictr iteritems)r*keyfn lastkeepkeyskeepkeysrevs prefetchrevskeep processed lastmanifestrmrrfilenoderrrkeepsetsB       rc@sHeZdZdZ   dddZddZdd Zd d Zd d ZddZ dS)rz^Class for orchestrating the repack of data and history information into a new format. FNc Cs`||_||_||_||_t||_||_||_|jr.|s!t dt |dd|_ ||_ dSdS)Ns*Function 'isold' is not properly specifiedcSs||fSrOr)rrrrrrrz#repacker.__init__..) r*rrrrgetunitsunitrr4 ValueErrorrrr) selfr*rrrrrrr4rrr__init__s   zrepacker.__init__cCst}tjt|jdddd8|jd|jj||jd|j j||jd| ||| |||j D]}| |q7WddS1sJwYdS)Ns repacklockr)desctimeouts prerepackr3) repackledgerlockmodr repacklockvfsr*hookr markledgerr4r repackdata repackhistorysourcescleanup)r targetdata targethistoryledgersourcerrrrs     "z repacker.runc ss|Sd fdd }ttt|dd|jr-|dddd DftD]#\}}|dkrBjjdf||<q1|d } | || d d f||<q1fd d |D}|7}|S)aqReorderes ``orphans`` into a single chain inside ``nodes`` and ``deltabases``. We often have orphan entries (nodes without a base that aren't referenced by other nodes -- i.e., part of a chain) due to gaps in history. Rather than store them as individual fulltexts, we prefer to insert them as one chain sorted by size. rcs&j|}tj|vr|tjS|SrO)rgetmetar METAKEYSIZE)rdefaultmeta)rrrrgetsize5s  z'repacker._chainorphans..getsizeT)rrs%s: orphan chain: %s s, cSsg|]}t|qSrrr~rrrrqDsz*repacker._chainorphans..rcsg|]}|vr|qSrrr)orphansrrrqNsNrm)rr` debugflagrsrYrr*nullid) rr$rnodesr deltabasesrrrparentr)rrrr _chainorphans)s&   zrepacker._chainorphansc' s|jj}|ddd}i}t|jD]}|jr"|||ji|j <qd}|j t d|j t |d}tt|D]\} } ||i} tdd| Dg} |j t d d t d} tD]*\}}|| vriq`| |z| |jj| || d Wq`ty| |Yq`w| tt|| }t | dkr|d t | |t| ttfd d|}|jrg}|D]}| |f|jvr||j| |rd| |_ q||q|}i}t!}t!}t!|j t dd t |d}t|D]j\}}|||"|d}|dur$|jj#d}}|jj#df||<|$|n |\}}|$|| "|}|re|\}}}}|rB|jj#}||kre||jj#krV||df||<||jj#kre||df||<q|%ddrz||}|&|| |||}t|D]x\}}||\}}||jj#kr|j'(| |}|\} }!}"}#|#"t)j*}$|!| ks|"|ks|$dur|j'"| |}%|j'"| |}&t |&}$t+,|%|&} n|j'"| |} t | }$|j'-| |}#t)j*|#vr|$|#t)j*<|$| ||| |#d| |_.q~||d7}q9||j/|ddS)Nres maxchainlenirsrepacking datartotalcs|]}|VqdSrOrr[rrrrr]cz&repacker.repackdata..sbuilding historysnodesknowns%repackdata: %d nodes without history cs|vSrOr)rrrrrrz%repacker.repackdata..Tsprocessing nodesrrschainorphansbysizer)0r*r$rr itervaluesentriesr@ setdefaultrr makeprogressrrrrrupdater`rr getancestorsKeyErrorr#completer _toposortrsrfilterrrrgcedrgetrrr&rrgetdeltarr r textdiffr  datarepackedclose)'rrtargetr$ maxchainlenbyfileentrycountrepackprogressrr  ancestors nohistory buildprogressrr orderednodesneworderednodesrnobase referencedprocessprogress deltatuple deltabasechainlen ancestorinforp2linknodecopyfromr deltaentrydelta deltabasename origdeltabaser r deltabasetextoriginalrrrrRs                          zrepacker.repackdatac Cst|jj}i}t|jD]}|jr|||ji|j<q |j t d|j t |d}t t|D]}\}}i} tdd|D} | D]} | | vrHqA| |jj|| | dqAt|| } t} | D]G} | | \}}}}| | vr| |vr||jjkr||kr| |||jjkr| |qa|r| |||| ||||| |vrd|| _qa|q0||j|ddS)Nsrepacking historyrcsrrOrrrrrr]rz)repacker.repackhistory..rTr)r*r$r rr rAr!rrr"rrrrrr`r#rr$rr'rrrhistoryrepacked incrementr&r.)rrr/r$r1r2progressrr r5rrr8 dontprocessrrArBrCrrrrsJ       zrepacker.repackhistorycs"fdd}t|}|S)NcsD|\}}}}g}|jjkr|||jjkr |||SrO)r*rr#)rrrArBrCparentsr5rrr parentfunc+s    z&repacker._toposort..parentfunc)r sortnodeskeys)rr5rP sortednodesrrOrr'*s zrepacker._toposort)FNN) rrr__doc__rrrrrr'rrrrrs  ) s zrepackledger.__init__cC@|||}d|_|j|}|st}||j|<||dS)z`Mark the given filename+node revision as having a data rev in the given source. TN)_getorcreateentryr@rr*rrrr rrr2r rrr markdataentryC   zrepackledger.markdataentrycCrV)zcMark the given filename+node revision as having a history rev in the given source. TN)rWrArr*rrrXrrrmarkhistoryentryOrZzrepackledger.markhistoryentrycCs0||f}|j|}|st||}||j|<|SrO)r r* repackentry)rrrrvaluerrrrW[s    zrepackledger._getorcreateentrycCs|j|dSrO)rUr)rr]rrr addcreateddszrepackledger.addcreatedN) rrrrTrrYr[rWr^rrrrr8s  rc@seZdZdZdZddZdS)r\zFSimple class representing a single revision entry in the repackledger.rrr@rAr-rJr)cCs.||_||_d|_d|_d|_d|_d|_dS)NFr_)rrrrrrrus zrepackentry.__init__N)rrrrT __slots__rrrrrr\hs r\cCs*t|drt|tj}t|S|jS)Nname)r r6rr:rr;rrS)r*sharedcachepathrrrrs  r)TFrO)FN)NN)2 __future__rrWrmercurial.i18nrmercurial.noder mercurialrrrrr r r r r rmercurial.utilsrrrrrrr importmodrAbortrr0rKrNr>rdrzrMrrrrr<robjectrrr\rrrrrs@   ,     8,& >? ,>C0