o ]Lbim@sddlmZddlZddlZddlZddlZddlZddlZddlZddl m Z ddl m Z ddl mZddlmZmZmZmZmZmZmZmZmZddlmZmZdd lm Z e!d Z"Gd d d ejZ#Gd dde#Z$Gdddej%Z&Gddde'Z(ddZ)Gdddej*j+Z,Gddde'Z-Gddde'Z.Gddde'Z/Gddde'Z0e-e-e.e/e0dZ1d d!Z2d"d#Z3d*d$d%Z4Gd&d'd'ej5Z6Gd(d)d)ej7Z8dS)+)absolute_importN)_)getattr)hex) encodingerrorhttpconnectionpathutilpycompaturlutilvfsworker) stringutilurlutil)lfutils\A[a-f0-9]{64}\Zcs&eZdZfddZdddZZS)lfsvfscs:t|s td|tt||dd|ddS)z9split the path at first two characters, like: XX/XXXXX...sunexpected lfs path: %srrN)_lfsrematchrProgrammingErrorsuperrjoin)selfpath __class__5/usr/lib/python3/dist-packages/hgext/lfs/blobstore.pyr.s "z lfsvfs.joinNc#stj|j}tt|}g}tj||j|pd|dD]\}}|dtdkr<| fdd|Dqdg|fVdS)zYield (dirpath, [], oids) tuple for blobs under path Oids only exist in the root of this vfs, so dirpath is always ''. )onerrorNrcs"g|] }t|r|qSr)rr).0fdirpathrr Gs"zlfsvfs.walk..) osrnormpathbaselenr normasprefixwalkreljoinextend)rrr root prefixlenoidsdirsfilesrr#rr+4s  z lfsvfs.walkNN)__name__ __module__ __qualname__rr+ __classcell__rrrrr-s rc@s6eZdZddZddZddZd dd Zd d ZdS) nullvfscCdSNrrrrr__init__Nznullvfs.__init__cCdS)NFrroidrrrexistsQr=znullvfs.existscCsttjtd|)Ns%s: No such file or directory)IOErrorerrnoENOENTr sysstrr?rrrreadTs z nullvfs.readNcCs dggfS)Nrr)rrr rrrr+^s z nullvfs.walkcCr9r:r)rr@datarrrwritear=z nullvfs.writer3)r4r5r6r<rArFr+rHrrrrr8Ms  r8cs(eZdZdZfddZddZZS) lfsuploadfilez+a file-like object that supports keepalive.cs"tt|||d|jj|_dSNsrb)rrIr<_datarF)ruifilenamerrrr<hszlfsuploadfile.__init__cCr9r:rr;rrr _makeprogresslr=zlfsuploadfile._makeprogress)r4r5r6__doc__r<rNr7rrrrrIes rIc@sjeZdZdZddZddZddZdd Zd d Zd d Z ddZ dddZ ddZ ddZ ddZdS)localzLocal blobstore for large file contents. This blobstore is used both as a cache and as a staging area for large blobs to be uploaded to the remote blobstore. cCsR|jd}t||_|jddrt|_n t |jd}t||_|j|_dS)Ns lfs/objects experimentalslfs.disableusercachelfs) svfsrrr rL configboolr8cachevfsr _usercachedir)rrepofullpath usercacherrrr<ws     zlocal.__init__cCst||dS)zgOpen a read-only file descriptor to the named blob, in either the usercache or the local store.rb)openrr?rrrr[sz local.opencCs$|j|r |j|S|j|S)a{Build the path for the given blob ``oid``. If the blob exists locally, the path may point to either the usercache or the local store. If it doesn't, it will point to the local store. This is meant for situations where existing code that isn't LFS aware needs to open a blob. Generally, prefer the ``open`` method on this class. )rUrArr r?rrrrs   z local.pathc Cst}d}|j|dddL}tj|ddD]}|||||t|7}q|durAt||krAd}t t ||t|ft | } | |krSt t d |Wdn1s]wY||dS) znRead the blob from the remote source in chunks, verify the content, and write to this local blobstore.rwbT atomictempsizeNsXResponse length (%s) does not match Content-Length header (%d): likely server-side crashscorrupt remote lfs object: %s)hashlibsha256r r filechunkiterrHupdater)intLfsRemoteErrorrrdigestLfsCorruptionError_linktousercache) rr@srccontent_lengthrcrafpchunkmsgrealoidrrrdownloads&    zlocal.downloadcCsH|j|ddd }||Wdn1swY||dS)zWrite blob to local blobstore. This should only be called from the filelog during a commit or similar. As such, there is no need to verify the data. Imports from a remote store must use ``download()`` instead.r\Tr]N)r rHrj)rr@rGrmrrrrHs z local.writecCsTt|jts&|j|s(|jtd|t |j ||j |dSdSdS)aLink blobs found in the user cache into this store. The server module needs to do this when it lets the client know not to upload the blob, to ensure it is always available in this store. Normally this is done implicitly when the client reads or writes the blob, but that doesn't happen when the server tells the client that it already has the blob. lfs: found %s in the usercache N) isinstancerUr8r rArLnoterrlinkrr?rrrlinkfromusercaches  zlocal.linkfromusercachecCsT|j|s&t|jts(|jtd|t|j ||j |dSdSdS)Ns lfs: adding %s to the usercache ) rUrArsr8rLrtrrrur rr?rrrrjs  zlocal._linktousercacheTcCs|j|s5||j||}|stt||kr3|j t d|t |j ||j ||S|j t d|||j||}|S)zRead blob from local blobstore.rrs%lfs: found %s in the local lfs store )r rA_readrUrrbrcrhrLrtrrrur)rr@verifyblobrrrrFs z local.readcCs||}|r t|||S)z0Read blob (after verifying) from the given store)rF_verify)rr r@rxryrrrrws  z local._readcCs`t}||}tj|ddD]}||qWdn1s#wY|t|kS)zQIndicate whether or not the hash of the underlying file matches its name.r_r`N)rbrcr[r rdrerrh)rr@rcrmrnrrrrxs  z local.verifycCs|j|p |j|S)zYReturns True if the local blobstore contains the requested blob, False otherwise.)rUrAr r?rrrhassz local.hasN)T)r4r5r6rOr<r[rrqrHrvrjrFrwrxr{rrrrrPps     rPc Cs|}t|jtr |j}t|dr5z|jjd}Wnttfy'|j}Ynwt|tj r3t |}|St |ddrAt |jSt|S)zbCreate a friendly message for the given URLError to be used in an LfsRemoteError message. sreasonstrerrorN)rsreason Exceptionr safehasattrargsAttributeError IndexErrorr unicoder unitolocalr strtolocalr}r forcebytestr)urlerrorinstr~rrr_urlerrorreasons        rc@seZdZdZddZdS)lfsauthhandlericCsR|dd}|r'|d}|dkr'td}tj||t |||dS)zEnforces that any authentication performed is HTTP Basic Authentication. No authentication is also acceptable. zwww-authenticateNrbasics,the server must support Basic Authentication) getsplitlowerrr urlerr httperror get_full_urlr strfromlocal)rreqrmcoderoheadersauthreqschemerrrhttp_error_401"s   zlfsauthhandler.http_error_401N)r4r5r6 handler_orderrrrrrrs rc@sTeZdZddZddZddZddZd d Zd d Zd dZ ddZ ddZ dS) _gitlfsremotecCsr|j}||_|\}}|d|_|jdd}|s!dt}t||||_ |j t | dd|_ dS)N/rQslfs.user-agentsgit-lfs/2.3.4 (Mercurial %s)rRsretry)rLauthinforstripbaseurlconfigr versionurlmodopener urlopener add_handlerr configintretry)rrWr rLrr useragentrrrr<7s   z_gitlfsremote.__init__cC|t||ddS)z,Batch upload from local to remote blobstore.uploadN_batch _deduplicaterpointers fromstorerrr writebatchCz_gitlfsremote.writebatchcCr)z-Batch download from remote to local blostore.downloadNrrrtostorerrr readbatchGrz_gitlfsremote.readbatchc Cs,dd|D}tt|t|d}d|j}tjjt||d}| dd| ddzt |j | }|}Wd n1sJwYWnYtjjy} z+td |j|ftd |jd } | | jtd ||f} ttdt| | dd } ~ wtjjy} ztd |j} ttdt| | dd } ~ wwzt|} Wntyttd|dw|jjr |jd|jt |!"} |jdd#t$| %d| vrt$| dddd| d<|jdttj| dddddd}t&|| S) zGet metadata about objects pointed by pointers for given action Return decoded JSON object like {'objects': [{'oid': '', 'size': 1}]} See https://github.com/git-lfs/git-lfs/blob/master/docs/api/batch.md cSs$g|]}t||dqS))r@ra)r strurlr@rar!prrrr%Qsz/_gitlfsremote._batchrequest..)objects operations%s/objects/batch)rGAcceptzapplication/vnd.git-lfs+json Content-TypeNs=check that lfs serving is enabled on %s and "%s" is supporteds/the "lfs.url" config may be used to override %s)isapi=%s, action=%ssLFS HTTP error: %shint LFS error: %ss#LFS server returns invalid JSON: %sutf-8 Status: %d %s  rcSs|dS)Nr@r)rrrrsz-_gitlfsremote._batchrequest..keyr)z: T)indent separators sort_keyscSst|tjr |dS|S)Nr)rsr rencode)xrrr encodestrs  z._gitlfsremote._batchrequest..encodestr)'r bytesurljsondumpsrrr urlreqrequest add_header contextlibclosingrr[rFrrrrrrgrrrr json_loads ValueErrorrrL debugflagdebugstatusbytestrinfostriprsorted splitlinesrapply)rractionr requestdatar batchreqrsprawjsonexhintsrresponserrrrr _batchrequestKs          z_gitlfsremote._batchrequestc Cs|D]X}d|vr|dkr||dgvrd}n q|ddd}dd|D}||d d }|rQt|d d }d dddd} | |d|} ttd|| fttd|d d S)zUScans errors from objects Raises LfsRemoteError if any objects have an errorserrorractionsrscodecSsi|]}||qSr)r@rrrr sz6_gitlfsremote._checkforservererror..oidNrMsunknownsThe object does not exists#The object was removed by the ownersValidation errorsInternal server error)riirsstatus code %dsLFS server error for "%s": %ss1LFS server error. Unsolicited response for oid %s)rrrgr) rr responsesrrrptrmaprrMerrorsrorrr_checkforservererrors6 z"_gitlfsremote._checkforservererrorcs0|dg}|||fdd|D}|S)zextract objects from response of the batch API response: parsed JSON object returned by batch API return response['objects'] filtered by action raise if any object has an error sobjectscs g|] }|dgvr|qS)rr)r!orrrr%sz1_gitlfsremote._extractobjects..)rr)rrrrrfilteredobjectsrrr_extractobjectss  z_gitlfsremote._extractobjectsc Cs|d}|d|d}|d|di}tjt|}|dkr8||s8tj t d|t dd|D]\}} | t|t| q:zz|dkrnt |j |||_d d |_| d d | d |jjt|j|b} | d} |j } |j jr| d| jt| }| ddt||dkr||| | n!g} | d}|sn| !|qd| }|r| d||fWdn1swYWnTtj"j#y}z|j jr|j d|| ft$t dt%&|||fd}~wtj"j'y6}zt dt(tj)*|}t$t dt+||dd}~wwW|jrC|j,dSdS|jrO|j,ww)aCDownload or upload a single object using basic transfer protocol obj: dict, an object description returned by batch API action: string, one of ['upload', 'download'] localstore: blobstore.local See https://github.com/git-lfs/git-lfs/blob/master/docs/api/ basic-transfers.md rrshrefsheaderrdetected corrupt lfs object: %s run hg verifyrcSr>)NPUTrrrrrrz._gitlfsremote._basictransfer..rzapplication/octet-streamzContent-Lengthscontent-lengthrrrrTr_rslfs %s response: %sNs%s: %s s&LFS HTTP error: %s (oid=%s, action=%s)sattempted connection to %sr)-ritemsr rrr rrxrAbortrrrIrLrrG get_methodlengthrrrr[rrrrrrrrrrqrFappendrrrgrrrr urllibcompat getfullurlrclose)robjr localstorer@hrefrrkvres contentlengthrLblocksrGrrrrrr_basictransfers            z_gitlfsremote._basictransferc sdvr td|}||}tdd|D}i}|D] }|dd||d<q%tdtd d} t|d krUj td t|t |ffd d } j ddrut jd| dt|ddd} n | t|ddd} jj| td|d1} | dd} d}| D]\}}| ||7} |d 7}| | j td|qWdn1swY|dkrdkr܈jtd|t | fdSdkrjtd|t | fdSdSdS)N)rrsinvalid Git-LFS action: %scss|] }|ddVqdS)sizerNr)r!rrrr 0sz'_gitlfsremote._batch..rrrs lfs uploadingslfs downloadingr|s&lfs: need to transfer %d objects (%s) c 3s|D]n}|dd}jjr0dkrtd}ndkr td}j||dt|fj} z|d |dfVWn+t j yp}z|dkrkjtd t ||f|d 8}WYd}~q3d}~wwqdS) Nrrrslfs: downloading %s (%s) rslfs: uploading %s (%s) rTr|s%lfs: failed: %r (remaining retry %d) ) rrLverboserrtr bytecountrr socketrrr)rnrobjsizerorrrrrrrtransfer>s>    z&_gitlfsremote._batch..transferrQslfs.worker-enableg?rcS |dSNrrrrrrr` z&_gitlfsremote._batch..rcSrrrrrrrrcrsbytes)unittotalslfs: processed: %s rslfs: uploaded %d files (%s) rslfs: downloaded %d files (%s) )rrrrsumrrr)rLrtr rrTrr makeprogressrer)rrrrrrrsizesrtopicrr0progress processedblobs_oner@rrrr*st          z_gitlfsremote._batchcCs>t|dd}|r|jD]}|t|dddq dSdS)Nr close_allcSr9r:rrrrrrrz'_gitlfsremote.__del__..)rhandlersr)rrhrrr__del__}s  z_gitlfsremote.__del__N) r4r5r6r<rrrrrr rr'rrrrr6s U)N Src@(eZdZdZddZddZddZdS) _dummyremotez,Dummy store storing blobs to temp directory.cCs|jd|j}t||_dS)NrR)r rrr)rrWr rXrrrr<sz_dummyremote.__init__c Csbt|D]*}|j|dd}|j|ddd }||Wdn1s)wYqdS)NT)rxr\r])rrFr@r rH)rrrrcontentrmrrrrs  z_dummyremote.writebatchc CsTt|D]#}||d}|||dWdn1s"wYqdSrJ)rr r@rq)rrrrrmrrrrs z_dummyremote.readbatchNr4r5r6rOr<rrrrrrr)s  r)c@r() _nullremotez&Null store storing blobs to /dev/null.cCr9r:rrrWr rrrr<r=z_nullremote.__init__cCr9r:rrrrrrr=z_nullremote.writebatchcCr9r:rrrrrrr=z_nullremote.readbatchNr+rrrrr,s  r,c@s4eZdZdZddZd ddZd ddZd d ZdS) _promptremotez)Prompt user to set lfs.url when accessed.cCr9r:rr-rrrr<r=z_promptremote.__init__NcC |dSr:_prompt)rrrrLrrrr z_promptremote.writebatchcCr/r:r0)rrrrLrrrrr2z_promptremote.readbatchcCsttd)Nslfs.url needs to be configured)rrrr;rrrr1sz_promptremote._promptr:)r4r5r6rOr<rrr1rrrrr.s    r.)httpshttpsfilesnullNcCs&t}|D]}|||<q|S)z0Remove any duplicate oids that exist in the list)r sortdictr@values)rreducedrrrrrsrcCs6tt|}||krttd|tdddS)Nrrr)rrbrcrhrir)r@r*rprrrrzs rzcCs|jdd}t|p d}|dur`|r|}nt|dr!|j}n |jddp)d}t|}|jdvr`|jrG|jdd d krG|jd 7_|jpKdd |_tt |}|j t d ||j}|t vrpt t d |t |||S)aremotestore factory. return a store in _storemap depending on config If ``lfs.url`` is specified, use that remote endpoint. Otherwise, try to infer the endpoint, based on the remote repository using the same path adjustments as git. As an extension, 'http' is supported as well so that ``hg serve`` works out of the box. https://github.com/git-lfs/git-lfs/blob/master/docs/api/server-discovery.md rRsurlrNs _subtoppathspathssdefault)r4r3rs .git/info/lfsslfs: assuming remote store: %s slfs: unknown url scheme: %s)rLrrr r r _subtoppathrrbytesrtr _storemaprr)rWremotelfsurlr r defaulturlrrrrr<s&    r<c@s eZdZdS)rgN)r4r5r6rrrrrgsrgc@seZdZdZdS)riz~Raised when a corrupt blob is detected, aborting an operation It exists to allow specialized handling on the server side.N)r4r5r6rOrrrrrisrir:)9 __future__rrrCrbrr&rermercurial.i18nrmercurial.pycompatrmercurial.noder mercurialrrrhttpconnectionmodr r r rr r vfsmodrmercurial.utilsrr largefilesrcompilerrr8 httpsendfilerIobjectrPrr basehandlerrrr)r,r.r;rrzr< StorageErrorrgrrirrrrsN    ,    R   +