o .&aԜ@sddlZddlZddlZddlZddlZddlZddlZddlZddlmZ ddl m Z m Z ddl m Z mZddlmZddlZddlmZmZddlmZddlmZdd lmZeeZd Zd Zd d Z ddZ!ddZ"ddZ#ddZ$ddZ%   d,ddZ&Gddde'Z(Gddde)Z*Gddde*Z+Gd d!d!e*Z,Gd"d#d#e'Z-Gd$d%d%e'Z.Gd&d'd'e'Z/Gd(d)d)e'Z0Gd*d+d+eZ1dS)-N)error)datetime timedelta)tzparser) PyAsn1Error)get_trail_by_arnget_account_id_from_arn) BasicCommand) ClientError)ParameterRequiredErrorz%Y%m%dT%H%M%SZz%Y-%m-%dT%H:%M:%SZcC |tS)z;Returns a formatted date string in a CloudTrail date format)strftime DATE_FORMATdaterM/usr/lib/python3/dist-packages/awscli/customizations/cloudtrail/validation.py format_date( rcCr )z4Returns a formatted date string meant for CLI output)rDISPLAY_DATE_FORMATrrrrformat_display_date-rrcCs|jtdS)z.Returns a normalized date using a UTC timezone)tzinfo)replacertzutcrrrrnormalize_date2srcCs |ddS)zExtract the timestamp portion of a manifest file. Manifest file names take the following form: AWSLogs/{account}/CloudTrail-Digest/{region}/{ymd}/{account}_CloudTrail -Digest_{region}_{name}_region_{date}.json.gz iir) digest_s3_keyrrrextract_digest_key_date7s rcCs(zt|WStytd|w)NzUnable to parse date value: %s)rparse ValueError) date_stringrrr parse_dateAs    r!cCs$td}||std|dS)zlEnsures that the arn looks correct. ARNs look like: arn:aws:cloudtrail:us-east-1:123456789012:trail/fooz$arn:.+:cloudtrail:.+:\d{12}:trail/.+zInvalid trail ARN provided: %sN)recompilematchr) trail_arnpatternrrrassert_cloudtrail_arn_is_validHs   r'c  Cst|d} |dur4t||} td| | d}| dd} | d} | r4| s,td|dd} |d d }|d d }| sHt|} t | ||||| d }t ||| |||t |dS)aCreates a CloudTrail DigestTraverser and its object graph. :type cloudtrail_client: botocore.client.CloudTrail :param cloudtrail_client: Client used to connect to CloudTrail :type organization_client: botocore.client.organizations :param organization_client: Client used to connect to Organizations :type s3_client_provider: S3ClientProvider :param s3_client_provider: Used to create Amazon S3 client per/region. :param trail_arn: CloudTrail trail ARN :param trail_source_region: The scanned region of a trail. :param on_invalid: Callback that is invoked when validating a digest fails. :param on_gap: Callback that is invoked when a digest has no link to the previous digest, but there are more digests to validate. This can happen when a trail is disabled for a period of time. :param on_missing: Callback that is invoked when a digest file has been deleted from Amazon S3 but is supposed to be present. :param bucket: Amazon S3 bucket of the trail if it is different than the bucket that is currently associated with the trail. :param prefix: bucket: Key prefix prepended to each digest and log placed in the Amazon S3 bucket if it is different than the prefix that is currently associated with the trail. :param account_id: The account id for which the digest files are validated. For normal trails this is the caller account, for organization trails it is the member accout. ``on_gap``, ``on_invalid``, and ``on_missing`` callbacks are invoked with the following named arguments: - ``bucket`: The next S3 bucket. - ``next_key``: (optional) Next digest key that was found in the bucket. - ``next_end_date``: (optional) End date of the next found digest. - ``last_key``: The last digest key that was found. - ``last_start_date``: (optional) Start date of last found digest. - ``message``: (optional) Message string about the notification. NzLoaded trail info: %s S3BucketName S3KeyPrefixIsOrganizationTrailzAMissing required parameter for organization trail: '--account-id' OrganizationId:/) account_id trail_names3_client_providertrail_source_regiontrail_home_regionorganization_id)digest_providerstarting_bucketstarting_prefix on_invalidon_gap on_missingpublic_key_provider) r'rLOGdebuggetr describe_organizationsplitr DigestProviderDigestTraverserPublicKeyProvider)cloudtrail_clientorganization_clientr3r%r4r:r;r<bucketprefixr1r6 trail_info is_org_trail trail_regionr2r7rrrcreate_digest_traverserQsF(    rMc@s2eZdZdZd ddZddZddZd d Zd S) S3ClientProviderzCreates Amazon S3 clients and determines the region name of a client. This class will cache the location constraints of previously requested buckets and cache previously created clients for the same region. us-east-1cCs||_||_i|_i|_dSN)_session_get_bucket_location_region _client_cache _region_cache)selfsessionget_bucket_location_regionrrr__init__s zS3ClientProvider.__init__cCs||}||S)z=Creates an S3 client that can work with the given bucket name)_get_bucket_region_create_client)rU bucket_name region_namerrr get_clients  zS3ClientProvider.get_clientcCsB||jvr||j}|j|d}|dpd}||j|<|j|S)zReturns the region of a bucket)BucketLocationConstraintrO)rTrZrRget_bucket_location)rUr[clientresultregionrrrrYs      z#S3ClientProvider._get_bucket_regioncCs,||jvr|jd|}||j|<|j|S)z5Creates an Amazon S3 client for the given region names3)rSrQ create_client)rUr\rarrrrZs   zS3ClientProvider._create_clientN)rO)__name__ __module__ __qualname____doc__rXr]rYrZrrrrrNs   rNc@seZdZdZdS) DigestErrorz0Exception raised when a digest fails to validateN)rfrgrhrirrrrrjsrjc eZdZdZfddZZS)DigestSignatureErrorz3Exception raised when a digest signature is invalidc d||f}tt||dS)Nz=Digest file s3://%s/%s INVALID: signature verification failed)superrlrXrUrHkeymessage __class__rrrXszDigestSignatureError.__init__rfrgrhrirX __classcell__rrrrrrlrlcrk)InvalidDigestFormatz4Exception raised when a digest has an invalid formatcrm)Nz.Digest file s3://%s/%s INVALID: invalid format)rnrwrXrorrrrrXszInvalidDigestFormat.__init__rtrrrrrrwrvrwc@ eZdZdZddZddZdS)rEz:Retrieves public keys from CloudTrail within a date range.cCs ||_dSrP)_cloudtrail_client)rUrFrrrrXs zPublicKeyProvider.__init__cCs6|jj||d}|d}td|tdd|DS)aLoads public keys in a date range into a returned dict. :type start_date: datetime :param start_date: Start date of a date range. :type end_date: datetime :param end_date: End date of a date range. :rtype: dict :return: Returns a dict where each key is the fingerprint of the public key, and each value is a dict of public key data. ) StartTimeEndTime PublicKeyListzLoaded public keys in range: %scss|] }|d|fVqdS) FingerprintNr).0rprrr sz4PublicKeyProvider.get_public_keys..)rylist_public_keysr>r?dict)rU start_dateend_date public_keyspublic_keys_in_rangerrrget_public_keyss   z!PublicKeyProvider.get_public_keysN)rfrgrhrirXrrrrrrEs rEc@s>eZdZdZ  d ddZddZddZd d Zd d ZdS)rCa5 Retrieves digest keys and digests from Amazon S3. This class is responsible for determining the full list of digest files in a bucket and loading digests from the bucket into a JSON decoded dict. This class is not responsible for validation or iterating from one digest to the next. NcCs,||_||_||_||_|p||_||_dSrP)_client_providerr2r1r5r4r6)rUr3r1r2r5r4r6rrrrXs   zDigestProvider.__init__cCsg}|||}|j|}|d}|j||d} | d} tt|} tt|tdd} t | |} | D]}| |rUt |}|| krL|S|| krU||q:|S)aReturns a list of digest keys in the date range. This method uses a list_objects API call and provides a Marker parameter that is calculated based on the start_date provided. Amazon S3 then returns all keys in the bucket that start after the given key (non-inclusive). We then iterate over the keys until the date extracted from the yielded keys is greater than the given end_date. list_objects)r^MarkerzContents[*].Key)hours)_create_digest_keyrr] get_paginatorpaginatesearchrrrr"r#_create_digest_key_regexr$rappend)rUrHrIrrdigestsmarkerra paginator page_iterator key_filtertarget_start_datetarget_end_datedigest_key_regexrpextracted_daterrrload_digest_keys_in_ranges*       z(DigestProvider.load_digest_keys_in_rangec Cs|j|}|j||d}zt|dtjdB}t| }Wnt t fy1t ||wd|dvs>d|dvrCt |||dd|d<|dd|d<||fS) zlLoads a digest by key from S3. Returns the JSON decode data and GZIP inflated raw content. r^KeyBody signatureMetadatazsignature-algorithm _signature_signature_algorithm)rr] get_objectzlib decompressread MAX_WBITSjsonloadsdecoder ZLibErrorrwrl)rUrHrprarbdigest digest_datarrr fetch_digest#s"      zDigestProvider.fetch_digestcCsz|tdd}d}|jt||d|j|j|jd}|jr'|d7}|j|d<|d7}|jd i|}|r;|d |}|S) a~Computes an Amazon S3 key based on the provided data. The computed is what would have been placed in the S3 bucket if a log digest were created at a specific time. This computed key does not have to actually exist as it will only be used to as a Marker parameter in a list_objects call. :return: Returns a computed key as a string. r)minutesAWSLogs/z%Y/%m/%d)r1rymd source_region home_regionname{organization_id}/r6z{account_id}/CloudTrail-Digest/{source_region}/{ymd}/{account_id}_CloudTrail-Digest_{source_region}_{name}_{home_region}_{date}.json.gzr/Nr) rr1rrr4r5r2r6format)rUr key_prefixrtemplatetemplate_paramsrprrrr:s&   z!DigestProvider._create_digest_keycCsd}t|jt|jt|jt|jd}|jr%|d7}|j|d<|d7}|jd i|}|rr?)rUr before_keyr next_key_datebefore_key_daterrrrs&    z DigestTraverser._get_last_digestc Cs|j||\}}|jD] }||vrt||q |d|ks$|d|kr,td||f|d}||vr@td|||jj|f||d}|j|||||tt |d} || fS)zLoads and validates a digest from S3. :param public_keys: Public key dictionary of fingerprint to dict. :return: Returns a tuple of the digest data as a dict and end_date :rtype: tuple rrzIDigest file s3://%s/%s INVALID: has been moved from its original locationrzTDigest file s3://%s/%s INVALID: public key not found in region %s for fingerprint %sValuer) r7rrequired_digest_keysrwrjr5rvalidaterr!) rUrrHrprr required_key fingerprintpublic_key_hexrrrrrs6       z)DigestTraverser._load_and_validate_digestcCs.|j||}|stdt|t|f|S)Nz&No public keys found between %s and %s)rr RuntimeErrorr)rUrrrrrrrsz!DigestTraverser._load_public_keys)NNNNrP)NFN) rfrgrhrirrXrrrrrrrrrrrDss A  rDc@rx)rz Validates SHA256withRSA signed digests. The result of validating the digest is inserted into the digest_data dictionary using the isValid key value pair. c Csz$t|}tjj|dd}|||}t|d}t|||WdSt y6t d|||dftj j yCt ||w)aValidates a digest file. Throws a DigestError when the digest is invalid. :param bucket: Bucket of the digest file :param key: Key of the digest file :param public_key: Public key bytes. :param digest_data: Dict of digest data returned when JSON decoding a manifest. :param inflated_digest: Inflated digest file contents as bytes. DER)rrzNDigest file s3://%s/%s INVALID: Unable to load PKCS #1 key with fingerprint %srN)base64 b64decodersa PublicKey load_pkcs1_create_string_to_signbinascii unhexlifyverifyrrjpkcs1VerificationErrorrl) rUrHrp public_keyrinflated_digest decoded_keyto_signsignature_bytesrrrr0s     z!Sha256RSADigestValidator.validatecCsP|d}|dur d}d|d|d|dt||f}td||S)Nrnullz%s %s/%s %s %srrrzDigest string to sign: %s)hashlibsha256 hexdigestr>r?encode)rUrrprevious_signaturestring_to_signrrrrNs  z/Sha256RSADigestValidator._create_string_to_signN)rfrgrhrirrrrrrr(s rc seZdZdZdZdZdddddd ddd dd dd d dddd dddd dddd dddddgZfddZddZddZ dd Z d!d"Z d#d$Z d%d&Z dsz6CloudTrailValidateLogs._download_log.. hashValuerzLog file s3://%s/%s validrrrN)r3r]rr decompressobjrrriterrupdateflushr_on_log_invalidrr-r r_on_missing_logr_on_invalid_log_format) rUr1ra gzip_inflater rolling_hashchunkdataremaining_data computed_hashrrr7rr.s8       z$CloudTrailValidateLogs._download_logFcCsZ|r|jrtjd|ntjd|d|_dS|jr+d|_tjd|dSdS)Nz%s z %s TFz%s )rsysstderrwriterstdout)rUrqis_errorrrrr-/s z$CloudTrailValidateLogs._write_statuscCs(tjd|jt|jt|jfdS)Nz5Validating log files for trail %s between %s and %s )rHrKrJr%rrrrUrrrr+:s z*CloudTrailValidateLogs._write_startup_textcCs|js tjdtjdt|jt|jf|js'|js'tjddS|j r-|j s4tjdntjdt|j t|j f| |j|jd| |j |j dtjddS)N zResults requested for %s to %s zNo digests found z No valid digests found in range zResults found for %s to %s: rr1)rrHrKrJrrrrrrr _write_ratiorrrMrrrr/@s(    z*CloudTrailValidateLogs._write_summary_textcCsP||}|dkr$tjd|||f|dkr&tjd|||fdSdSdS)Nrz %d/%d %s files validz, %d/%d %s files INVALID)rHrKrJ)rUvalidinvalidrtotalrrrrOSs  z#CloudTrailValidateLogs._write_ratiocKs&|jd7_|d||fddS)Nrz)Digest file s3://%s/%s INVALID: not foundTrr-)rUrHrkwargsrrrr([s z)CloudTrailValidateLogs._on_missing_digestcKs(|dt|dt|dfddS)Nz;No log files were delivered by CloudTrail between %s and %srrT)r-r)rUrTrrrr*`s  z%CloudTrailValidateLogs._on_digest_gapcKs|jd7_||ddS)NrTrS)rUrqrTrrrr)fsz)CloudTrailValidateLogs._on_invalid_digestcC.|jd7_|d|d|dfddS)Nrz+Log file s3://%s/%s INVALID: invalid formatr5r6Trr-rUlog_datarrrrAjz-CloudTrailValidateLogs._on_invalid_log_formatcCrU)Nrz5Log file s3://%s/%s INVALID: hash value doesn't matchr5r6TrVrWrrrr?prYz&CloudTrailValidateLogs._on_log_invalidcCrU)Nrz&Log file s3://%s/%s INVALID: not foundr5r6TrVrWrrrr@vrYz&CloudTrailValidateLogs._on_missing_log)F)rfrgrhriNAME DESCRIPTION ARG_TABLErXr!rrrr,r.r-r+r/rOr(r*r)rAr?r@rurrrrrr]sX-   r)NNNNNNN)2rrrrloggingr"rHrrrrrdateutilrr pyasn1.errorrr&awscli.customizations.cloudtrail.utilsrr awscli.customizations.commandsr botocore.exceptionsr awscli.schemar getLoggerrfr>rrrrrrr!r'rMobjectrNrrjrlrwrErCrDrrrrrrsR          N#65