What to do if you inadvertently expose an AWS Access Key or STS token ?

AWS IAM keys can be exposed by mistake, by automation and also intentionally. Another possibility is exposure of STS token i.e. short term/temporary credentials for an AWS account. Temporary credentials can be valid up to 12 hours.

STS token access key always starts with “AKIA”.

Organizations need to pro-actively remediate these incidents in most effective way. On other hand, AWS also detect these exposed IAM keys in public git repos and send email to account DL.

Steps to follow :

1. Check your cloudtrail logs → Filter logs with AWS access key ID or STS token. Use Athena or SIEM tool to query access key usage/traces and also determine level of access.
2. Revoke/Delete access -> Identify AWS account via querying centralized cloudtrail logs; disable Secret/Access keys credentials and remove IAM user. For STS /temporary credentials , revoke session permission. More details → https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_revoke-sessions.html
3. Restore access → Create another IAM user and generate credentials with same IAM permissions. Make sure to check if IAM role can be used and attach to ec2 profile (instead of static credentials).
4. Review AWS account → Check all actions performed by exposed IAM keys in cloudtrail logs during compromised timeframe. Also check other logs such as S3 access logs/ Guardduty findings/ AWS Config logs to correlate with this incident. Delete resources (cross account IAM roles / ec2/reds snapshots etc. created via compromised credentials during that time period.

Ponder over below questions after issue containment:-

• Why it happened ? → Find out the reason. E.g.:- Was it intentional/ disgruntled employee or by mistake. Can we harden our security controls in place.
• When it happened ? → Check exact time about this incident to measure overall impact of this security incident. Extract logs, correlate with timings/logs.
• How did we remediate ? → Document all the steps and update if there is any new findings or log that helped with investigation.
• What was the impact of this exposure ? → It all depends on how long security keys were in compromised state and IAM permissions attached. List all events based on access key ID.

As per best security practices-> Always enable AWS cloudtrail (multi-trail AWS Org level), AWS Config, Guardduty and VPC flow logs — and store logs in centralized S3 location.

AWS Cloudtrail → Stores all API activities in your AWS account.

AWS Config → It’s AWS configuration recorder. Stores all information related to resources and timelines with change history.

Guardduty → Logs can be analyzed to see if any resource is compromised or connected to malacious URL.

VPC Flow logs → You can enable flow-logs at VPC level and collect/analyze traffic pattern using 3rd party / ELK stack during this compromise to assess all aspects. E.g — any anomalies or connectivity from cloud to on-premises env.

Additional Measures :-

• RCA (Root cause analysis) of this incident and document it.
• Use IAAC tools (Infra-as-a-code) to deploy resources in cloud. Also use tfsec/VSCode extension/ gitleaks to detect secrets in plaintext. Add it to your pipeline.
• Scan your terraform backend state file and make sure there is no secret present.
• Rotate IAM (static keys) every 90 days.
• Use IP whitelisting in IAM policy.
• Re-generate IAM (static credentials) when someone leaves the company.
• Adopt CSPM (Cloud security posture management) tools that can detect/send you an alert for anomaly detection based on IP of Access keys usage (UEBA — part of machine learning platform) — Costly but recommended for mission critical services.
• Use only IMDSV2 (metadata service) — IMDSv2 is a more secure version of metadata service, that makes it harder to steal the IAM role from an EC2 unless you have full RCE (remote code execution) on the EC2. By default, all EC2s still allow access to the original metadata service, which means that if an attacker finds an EC2 running a proxy or WAF, or finds and SSRF vulnerability, they likely can steal the IAM role of the EC2.

it’s important to deep dive/analyze all AWS logs during such incidents to rule out all possible signs of compromise E.g. → Cross account IAM role, EBS/RDS snapshots or sharing with 3rd party accounts, S3 access logs etc.

Incident Response Playbook from AWS (Compromised IAM Credentials) → https://github.com/aws-samples/aws-customer-playbook-framework/blob/main/docs/Compromised_IAM_Credentials.md