Disrupting AWS S3 Logging

This post continues the series of highlights from our recent BlackHat USA 2017 talk. An index of all the posts in the series is here.


Introduction

Before today's public clouds, best practice was to store logs separately from the host that generated them. If the host was compromised, the logs stored off it would have a better chance of being preserved.

At a cloud provider like AWS, a storage service within an account holds your activity logs. A sufficiently thorough compromise of an account could very well lead to disrupted logging and heightened pain for IR teams. It's analogous to logs stored on a single compromised machine: once access restrictions to the logs are overcome, logs can be tampered with and removed. In AWS, however, removing and editing logs looks different to wiping logs with rm -rf.

In AWS jargon, the logs originate from a service called CloudTrail. A Trail is created which delivers the current batch of activity logs in a file to a pre-defined S3 bucket at variable intervals. (Logs can take up to 20 mins to be delivered).

CloudTrail logs are often collected in the hope that should a breach be discovered, there will be useful audit trail in the logs. The logs are the only public record of what happened while the attacker had access to an account, and form the basis of most AWS defences. If you haven't enabled them on your account, stop reading now and do your future self a favour.

Prior work

In his blog post, Daniel Grzelak explored several fun consequences of the fact that logs are stored in S3. For example, he showed that when a file lands in an S3 bucket, it triggers an event. A function, or Lambda in AWS terms, can be made to listen for this event and delete logs as soon as they arrive. The logs continue to arrive as normal (except for the logs evaporating on arrival.)

Flow of automatic log deletion

Versions, lambdas and digests

Adding "versioning" to S3 buckets (which keeps older copies of files once they are overwritten) won't help, if an attacker can grant permission to delete the older copies. Versioned buckets do have the option of having versioned items protected from deletion by multi-factor auth ("MFA-delete"). Unfortunately it seems like only the AWS account's root user (as the sole owner all S3 buckets in an account) can configure this, making it less easy to enable in typical setups where root access is tightly limited.

In any case, an empty logs bucket will inevitably raise the alarm when someone comes looking for logs. This leaves the attacker with a pressing question: how do we erase our traces but leave the rest of the logs available and readable? The quick answer is that we can modify the lambda to check every log file and delete any dirty log entries before overwriting them with a sanitised log file.

But a slight twist is needed: when modifying logs, the lambda itself generates more activity which in turn adds more dirty entries to the logs. By adding a unique tag to the names of pieces of the log-sanitiser (such as name of the policies, roles and lambdas), these can be deleted like any other dirty log entries so that the log-sanitiser eats it's own trail. In this code snippet, any role, lambda or policy that includes thinkst_6ae655cf will be kept out of the logs.

That would seem to present a complete solution, except that AWS Cloudtrail also offers log validation (aimed specifically at mitigating silent changes to logs after delivery). At regular intervals, the log trail delivers a (signed) digest file that attests to the contents of all the log files delivered in the past interval. If a log file covered by the digest changes, that digest file validation fails.

A slew of digest files

At first glance this stops our modification attack in its tracks; our lambda modified the log after delivery, but the digest was computed on the contents prior to our changes. So the contents and the digest won't match.

Also covered by each digest file, is the previous digest file. This creates a chain of log validation starting at the present and going back up the chain into the past. If the previous digest file has been modified or is missing, the next digest file validation will fail (but subsequent digests will be valid.) The intent behind this is clear: log tampering should show that AWS command line log validation shows an error.

Chain of digests and files they cover
Contents of a digest file



It would seem that one option is to simply remove digest files, but S3 protects them and prevents deletion of files that are part of an unbroken digest chain.

There's an important caveat to be aware of though: when log validation is stopped and started on a Trail (as opposed to stopping and starting the logging itself), the log validation chain is broken in an interesting way. The next digest file that is delivered doesn't refer to previous digest file since validation was stopped and started. Instead, the next digest file references null as its previous file, as if it's a new digest chain starting afresh.

Digest file (red) that can be deleted following a stop-start
In the diagram above, after the log files in red were altered, log validation was stopped and started. This broke the link between digest 1 and digest 2.

Altered logs, successful validation

We said that S3 prevented digest file deletion on unbroken chains. However, older digest files can be removed so long as no other file refers to them. That means we can delete digest 1, then delete digest 0.

What this means is that on the previous log validation chain, we can now delete the latest digest entry file without failing any digest log validation. The log validation will start at the most recent chain, and move back up. When the validation encounters the first item on the previous chain, it simply moves on to the latest available item of the previous chain. (There may be a note about no log files being delivered for a period, but this is the same message that arrives when no log files are delivered as well.)

No complaints validity complaints about missing digest files

And now?

It's easy to imagine that log validation is simply included in automated system health-checks; so long as it doesn't fail, no one will be verifying logs.  Until they're needed, of course, at which point the logs could have been changed without validation producing an error condition.

This attack signature is: validation was stopped and started (rather than logging being stopped and started). It underscores the importance of alerting on CloudTrail updates, even if it doesn't stop logging. (One way would be to alert on UpdateTrail events using the AWS CloudWatch service.) A single validation stop and start event, means it is not a safe to assume that the AWS CLI tool reporting that all logs validate means that the logs haven't been tampered with. The log validation should be especially suspect if there are breaks in the digest validation chain, which would have to be manually verified.

Much like in the case of logs stored on a single compromised host, logs should be interpreted with care when we are dealing with compromised AWS accounts that had the power to alter them..

27 comments :

  1. Excellent information on your Article, thank you for taking the time to share with us such a nice article. Amazing insight you have on this, it's nice to find a website that details so much information about different artists.
    assignment writing service

    ReplyDelete
  2. Are you searching for an online assignment help service provider? Contact AllAssignmentHelp provides you in your assignment writing with the best price. We offer the best assignment help to make sure you get A+ grade in your assignment within a short time period.
    Assignment Writing Service
    Assignment Help
    do my assignment

    ReplyDelete
  3. Epson printer is one of the greatest printers for commercial establishments and has earned numbers of positive feedback from the users just because of its reliability and utmost features. The users can print the document with excellent quality and clarity into the earlier time. Let’s get together with our talented and experienced engineers who will reliably support you for Epson Printer Setup. They have an idea to deal with your nasty problem without any hassle.

    ReplyDelete
  4. Assignment Help is the good and reliable source of completing your work on time by spending minimum effort. All students who need help for raising their marks, they can connect with academic experts of greatassignmenthelp Don’t spend your time and place your order right now.
    Assignment Help Online
    Online Assignment Help
    Assignment Help Online Services
    Assignment Helper
    Assignment Assistance
    Assignment Help Experts
    Online Assignment Help Services

    ReplyDelete
  5. Nice Blog, I have get enough information from your blog and I appreciate your way of writing.
    Thanks for everything and dial our Outlook support phone number for technical support and information of Outlook.
    How do I contact Microsoft tech support?”
    How do I contact Microsoft Outlook support
    Is there a phone number for Outlook support

    ReplyDelete
  6. When students have the burden of their homework writing then they can take help from our homework helpers in the UK. They provide the highest quality homework help UK to the students of the UK in their budget.

    ReplyDelete
  7. We provide the assignment at the cheapest price.24/7 hours Allassignmenthelp is always ready for you in times of emergency. Proofreading and editing service

    ReplyDelete
  8. Wow! Thanks for sharing your ideas. Every person does not have the in-depth knowledge how to deal certain critical problem in quicken account. If you are one of them to accept the massive rage of technical failure and unable to do different business works, then you must take the full association of our technical team. Our technical team tells you full information how to get the rid of annoying pitfalls. Once you take the full association of our team, you do not come across with innumerable failures. Feel free to contact us our team. Dial Quicken Support Number.

    ReplyDelete
  9. We are a prime SEO services provider in USA, offering innovative, unique, and affordable SEO Services USA as per client’s needs and goals. Our customized SEO solutions give you better local search results.

    ReplyDelete
  10. Thank you for share this article. This article is really informative. Using this article we can easily fix our outlook profile issues. The writing style is really nice so that we can easily understand the details. It's an informative article. I really like it. Get support click here Outlook Technical Support Phone Number
    Microsoft Support Number
    Microsoft Office 365 Support

    ReplyDelete
  11. Whenever your Windows gives you one error or the countless, you just need to grab Windows support. The available professionals of this support are well-educated and always ready to give their best. I am telling you all this because I have experienced all of these. So, when on Earth you need to resolve your issues associated with Windows, then I personally suggest you to make connection with the Microsoft windows support team’s professionals.
    Website:- https://www.microsoftoutlookoffice.com/windows-support-number
    office Support
    office Help
    contact microsoft
    How do I contact Microsoft Office Support?
    office 365 Support
    office 365 Support
    office 365 help
    office 365 help desk
    Microsoft online support
    Microsoft office 365 support
    Microsoft 365 support
    Microsoft office 365 Help
    Microsoft 365 Help
    outlook keeps asking for password

    outlook Support Number
    outlook Support Phone Number
    Microsoft outlook Support
    outlook Support
    Change Hotmail Password
    hotmail change password
    hotmail forgot password
    hotmail password reset
    how to change hotmail password
    microsoft password reset
    microsoft reset password
    reset microsoft password
    Forgot Outlook Password

    ReplyDelete
  12. If your HP Printer Not printing Black, it means that your HP printer is passing through serious technical faults. This issue is very serious technical error that cannot be resolved simply. Our printer experts have technical skills to fix this error instantly.

    ReplyDelete
  13. HP devices are very reliable and have numerous features to complete the task easily. You should get in touch with HP Support to get quick response from experts to resolve HP device related query.

    ReplyDelete
  14. Very nice!!! This is really good blog information thanks for sharing. We are a reliable third party QuickBooks Error Code 80029c4a company offering technical support for various any types of technical errors.

    ReplyDelete
  15. If anyone has his/her website hosting like AWS compromised, then this article is absolutely for them , they would find a solution to make their website hosting more secure, it helped me to strengthen hosting of my digital marketing agency website.

    ReplyDelete
  16. When you give a command to your Epson printer, suddenly your Epson printer is showing offline issue. Epson printer offline error is one of the most irritating issues for Epson printer users. To solve it, you need to take the best technical resolutions. We work closely as an independent third party technical support service provider, offering online technical support services for Epson printer users in very nominal charges. Our printer experts have technical skills and extensive experience for solving Epson printer offline error. Our techies are easily available round the clock to solve Epson printer offline error.
    Epson Printer Error Code 0x97

    Epson Error Code 0x97

    Epson windows service disabled error

    Epson error code 0xf1

    Epson wireless printer setup

    Epson printer offline

    Epson Printer in error state

    Epson Support

    Epson Printer Support
    Epson Printer Not Printing
    Epson Printer Setup
    Epson 0x97 fix patch

    ReplyDelete
  17. Nice Post..
    HP Printers top the preference list of every customer who is on a printer hunt. People from all over the world favor HP Printers over any other printer brands due to their magnificent features. With the help of HP Status monitor, you can see printer’s status and can detect printer error with no trouble. Therefore, if you are facing any error within your HP printer in Windows 10, then you can resolve all of them easily. You just need to follow the stated instructions on www.hp.com/support as these are very helpful in resolving several printer related issues.Otherwise you can also put a call on help line number to make direct contact with skilled professionals for resolving this

    ReplyDelete
  18. This is the very first time I called on Outlook Support Phone Number in order to get rid of my outlook issues. The technician who attends my call is very helpful and friendly in nature. He guided me step by step to fix my entire issues. So, I would recommend to everyone that whenever you face outlook issues, then dial this number. Get support click here
    Microsoft Office 365 Support
    Microsoft Support

    ReplyDelete
  19. Does your Epson printer have malfunction? In this situation, you can go to approach for www.epson.com/support to look out the expected change in it. Definitely, you have approach on the right cover page if certain technical difficulties arise in it. No need to do any experiment while you are facing any technical flaws. Doing the futile experiment brings some demolish effect in Epson printer which does not let you to get the positive result. Taking the association of Epson.com/support representative does not allow to get overcome from technical glitches, but also it gives the valuable support to know the wide information about the Epson printer utility as well.

    ReplyDelete
  20. To bring your HP printer setup, you need to give a click at 123.hp/setup by opening a web browser on your computer system. The effectual and hands-free solution is there, you only have to follow the guidelines as mentioned on that page. Once you reach to the bottom of the page, you will see your HP printer has been setup. Be careful while downloading and installing printer driver and software as during this setting up process in case the network gets lost, your printer won’t setup properly and might create some problems during print job.

    ReplyDelete
  21. Setup Wireless Printer easily. Go through step by step direction in blog for your Printer Setup. If you still face difficulty,you can connect with advanced printer setup experts for expert assistance for “How to Setup Network Printer”.

    ReplyDelete
  22. Roadrunner email application is renowned software that offers incredible user experience. Though it is very easy to setup roadrunner email, but for a layman it can be troublesome. Therefore, if you are encountering issues in roadrunner email settings, just dial our toll-free number +1-833-441-9444. Our team of experts has in-depth knowledge about Roadrunner software and will guide you personally in step-by-step manner.

    ReplyDelete
  23. Get the best internet bundles from CenturyLink, Windstream, Viasat and Frontier

    ReplyDelete
  24. Learn How To Connect Canon Printer To Wi-Fi in a proficient manner
    Canon has made its presence in the international market for offering excellent quality printers, both for household as well as official use. Most of the new models of Canon Printer are built with the Wi-Fi compatibility. By using these Canon printers in your working premises, you can print outstanding quality documents from a computing device or even a Smartphone by just giving a command. In this, you don’t even need to set up any wired connection to connect your canon printer with computer. You must have the proper knowledge to Connect Canon Printer To Wi-Fi. The steps are very simple and you can get them by dialing given helpline number and taking assistance from ingenious professionals.

    ReplyDelete