On-screen Keyboards Considered Harmful

(aka: Shoulder Surfing: There's an App for that!)

We rarely talk about it these days, but shoulder surfing is a pretty old (but reliable) attack. This is why most password prompts are masked. Many modern mobiles (and tablets) however will highlight keys pressed on the keyboard making old style shoulder surfing attacks trivial (and reasonably automatable) again.

In an effort to (help) bring back the 90's we decided to do some fiddling and built a quick app(on top of the awesome OpenCV framework) to automate shoulder surfing against iPads.
(You can read more about it [here], download a short pdf on it [here] or just watch the youtube video below (but we think the pdf is more fun!))

There are a few more videos (available after the break)

This was an early version of shoulderPad. We assumed we had won by simply locating the blue key-presses.

One of the previewers asked: "Are you deliberately moving your fingers out of the way?"
We decided to answer by quickly typing A-E with normal cadence and normal movement.

All in all.. a fun time was had by all :>


  1. This has been done on the native iPhone keyboard months ago, without requiring any blue key nor that the screen stands still. Both spying camera and victim's iPhone are allowed to move freely.

    Details: http://home.dei.polimi.it/fmaggi/downloads/publications/2010_maggi_volpatto_gasparini_boracchi_zanero_clearshot.pdf

  2. Hi Federico.

    That paper looks awesome. I'll do a short follow up post and make sure to mention it. Interestingly, i dont think it negates searching for the blue key, since your attacks dont cover the password login screen (where the keys do not raise, and the discernable difference _is_ the color)

  3. Haron,

    thanks. However, in the native keyboards (at least on the iPhone) there are no blue keys and, even in the login/password dialogs, keys do raise in a predictable manner.

  4. Federico.

    Ah.. Just spotted the big difference. My tests are against the iPad (with complex password) (blue - no raise), and iPhone (5.x) (blue - no raise).

    Color spotting is also interesting for the android devices, where a pattern is used for unlock (and tracks pretty well).

    Thanks again for commenting.. (and for the link)

  5. Haron,

    AFAIK Android also employs key magnification; also, the iPad keyboard on your video looks different (i.e., black) from the native one. Also, iOS 5 on iPhone has key magnification as well.


  6. Hi Federico

    The android devices ive briefly examined make use of the unlock pattern.
    Not sure why the keypad looks different (nothing altered there).
    Heres a pic of my iPhone4 (iOS 5) with a button being pressed: https://img.skitch.com/20110713-et8a9ekuntuct6gk6b3uk47jyj.png

  7. Haron,

    ah! Now I see your point :) You target the unlock screen, whereas we target the full QWERTY keyboard. Unlock codes have just 4 digits. Is it really worth to automate the whole process?


  8. Fedrico..

    The black keyboard shows up for password unlock screens (the keyboard you are used to seeing is the regular one). (Just checked across iOS and iPad versions).
    My point there remains.. ie.. For that screen, apple is aware its a password, they are masking the text box, they can give us an option to not highlight the keys

  9. (last 2 comments missed each other in flight) :>

    i think we have agreement now :>
    (will still go through your paper properly this weekend.. it looks great)


  10. BTW, here are the video demos of our tool in action: http://www.youtube.com/playlist?list=PL81F91E404B928833

  11. Android devices can have the unlock gesture or a PIN or password... Keyboards can also vary between devices as some use the standard Android one while others use Swype or other manufacturer supplied alternatives.

    You'll see more variety there, but the basic approach would be the same.

  12. where is the download for the app!?

  13. Where is the Downloadlink? ;-)

  14. Awesome to see your site. Keep up the great work. Thanks for all you could do.Lets stay in touch.

    Iphone 4 skal


Post a Comment

Check out some of our other popular posts:

A “Safety Net” for AWS Canarytokens

Sensitive Command Token - So much offense in my defense

Building WireGate: A WireGuard front to detect compromised keys