Authors: Macks Tam, Spencer Tam, and Tian Xia

💻 Source Code: VaultBreaker, UrScrewed

Design Summary

In this project, we designed and built two interactive cell phone unlocking interfaces by leveraging the user’s on-screen rotating gestures (unscrewing the fake apps) and gyroscopic motion (simulated vault break-in) respectively.

Sensory Unlocking - VaultBreaker

We used the cell phone’s built-in gyroscope to simulate turning a lock. This idea is to simulate breaking into a vault or safe to unlock the phone. Firstly, the lock itself turns by any gyroscopic motion. There are three numbers the user has to find in the correct order to unlock the phone. These numbers are indicated by a clicking sound that cannot be heard if the stethoscope on the screen is not clicked. If the stethoscope remains unclicked, there will be no clicking sound or check mark, even if the user turns the phone in full 360 degrees. Once the user taps on the stethoscope, it will appear highlighted on the screen. From here, the user should attempt to slowly turn the lock in one direction until they hear a click indicating the user has hit the first number. A green check mark also appears on the screen. The user must now change directions and tilt the phone in the other direction until they find the next number. When the next number is hit, a second click will be heard and another green check mark will appear on the screen. Finally the user changes directions for the last time and rotates the phone until the third and final click occurs and a check mark appears. The cell phone is then unlocked.

Gestural Unlocking - UrScrewed

The user is presented with a fake screen with seemingly no interaction. Attempting to drag or tap the fake apps will do nothing, this prevents attackers from using the phone. To get past the fake screen, the user has to unscrew the apps placed in each of the four corners of the screen. In our prototype, we have YouTube, Chrome, Settings, and Apps as our four screws. The user has to rotate the “screw” apps in the correct direction (clockwise in our demo). Once the app has been rotated enough times, it falls off. The user must then continue by unscrewing the next app. The order for this matters, and for our prototype, the user must unscrew YouTube, Chrome, Settings, and Apps in the exact order. If the user does not unscrew these apps in the correct order, the screen does not unlock. If the user manages to unscrew all four apps properly and in the correct order, the fake screen falls off revealing the real screen and unlocking the phone.

Design Process

Our design process started with 10 preliminary design sketches from each of the three team members (30 in total). We selected 10 best preliminary designs to present in class and received feedback. We then chose one gesture/multitouch-based idea (Macks’ Multi-Touch) and one sensor based idea (Tian’s GyroLock) as the base designs to develop 10 variants (five for each category) from each member. We later decided not to proceed with Macks’ Multi-Touch idea and switched to Macks’ ScrewLock idea from the refinement. We also tested a variation of Spencer’s Finding Waldo AR game, but we decided to go with Tian’s GyroLock idea in the end. Here we show our selected 10 base ideas and 10 refinement ideas.

Preliminary Design

Final Base Design for Sensory Unlocking - GyroLock (Tian)

We use this idea to simulate a safe lock and fulfill the sensor requirement of the assignment. We decided to use the gyro sensor that most phones today use. The basis of this idea is that the user would spin the phone as if they were trying to unlock a combination lock on a safe. This became the basis of our final sensory idea.

Final Base Design for Gestural Unlocking - Multi-Touch (Macks)

In this design, we use multi-touch in a specified order to unlock the screen. The user would have a preset sum, such as 10, and she/he would then have to tap on the screen with a specified number of fingers in a specific order, which she/he already set. For example, the password can be 10 as the preset number, and the sequence is 1,1,3,2,2,1, which adds up to 10. If an unauthorized user tries to unlock the phone by touching 2,2,2,2,2 or 5,5, it would not unlock the phone. We were planning to use this idea for the gestural design, but after more discussion, we felt the idea was too simple in the creative aspect.

Piano Unlock (Tian)

This idea is based on the multi-touch/gesture requirement, where the user must play a specific tune on the piano to unlock their phone. The reason we decided not to go with this idea is that we felt the creative aspect / wow factor was not as high as we would like.

Rocket Pump (Tian)

This idea is to use the phone’s accelerometer sensor to pump the rocket with a certain amount of accumulated power by shaking the phone repeatedly up and down. In order to unlock the phone the rocket must reach within a specific range. We decided to not go with this idea because of the impreciseness of the sensory data during each shake.

Scream Lock (Tian)

For this idea, we would use the phone’s microphone to unlock the phone with voice input. The user should scream at a certain volume to shatter the glass. The voice input is a close contender for our final design, but we prefer the lock simulation with gyroscope.

Match Lock (Spencer)

This idea is based around AR, using the camera to find the dotted outline of shapes or images. The user then matches the image provided on your screen with the dotted outline. You would have to tilt your phone at certain degrees to get it to match. We decided to pass on this idea as we felt that the level of creativity was too low.

Where’s Waldo (Spencer)

This idea is based around the well-known game, Where’s Waldo. The user would use the camera and motion related sensors to find Waldo in your current room. Once you find Waldo, you click on him; if you are correct, the phone will unlock. We almost went with this idea, but we found that the implementation would not be optimized enough to enable 3D reconstruction in the time we had, as the conventional AR with camera feed does not work; when you spawn an object (Waldo) into AR, it will always appear directly in front of where your camera is facing.

Drag Pattern (Spencer)

This idea is based on the 3x3 lock pattern currently used by most touch-screen phones. The basis of this idea is to use the camera, AR, and a 3x3x3 cube of dots. The user would drag their phone in a preset 3d pattern to unlock their phone. We decided not to go with this idea, as we felt that though it was secure, it was too similar to the already existing lock screen, so the creativity was too low.

Song Arrangement (Macks)

In this idea, the lock screen would display 6 random songs in a random order. The user would then have to swipe up and down to rearrange the songs from best to worst in the correct order. These songs would be taken from a specific list so that only the user would know how to arrange the songs. We did not use this as we felt it was not very creative and was not the most interactive gesture-based project.

Musical Hand Signs (Macks)

This design is based on the user’s music preference. The user should have a preset list of songs she/he likes, and on the lock screen, the phone would play random songs in a random order. In front of the back-facing camera, the user should give a thumbs-up if the song is in the preset list, or a thumbs down if it is not. Once the user has got them all right, the phone unlocks. We passed on this idea because, while the creativity of this idea was high, it is too similar to a previous project we came across.

Design Refinement

Final Sensory Unlocking - VaultBreaker Pro

For this idea, we decided to refine the combination gyroscope lock into a more fun, interactive lock. We made it so that the combination will be random, and the user won’t know the combination. The only way to unlock the phone is to use the stethoscope provided to listen to the clicks. This was to give the user a realistic experience of breaking into a secure vault. We decided to go with this idea as our final idea as we felt the creativity was on a higher level than just a simple combination lock, and using the gyroscope sensor was a fun and interactive way to unlock the phone.

Final Gestural Unlocking - UrScrewed Lock

In this design, the user is presented with a fake and seemingly non-functional app screen. This would ward off attackers because they would assume the phone is not working. In reality, certain apps on the screen are totatable like screws, and once they are rotated in a certain direction for a certain amount of times, the apps would fall off the screen. After unscrewing all the rotatable apps, the fake screen would also fall off revealing the real phone screen and unlocking the phone. We ended up going with this idea as it was more creative and interactive than our original multi-touch/gesture idea.

Gestural Variation - Screw Puzzle

This design centers around a puzzle that the user has to complete to unlock the phone. The user is presented with what looks like a fake lock screen with no way to unlock it. The time on the screen is 10:00, and there is a screwdriver and a hammer in the background. There is a screw in each corner of the screen. The user would have to hold and drag the screwdriver to each of the 4 screws. Once dragged on top of each screw, the screw becomes turnable and once all 4 screws are unscrewed, there would be an animation of the screen hanging off the 0 in the displayed time. You then drag the hammer on top of the 0 and then tap on it. Tapping would make the hammer use the back side and remove it. The screen would then fall and reveal the “1” in “10:00” is a key. The user can drag the key to the center of the screen with a revealed keyhole and unlock the phone. We did not go with this idea because it is cumbersome to go through so many different steps to unlock the phone, though this is a creative puzzle.

Gestural Variation – VaultBreaker Pro Touch Version

This idea is similar to the sensory version, as we were debating whether to implement the lock simulation as sensor- or gesture-based. The idea is the same, but the user would have to spin the lock with her/his finger to hear the clicks. We decided not to go with this approach because this idea is not creative for a gesture-based design but more creative for a sensor-based design.

Gestural Variation – Multi-Lock

This idea is a variation of the multi-touch lock simulation. The user would have to hit the correct number of each lock in the correct order to unlock the phone. The numbers would be preset, so only the owner of the phone would know the correct combination of numbers. We decided not to use this idea; though it was very fun and interactive, we felt the creativity was not as high as the four screws with the fake screen.

Gestural Variation – Beer Bump

This design leverages multi-device interaction and a spatial gesture, holding and bumping beer pint glasses with others. In AR, the user sees a beer pint glass attached to the camera feed in the first person view. The user should find a “beer buddy” and bump that person’s virtual beer to unlock the phone. We did not proceed with this idea because the bumping gesture can be subtle and hard to detect because of forward moving direction, and the unlocking condition is somewhat limited due to its required cross-device interaction.

Hybrid Variation – Gyrolock into Multi-Touch

This idea is to combine both projects into one. We would have the user either crack the safe or enter the correct combination using the gyroscope. Afterwards a vault door would open on the screen revealing a locked phone screen. Clicking on the phone screen would then bring the user to the phone screen itself. Here the user would have to complete the multi-touch (one of 10 base ideas) to unlock the phone. So the user would have to use a gyroscope (sensor) and multi touch (gesture/multitouch) to unlock the phone. We decided against this as it would be very complex to combine both and decided to just make two separate projects. On top of this, we soon decided against doing the multi-touch for our gesture/multi-touch design since it is not as creative.

Sensory Variation – Pouring out Water

This design is a variation of the gyroscope lock, where instead of a lock, the user would have to tilt the phone to pour out all the water to unlock the phone. We ended up not going with this idea, as we felt the gyroscope lock with a stethoscope was a more fun and interactive design.

Sensory Variation – Leaf Blower

For this design iteration, the user should blow on the microphone to remove the leaves from the screen revealing a hidden button to unlock the phone. This button would spawn somewhere random on the screen so the user has to look for it. The leaves would move depending on the volume of the voice input, which depends on how hard the user blows on the microphone. The harder the user blows on the microphone, the more leaves get blown away. We also had a touch-based design where the user should swipe the leaves off the screen. This idea was one of our close runner-ups for our sensory based project. We decided it was not as creative as some of our other sensor-based ideas, and we have noticed another group also had a similar idea of breaking the glass to unlock.

Sensory Variation – GyroLock with Numbers in the Middle

This idea is a variation of the lock simulation using the gyroscope sensor, where the numbers are shown in the middle of the lock. Once the user has got all three correct numbers, the phone unlocks. We decided not to go with this refinement, as we felt that while the idea was creative, it was not as much fun as our stethoscope vault breaking variation.

Project Contribution

• Sketches: Our design process started with 10 preliminary design sketches from each of the three team members (30 in total). We selected 10 best preliminary designs to present in class and received feedback. We then chose three (one from each member) of our preliminary designs as the base design to develop 10 variants from each member. Among the 30 design variants, we selected our two best ones from both categories (gestural and sensory) as the final design decision for system implementation and demo videos.
• Implementation: We all worked together to implement the web application’s functions, including fake apps’ rotation (with the help of a JS open-source library, ZingChart), fake screen’s falling, binding the safe lock’s position to the gyroscope’s sensory data, and the realistic unlocking mechanism of the safe lock. Macks found royalty-free images and audio files on the internet and edited them to fit our project.
• Demo: We brainstormed and filmed our demo video together. Macks recorded a voice-over describing our project design. Tian edited the videos and uploaded them to YouTube.

References

• [Doc] Docs of HTML, CSS, and JavaScript from MDN and W3School
• [Open-Source Library] ZingTouch, a touch gesture library
• [Code] Gyroscope code and JQuery basics code from the course instructor
• [Code] Code for DOM element movement and interval setting and stopping
• [Code] Code for playing audio files (same as P1)
• [Audio] Click sound. Youtube Royalty Free Music Library
• [Image] Combination Lock. iStock
• [Image] Combination Lock. iStock
• [Image] Brushed Aluminium. iStock
• [Image] Check Mark. iStock