Overview
I started with finding new sensors and actuator I hadn't used before. I then brainstormed about what new interactions would be possible with these techniques. While building prototypes, new ideas came up. Documenting my experiments and sharing them with people in other departments paid off when I teamed up with some people that got inspired: we won a hackathon with a new invention.
Prototypes
Haptic motor tests
When looking through research papers to get inspired for new methods of interaction, I found the following: If two actuators are placed 6 cm apart and have overlapping vibration time, they can be perceived as one sensation that moves over the body in a certain direction. This haptic illusion can be used to give feedback about direction of movements. That sounded super interesting, so I got to work.
I did some tests with vibration motors on different places on the arms and legs, and different amounts of motors. I wrote some simple Arduino code that buzzes the vibration sensors in sequence with some overlap, with 2 variables that I could change: vibration strength and delay between each step in the vibration sequence. I also tested it on two participants to check if they could determine what the direction of the simulated movement was.
Findings after the tests
-
A spacing between the motors of around 6 cm seems best, just as the research paper suggests. A spacing of 4 cm was still working okay.
-
Using 4 motors gives a way better illusion of movement than 3 motors.
-
The delay between activating the motors has a big impact on making the movement feel real. There’s a sweet spot between the movement being so fast that you can’t perceive it well, and the movement being to slow and feeling “choppy” because the vibrations of the separate motors don’t seem to mix together anymore.
-
It seems that the delay between each motor can be lower when using more motors. Probably because then the duration of the complete movement stays the same.
-
Higher strength of vibration makes the feeling of movement clearer. Though I suspect that at a certain point vibrations could be too strong and interfere with each other so the movement becomes unclearer again.
-
Vibrations seem to be more clearly felt on harder parts of the body, like bones and less clear on softer tissue. This is probably due to the softer tissue dampening the vibrations.
-
I tested with 4 motors on the lower arm and 4 motors around the wrist on two participants and they could clearly tell the direction of movement (brushing over the arm towards the body & twisting around the wrist)
Hot/cold element
With a Peltier module we can electrically heat up and cool down a surface. It's an element that transfers heat from one side of the element to the other side when powered on, so it has a hot side and a cold side. This might be a way to provide feedback in situations where haptic or other feedback would not work so well, for example for swimmers. It could also be used in a smartwatch for navigating, by feeling if you're getting 'warmer' or 'colder', or it could be used in VR applications.
I first connected the Peltier module to an Arduino and tried controlling it using a MOSFET. The power to the Peltier module was hard to control, as it can draw a lot of power and heat up very quickly. As a safety precaution I connected the Peltier module directly to the variable (3V-12V) power supply with a push button in between. This way I could manually switch the power on and off. I also tried putting a potmeter in series to be able to control the current through the element. This seemed like a good idea at first, but of course the excess power (or "heat") would need to go somewhere and it immediately burned the potmeter, resulting in some nice smoke effects. I measured about 2 amps of power through the module when the power supply was set to 12V. I set it back to 3V (the lowest value) and then measured 1 amp through the module. Luckily the power supply didn't burn up, as it's only rated for 1 amp of power. In the end I kept the circuit simple: 3V of power from the power supply, a push button in series, and the Peltier module.
I tested pushing the push button for some seconds while holding the Peltier module between my fingers, to feel how quickly and how much it heats up and cools down and to experience this way of feedback that we could give. I first tested without any heat sink, so just the bare module. I did a second test with the module placed against the metal of my laptop, so the metal could act as a heat sink.
Findings after the tests
-
The Peltier module responds very quickly. Quick enough to give immediate feedback to a user. The speed can be improved a lot by adding a heat sink to the element.
-
The module also returns to its base temperature quickly, this makes it possible to give a sensation of 'heat pulses'.
-
The effect of heating and cooling is very noticeable for a user. It's an unconventional way of providing feedback, but it works and might be useful in some use cases.
-
The module should be handled with care, 3V is more than enough voltage and the element should not be turned on for more than 5-7 seconds as it can burn you. According to the manual it can heat up to around 135 degrees celsius. When used on the body there should be very strict safety limits put in place.
-
The module can also draw quite some power, this should be taken into account when looking at safety requirements and has an effect on battery life if used in a portable device.
Wireless prototype
To make prototyping interactions on the body easier, I’ve built a wireless prototyping platform. This prototype is smaller and connects to a tablet or phone using bluetooth. It can then be controlled by sending data through a bluetooth terminal app (I used Serial Bluetooth Terminal). This data gets sent to the Arduino using a direct serial connection and gets processed there. This way you can press a button on your phone and make something happen on the prototype. It gets its power from a power bank so you’re completely free of wires.
Some improvements over the normal prototypes:
-
Arduino Nano instead of Arduino Uno, to make it more compact
-
Bluetooth module (DSD Tech HM-10) to make bluetooth connection possible, so we can control the prototype from a phone or tablet
-
Smaller breadboards, these can be clicked together so we could make separate modules on each board
-
Flat wiring instead of regular breadboard wires, to make the prototype more compact and sturdier under movement
-
Power bank to support the prototype without connection to a laptop
Later a case could be added so everything is contained into one small package.
Testing the prototyping platform
Tests with LED light activated by a tablet over bluetooth
I’ve attached the bluetooth module to the serial port of the Arduino. The bluetooth module receives the commands to turn the LED on (‘a’) or off (‘aa’) from the connected tablet. It then sends them over the serial connection to the Arduino and the Arduino turns the LED on pin 3 on and off. There’s a slight delay, but not much. I guess it’s about 250 ms.
Tests with vibration module being activated by a tablet over bluetooth
For this test I attached another mini breadboard to the prototype, with a vibration (haptic) motor plugged in. I call this the vibration module. I attached it to pin 12 on the Arduino and made the LED light up when the vibration activates. The prototype can now vibrate and stop vibrating by pressing buttons on the tablet.
Test with multiple haptic motors attached, using different vibration patterns
I’ve connected 4 haptic motors to the vibration module breadboard. Then I added a function for triggering different vibration patterns and different speeds across the motors. These functions can be triggered through sending specific commands over the bluetooth connection. These commands are put behind custom buttons in the bluetooth terminal app so there’s an easy UI to quickly trigger different patterns. I had to do some filtering on the bluetooth connection to filter out noise, but in the end it worked great. I’ve tested with the power bank instead of power from the laptop and this works fine. The motors were also attached to the inside of a strap and tested around the wrist. This works well for attaching motors to the wrist, but the motors are not tight enough against the skin; this makes the vibrations hard to feel and makes the direction of movement unclear.
Conductive thread
Conductive threads can be used to add a capacitive touch interface to clothing. This interface could then be used to remotely control software or devices without having to use a screen. Conductive threads can we sewn into the fabric.
I tested if a single conductive thread worked as a capacitive touch interface. I also tested:
-
Using it as a touch slider along the length of the thread
-
Washing the thread with water & soap
-
Using the thread in a certain shape, vs a straight line
-
Differentiating between different touch inputs on a single thread
Findings after the tests
-
The main problem that I ran into was interference. Even if the threads don’t touch, the wires running to the thread can still interfere with each other. This is probably due to their electrical fields being too close to each other and changing the capacitance. I tried insulating the wires, but it was not possible to insulate the complete circuit in this setup. It probably helps to put the capacitive sensing IC close to the actual conductive thread, so there’s a shorter length of wires running to towards the thread.
-
Washing the thread did not affect the sensing capabilities.
-
When detecting different amounts of capacitance between different parts of the thread (for example when trying to create a slider or buttons), the interface should be calibrated. This is because the output is dependent on how conductive your body itself is, and that can change based on things like the humidity in the air, if you’re sweating, the surface you’re standing on, etc. A solution might be to place a second wire inside the clothing that touches the skin, as to always have a baseline with which the interface’s capacitance can be compared.
-
An interesting thing that I found was that the thread is less sensitive than an aluminum foil capacitive interface. It mostly just works if it’s directly touched. This is a good thing, as it prevents accidental touches.
Stretch sensor
Using a stretch sensor we can measure the amount of stretch in a strap. This might be useful to measure increase in muscle size for example, or for measuring muscle activation. That data might be useful in physiotherapy, rehabilitation or personal training. The sensor could also be placed in a fitness tool like rubber resistance bands, instead of wearing it on the body. This way progress could be measured by showing how hard a user can stretch the bands. Or it could be used inside your t-shirt to alarm you when your belly gets too fat ;)
I've attached a stretch sensor to the Arduino and added a simple voltage divider using two 10K resistors. The Arduino reads the analog input pin to check what voltage is coming through, converts this to a rough "percentage" value (determined by testing, sometimes it can exceed 100%) and sends out this through the serial port. Then I've made a Processing program that reads the incoming data from the serial port that the Arduino is connected to, and visualises this data into a coloured circle that grows when the sensor is stretched.
Findings after the tests
-
The sensor is a cheap and simple way to get body measurements, and it's probably quite easy to integrate into clothing.
-
Using the sensor to measure muscle tension would be easier than to measure muscle gain. For measuring muscle gain you need to establish a baseline that stays the same while putting the strap on multiple times. It might also not be sensitive enough to measure a small gain, but that could probably be solved by doing more windings around the arm/leg.
-
The sensor should be insulated so it doesn't short-circuit with itself and mess up the sensor values.
-
It's important that the sensor has sufficient length. This can be achieved by doing multiple windings around the muscle that you would want to measure.
Pressure band
Instead of using vibrations for haptic feedback, it might be more clear for users to use pressure. We can use a servo motor as a linear actuator to push directly onto the skin. I attached a servo motor with "rack and pinion" attachment to Arduino. The rack and pinion attachment transforms the rotational movement of the servo into a linear movement. I wrote some code for making the servo push in different ways. This servo was placed in a strap so it could be worn on the arm. I then tested the different forms of feedback.
Findings after the tests
-
The servo moves quite quickly without much delay, so it's possible to give multiple quick pokes
-
Pushing a small distance (10-20 degrees in this case, about 0.5-1 cm) works best for poking people without putting too much pressure on the device itself.
-
The feedback by pushing onto the skin feels more clear and less annoying than vibrations, it’s more similar to someone lightly poking your arm.
I encountered two challenges with this approach:
-
The pushing unit should be quite strong as to not break the gears when poking while pushed against skin. I used a micro servo and it feels as if it can break quite easily when pushed firmly against my arm. The servo uses small plastic gears and doesn't have much power, but this servo is still way too big to incorporate into a wearable. For use in a real product, the servo motor should be a lot smaller and also stronger.
-
The servo can make the band go up instead of pushing into the skin, so the band should be fastened pretty tightly
Foot progression angle correction using haptics
This prototype was made with our team during a hackathon. It's a combination of FPA measurement using a motion sensor on the shoe, and corrective haptic feedback using a strap around the shin.
Concept
The angle of your foot relative to the direction that you're walking is called the foot progression angle. There's an optimal angle that you should have, and deviating too much from this angle can give problems and pain in the knee. People that walk with their foot twisted too much to the inside or outside can get a retraining from a physiotherapist. We've build a prototype of a haptic strap that gives correctional feedback to a user wearing it, so someone can do gait retraining on the go, without needing supervision of a physiotherapist.
The foot progression angle is measured by a motion sensor that is placed on top of the shoe. The motion sensor is connected to an android app that calculates the angles in real-time and knows how much the current step is deviating from the optimal angle. The app is connected to a haptic feedback strap (through BLE) that is worn around the shin of the same leg. If for example the foot is pointing too much towards the outside, the app will send a message towards the haptic strap that it should tell the user to twist his foot more towards the inside. The haptic strap will then vibrate in a pattern to give the user a twisting sensation around his leg, twisting towards the inside. The user will then naturally correct his foot placement towards the indicated direction.
The user flow is shown below in storyboards:
Haptic strap
For the haptic strap we used 8 haptic motors, arranged in a circle around the shin. The motors were sewed into the strap. Another (larger) strap was placed around the outside of the strap to cover the wires to the haptic motors and to be used as a pocket for the haptic controller.
Haptic controller
At the heart of the haptic controller is a DFRobot Bluno Beetle, an Arduino with integrated BLE. This Bluno receives commands over bluetooth and then activates the haptic motors in specific patterns. The serial monitor is used to read out data for testing.
We connected the Bluno to a Bluno android app for controlling. Bluno didn't work with normal bluetooth terminal applications. The android app can connect to the Bluno over BLE and shows the serial messages from the Arduino.
The Bluno is powered by a power bank. The power bank was separated from its casing so it could fit into the strap. It can be charged by USB through the haptic controller. It has a cockpit switch to turn it on and off. This is the main power switch for the haptic controller. The haptic controller is connected to the haptic strap using a connector, so it can be easily decoupled for testing and flashing new firmware without having to take off the strap each time.
Tests
We tested the interaction with multiple (so far 8) people. Everyone could clearly feel the direction of movement and knew how to correct their step.
We also tested speeding up the movement (70ms between the motors), but this makes the direction unclear. We seem to have found the optimal speed with having 8 motors and a delay of 100ms between them.
Insights
-
The direction of movement is very clear with 8 haptic motors at this speed (always 2 motors activated, 100ms delay between motors)
-
The feedback causes the user to naturally want to move the foot in the right position
-
The strap with controller should be more robust. It now still breaks quite quickly as there's a lot of strain on the cable connections when people put on the strap
-
It's annoying that users have to take off their shoe when putting on the strap. It should be possible to put the strap on without having to take off your shoe first
-
The strap needs to be pretty tight against the skin for the haptics to work correctly, but it's not uncomfortably tight
-
It would be nice if strap auto-connects to the app when it's turned on
-
Some decisions need to be made about how often you want to give feedback. Do you want to measure every step and give feedback at every step or will this become annoying? Does the feedback need to be instant? There's currently a small delay between measurement and feedback. If the feedback needs to be instantly provided at every step, this delay is too big.
.png)
Reflection
Some insights I got from my prototyping experiments:
🏄🏻♂️ Sometimes it doesn't look like your prototypes are leading somewhere, but you gotta trust the process. Keep on chasing your curiosity and at some point it will 'click'.
✏️ Documentation is key. As the great Adam Savage from Mythbusters said: "Remember kids, the only difference between screwing around and science is writing it down". By documenting your experiments they can be reproduced and shared.
🚀 Sharing your experiments and the results will inspire other people. Forming a team with inspired people from different expertises can lead to amazing results. During my experiments I teamed up with people in other departments and this led to a completely new invention.