Aftertouch (example)

“Aftertouch” is pressure sensitivity on an electronic synthesizer, allowing the musician to change the tone or sound of a note after it is struck [Wiktionary]. For example, it could allow a musician to slightly bend the tone of a note or slowly add vibrato to a tone, by simply pressing a little harder while holding down the key to sustain the note.

(I can not confirm that the photos used for this example are from an aftertouch-enabled keyboard. I am using these photos for demonstration purposes only.)

Keyboard Aftertouch

The top layer of a keyboard is the obvious and most visible layer: the tone-producing key “button”. When a key is pressed, a tone is produced. Depending how advanced the keyboard is, would determine how much control the musician can have on producing the tone. For example, on more technologically advanced keyboards (and usually much more expensive), a musician could produce a tone of varied loudnesses, or varied percussive attacks of the tone. Versus, a cheaper keyboard that produces the same loudness and attack no matter how differently you press the key.

All of these elements pertain to the tone-producing layer of interaction on a keyboard.

Keyboard Aftertouch: button level

The “aftertouch” layer of interaction occurs only after a tone is produced in the upper layer. Presumably there are sensors underneath the key that determine when it is pressed to produce a tone, and also sense when slightly more pressure is given to the press to activate aftertouch. Aftertouch can only be activated once a tone is generated. This fact breaks one of the rules of determining a SUI: interface independence. The upper layer of interaction must be used in order to get to the lower layer of interaction.

Keyboard Aftertouch: touch level

Aftertouch is not a SUI for multiple reasons:

  • Multiple interfaces? The upper layer produces a tone, while the lower layer augments the tone. Both are related and interfacing with the tone generating interface of the keyboard. Are opposed interfaces necessary for a SUI?
  • Both interfaces described here occupy the same physical space, but it is unclear whether the interfaces described here are separate and plural.
  • There is no interface independence. Aftertouch can not be activated without first interacting with the upper layer of interaction.

Keyboard Aftertouch: SUI levels

Day 120

Only 5 weeks left in the thesis and only 1 week until I hand in my midway progress!

This week’s activities:

  • With 2 cubes now complete, I am delivering them to local designers to test their interaction and potential future use.
  • Working on interviewing techniques and questions that I will be asking. I’m video recording the interviews to be used for videos to support my final paper.
  • Writing initial drafts for final paper…

After midway critique:

  • Traveling to Stockholm and Århus for additional testing, along with testing more local designers and “people on the street”.
  • More writing and editing video…

Some photos from recent prototype building. In case you are wondering, I ran cords out of my 4th floor apartment into the courtyard so the apartment wouldn’t smell like burnt wood/metal/plastic.


Cords out the window.

All pieces inside each cube.

Cubes drying after being glued.

Circuit board with touch sensor and LEDs.

Final assembly of internals (button stand and cube in background).

iPod (example)

The original iPod, now labeled “iPod Classic”, is an example of a simplified SUI interface.

iPod Classic

Using your finger, you can touch the jog wheel and move your finger in a circular motion. This interaction modifies a number of controls on the screen.

  • It scrolls whichever list is visible on the screen. A clockwise motion scrolls down, while a counter-clockwise motion scrolls up.
  • If a horizontal status bar is visible, it can adjust the current position in the status bar to the left or right (clockwise for right, counter-clockwise for left). This is used for adjusting the volume and adjusting the time position of the song.
  • If the star rating is visible, scrolling to the left or right with the jog wheel selects more or less stars.
  • To use the Genius playlist control, scrolling the jog wheel to the right moves a selection arrow control, that initializes the Genius feature.

The jog wheel is typically used as the selection control for the iPod. The device does not have any type of pointing device to make selections, as someone is typically used to using on a desktop computer for example. Using the jog wheel as a selection interface is more apparent when trying to play Solitaire on the small device. Moving your finger to the left or right on the jog wheel moves a “selection hand” on the screen. The hand moves in a linear pattern allowing you to select a card to move in the game. This selection interaction on a desktop computer is strictly given to the mouse or trackpad to provide input.

iPod Classic: touch level

Underneath the jog wheel are 4 buttons that provide more input to the iPod interaction.

  • The top button (MENU) is typically used as a “back button”.
  • The right button is used to advance to the next song, or when held down, used to advance the current song by a few seconds.
  • The left button is used to go to the beginning of the current song, go back one song, or go back in time in the current song.
  • The bottom button is used play or pause the song.

The middle button that is not underneath the jog wheel, and thus not included in the SUI interface, is used to advance through the menus or commit to an action.

iPod Classic: button level

The iPod Classic represents a SUI interface:

  1. Multiple interfaces
    1. Jog Wheel controls menu selection (scrolling through lists), and parameter adjustments (changing volume, place in song, star rating).
    2. 3 of 4 buttons control music playing
    3. MENU button function is related to menu selection and not music playing.
  2. Interfaces occupy the same physical space.
  3. Each interface can be independently controlled without interfering with the other interface.

The iPod’s adherence to the SUI framework is weak for rule #1, though: there is not a strict separation of interface functions.

  • One button in the lower layer is used in conjunction with the jog wheel in the upper layer.
  • The jog wheel can be used to adjust a music playing parameter.

iPod Classic: SUI levels

Finished SUI Cube

The first cube is finished!

Cube Construction Progress #3

Time is quickly passing, so I’m taking a short break from button coding, and progressing on the cube construction.

My initial sketches, based on component sizes, showed a cube that would need to be approximately 7 cm square. I would then have enough room for an Arduino Pro board, 3-AA batteries, QT110 chip, RGB LEDs, and enough open air that the LEDs would be able to spread the light in the cube. I had a prototype cube laser cut at 1scale1 to use while figuring out component placements and PCB board sizes. Once all the pieces for inside the cube are determined, I will have 3 final cubes laser cut, with all the necessary holes for switches cut out as well.

Hightlights:

  • How to diffuse an LED” instructables article, showing how to sand the outside of an LED for better diffusion. I’m using this method because it doesn’t decrease the brightness too much while still nicely mixing the RGB LED.
  • Turkish Delight boxes make a great way of organizing all the pieces I need for each cube!
  • My original intent was to etch my own PCB design to speed up the cube construction process. I’m not currently familiar with the Eagle PCB design software, and it is shorribly difficult to learn, so I am opting to just use simple “swim lane” prototyping boards.

Almost all of this will go in the cube.

Proto board will be attached to top of Arduino.

Small board to capture "button press" state.

Sanding RGB LED for better color mixing.

Inside cube, with proto board, Arduino, and battery pack.

Inside cube with button in place.

Cube with all components.

Sketch for layout of components and circuits.

Collecting components for each cube.

Button Coding Progress #2

Added functionality:

  • button double-press (hold red light on)
  • button press hold + touch tap (switches rainbow color)
  • button press hold + touch hold (cross fades between rainbow colors)


Button Coding Progress #1

Building the “SUI cube” prototype is progressing quickly and with only some pain…

My initial electronics design made use of a very small code library that would allow me to sense the touch of the button. This method did not require any additional hardware and used only a small amount of code to detect the touch. (See photo above) It is a great piece of code and only requires connecting a single wire to a single digital pin on the Arduino.

However, this method did not work for the level of control that I need for my cube. My basic debugging showed that the touch is being consistently detected, but the detection cycles quickly between “touch” and “no touch”. For my cube, I need to be able to detect when a finger is touching the button for more than 1.5 seconds, an action I’m calling a “hold”. This “no hardware touch detection” would cycle between “touch” and “no touch” so quickly when my finger was “holding” on the button, that I wasn’t able to accurately interpret the “hold” action.

I instead moved towards a “hardware required touch detection”… (see video)

The button touch is now being detected by the Quantum QT110 QTouch Sensor IC (chip). Only a simple setup is required, connecting the chip to power, ground, a capacitor to filter out noise, a wire that will be touched, and one digital pin on the Arduino to receive the HIGH/LOW output signal. The good thing about using the chip, is that it does the touch processing for me and outputs a simple HIGH/LOW signal to the Arduino, and also allows me to detect the “hold” action! Integrating this chip to the electronics was almost too easy. The only custom wiring I did was to lengthen the “auto timeout” from 10 seconds to 60 seconds. I changed one wire from power to ground for one pin on the chip to achieve this.

Functionality so far:

  • button press (red light on)
  • tap button touch (blue light on, repeats number of taps)
  • hold button touch (blue light, cycles brightness up and down)

Touch Sensitive Button

…with conductive thread!

I never thought I would hear myself say that! Thanks to my classmate, Katrina Anderson, for the perfect idea of sewing conductive thread on top of a monome.org style rubber button.

Many updates to catch up on…

I have decided on a direction for the final prototype: a small cube with a single touch-sensitive button, that abstracts the idea of a SUI, with playful and contemplative interactions. The first step, of course, is to create this cube.

My plan is to laser cut plexi for the cube, use a monome.org style rubber button, make the top of the button touch-sensitive, include some AA batteries, throw in an Arduino Pro board and maybe an RGB LED… and POW!… a “SUI Cube!”

3rd Test of making the rubber button touch sensitive… (the 1st and 2nd attempts were with variations of using a QT110 touch sensing chip or a bare wire, both of which failed)

TRKBRD (example)

I place the TRKBRD design as the “ground zero” example of Stacked User Inputs (SUI). This prototype sparked the SUI for me, and inspired me to investigate it more for my final thesis.

The hardware design of the TRKBRD places an invisible light field over the keyboard, thus creating a “touch field” on top of the entire keyboard.


TRKBRD Top View



The purple area below is the “button level” of the SUI, where a user can interact with the keyboard input.


TRKBRD Top View: button level



The orange area is the “touch level” of the SUI, where a user can interact with the cursor and control the cursor position and action.


TRKBRD Top View: touch level




TRKBRD Side View


Physical button level of the SUI.


TRKBRD Side View: button level


Invisible touch field level of the SUI.


TRKBRD Side View: touch level


Both levels, creating two independent levels of interaction, controlling two independent inputs for the portable computer.


TRKBRD Side View: SUI levels


Arduino Touch Sensing

Currently researching “touch sensing” using the Arduino.

I know of capacitive touch chips that are available, for example, at Sparkfun. But why do I need an additional chip for sensing touch, when apparently I can do the same sensing with a piece of wire connected to the arduino? See video…


Hardware setup and Arduino code available in Arduino Playground: Native Capacitive Sensors without additional Hardware

THIS THESIS TOOK 254 DAYS