Blog  |  Publications  |  Puzzles Fun & Games  |  About

Visual Harmony

This weekend I’ve been playing, once again, with the ideas of experimental film pioneer John Whitney, using both graphics and audio. While Whitney was interested in turning musical ideas into motion graphics, I’m doing the inverse — turning one of his key animation ideas back into music.

Whitney made a number of films based around the simple idea of harmonic relationships. Above is a visual example of one his ideas that I implemented in Flash.

An early intuition about how to control total dynamics led me to activate all graphic elements through a motion function that advances each element differentially. For example, if one element were set to move at a given rate, the next element might be moved at two times that rate. Then the third would move at three times that rate and so on. Each element would move at a different rate and in a different direction within the field of action. So long as all elements obey a rule of direction and rate, and none drifts aimlessly or randomly, then pattern configurations form and reform. This is harmonic resonance, and it echoes musical harmony, stated in explicit terms. I tried this procedure in several films, and was gratified by the consistency of the confirmation it demonstrated.
– John Whitney, “Digital Harmony“, pp. 38

In this movie, each of the 48 dots is moving in a circle. Each of the dots is on a 3 minute cycle. At the end of 3 minutes, the outermost dot will have moved around the circle once (this dot represents the first harmonic or fundamental). The next dot will have moved around the circle twice (representing the second harmonic). The next dot three times, and so on. The innermost dot moves around the circle 48 times.

So, I had a cool idea: Imagine these dots are raised bumps on a disc which is controlling a music box, with each bump triggering a note when it passes the zero degree line (a line extending from the center to the east). The result would look and sound something like this three minute looping movie, which I built in Flash:

Whitney Music Box (chromatic version)

In this particular implementation, the music box has the bars for a 48-note chromatic scale. The pattern causes chords, ascending and descending runs and melodies to be heard.

You may notice some interesting links between the visuals and the audio, especially if you are a musician. For example, when the pattern forms a 3-arm starfish, the chords you are hearing are diminished chords, which consist of minor thirds, an interval in which the notes are 3 chromatic steps apart. The chords you hear always bear this type of relationship to the pattern you are seeing, consisting of intervals which match the arrangement of arms.

I generated the audio using my sound synthesis program, Syd, which provides a very elegant way to accomplish this kind of thing. Unfortunately, Syd is horribly supported these days (I haven’t updated the Macintosh version since before OS X). Here’s a sample Syd patch which was used to generate the audio (I actually used 3 similar patches and mixed the results together).

Update:

Here’s are some other variations. The first few assign pitches based on the harmonic series, rather than the chromatic scale (thus the dot that represents the 10th harmonic actually sounds like the 10th harmonic). In these variations, the “chords” sound like single notes with a buzzier timbre, because when you combine harmonics, you are performing a kind of additive synthesis.

Whitney Music Box (harmonic version – fundamental = 55hz)

Whitney Music Box (harmonic version – fundamental = 20hz)

The above variation uses a lower fundamental which is close to the limit of human pitch perception.

Whitney Music Box (harmonic version – fundamental = 48hz, reversed)

The above variation reverses the assignment of the pitches, so that the lower fundamentals are heard more frequently.

Whitney Music Box (harmonic version – Palindrome)

The above variation uses both forward and reverse pitch mappings at the same time, to create a palindrome effect. The oscillators with the reverse pitch mapping are 180 degrees out of phase, which causes some nodes to cancel each other out, and creates some mildly interesting filtering effects.

Whitney Music Box (chromatic version – 88 tines)

The above variation uses 88 tines in a chromatic arrangement with alternating ascending and descending notes. Because there are more bumps on the wheel, there are more notes in the sequence.

Update 4/25/06:

I’ve generated a few more variations, which you can see here. I’ve begun using STK (the synthesis toolkit) instead of Syd to generate the audio, since this is more commonly available software that will allow other folks to experiment with this type of system. I generate the audio score using a Perl script which outputs note events in the SKINI format. I use the command-line utility calico to render the audio.

Update 4/26/06:

Nyquist, the Lisp-based music language, also provides an elegant way to generate this type of music. Here’s a sample nyquist score that I used to generate variation 11.

Update 5/2/06:

http://www.krazydad.com/blog/2006/05/02/a-fractal/In this new post, I describe the nature of the star-fish patterns in more detail.

Update 9/22/11:

I added links to Whitney’s book in the blog post, and slightly modified the text for clarity. I changed the original Flash illustration to a GIF, so it can be viewed on IOS devices. I’ve updated the links to point to their current locations. You’ll find lots of sample code on Github.

Share and Enjoy:

62 Responses to “Visual Harmony”

I think you may enjoy the interactive musical art/toy by Toshio Iwai called Electroplankton. You will need a Nintendo DS for it.

2. jbum Says:

Thanks Chad. I’ve heard good things about Electroplankton, but haven’t had a chance to play with it yet (due to lack of said Nintendo DS…).

Regards,

- Jim

3. jabarton Says:

amaaaaazing. i need to read this book perhaps. can you do a subharmonic version?

4. KrazyDad » Blog Archive » A φ Fractal Says:

[...] « Visual Harmony [...]

5. loreto martin » Blog Archive » Whitney Music Box Says:

[...] John Whitney, Arabesque, 1975, [...]

6. KrazyDad » Blog Archive » I’ve been dugg! Says:

[...] About the Whitney Music Box [...]

7. oxenrig Says:

Dude…. this thing’s driving me insane…. like it makes perfect sense, but at the same time…. it doesn’t. It’s like an acid trip…. i’m seeing sounds & hearing colors…. is there something wrong with me?

8. Peripatetic Circumambulant » Whitney music box Says:

[...] Whitney music box is amazing. I found it via this post on Digg. This post by its maker explains it. I have not bought into music of the spheres, but many of them are strangely satisfying. [...]

9. tahome Says:

Nice! However, animation and sound go out of sync after one period (~2min.) and the colored balls stick to a straight line. I was trying var.9 in Mozialla Firefox on MacOS X (but I doubt it works right on Windows, can’t check from here now, sorry)

HTH,
–th

10. jbum Says:

tahome: Yeah, I know. The variations with more than 48 notes are particularly problematic with the sync.

There are two problems:

1) As time passes, audio is increasingly out of sync with animation – due to differences in playback speed, and imprecise loop point.

2) After 1 revolution, ball motion starts to acumulate errors, starting in the center and expanding outwards. I still haven’t figured out why the balls get out of alignment after 1 revolution, but I assume it’s some kind of cumulative floating point error. I haven’t spotted how this might occur in the code yet.

The interactive versions I have been working on (which generate MIDI events in real-time) don’t suffer from either of these problems.

11. jbum Says:

Okay, error #2 is fixed in all but variation 7. That one’ll get fixed later today.

12. ianmcarey Says:

Hi, I’m really enjoying all the variations. As a musician, a couple of really intesting things jump out at me when listening to them–first, they all make musical sense intuitively, probably because the ear detects the step-based patterns (whether chromatic or harmonic) even when there are mulitple iterations going simultaneously at different speeds. Second, even though the function is mathematical, they all come across as having a really great sense of pacing–that is, a balance between easily noticeable “runs” giving way to random-sounding arpeggios broken up by those giant chords, giving way gradually to groups of runs. I think serial composers would kill for these kinds of effects.

I’d be curious to see what a version would sound like with a more consonant scale, such as diatonic or pentatonic–or even using a recognizable melody (so that not everything goes by steps).

13. jbum Says:

ianmcary:

Cool! A comment of substance. I don’t see too many of those here in the netherreaches of the blogosphere.

I’m going to take you up on your suggestion and generate a few more variations. I was thinking the same thing (about using a scale) when I did the prime/non-prime variations.

How about one which uses the regular 48-note orbits, but uses a blues scale? I’ll do that next.

There’s lots of ways to do a blues scale, how about

C Eb E F F# G Bb C

Maybe I should throw in the D?

If you have any other suggestions, let me know, and I’ll throw ‘em in. I kind of like Lydian and Phygian modes. I haven’t done pentatonic, cause I thought it would reduce harmonic interest..

As far as a recognizeable melody, it would be good to use something that has a fairly regular rhythm, and somewhow fits with the theme. Mebbe “Across the Universe?” Naw… too uniform…

14. jbum Says:

http://www.coverpop.com/whitney/midi/

Which contains some midi experiments using scales (the blues ones sound pretty nifty. okay, they all sound nifty) and the perl script used to generate them.

Enjoy!

These are just music only, but have the same pattern as the 48 and 88 note sequences.

15. The Music Gush » Blog Archive » The Whitney Music Box Says:

[...] There’s a blog entry with further explanation here, but my understanding of it is still quite limited. I thought it might appeal to the musicians (and fans of Electroplankton, I agree with the commenter). It’s somewhat hypnotic . . . [...]

16. ianmcarey Says:

Wow, that was fast!

A couple of thoughts: the “Major88″ sounded exactly like I’d hoped it would–the intervals jump out even clearer than in the chromatic versions. Fantastic.

You probably noticed, but any diatonic mode, including lydian or aeolian (natural minor) is going to end up sounding the same when put through that pattern. The harmonic minor one is great, though, and definitely has a totally different feel to it.

The blues scale ones are interesting–the bluesy tonality really comes out in the first version, while I think it’s harder to hear in the “88″ version, since there’s so much happening at once.

Some other possibilities you could try would include the whole tone or diminished (half-step, whole-step, half, whole, etc.) scales; as for melodies, I’d be curious to see what would happen with some Bach fugue subjects. (You can find a nifty guide to all the Well-Tempered Clavier fugue subjects here.)

Another possibility (which would probably just sound like a mess) would be some tone rows (12-note patterns where each note occurs once). Adding a rhythmic element (so not all notes are of equal length) would definitely be interesting, too.

Thanks for indulging me on those!

17. jbum Says:

Some more midi files added that use the prime and non-prime methods to add rhythm. “primeblues” “nonprimemajor” etc. More stuff to come…

18. holotone.net Says:

[...] From the developer’s blog: This weekend Iâ€™ve been playing, once again, with the ideas of experimental film pioneer John Whitney, using both graphics and audio. While Whitney was interested in turning musical ideas into motion graphics, Iâ€™m doing the inverse â€” turning one of his key animation ideas back into music. [...]

19. jbum Says:

I’m noticing that the “major” version (in the midi collection mentioned above) seems to be going from I to V near the middle of the piece). It would be nice
to find a way to exploit this or accentuate this more.

20. Oh Internet » Blog Archive » Harmonics Says:

[...] The fantastically named Jim Bumgardner made this neat page that takes animation ideas from some of John Whiney’s films on harmonic relationships and uses them to create music. More details about the techniques can be found here. [...]

21. White Man Stew » Music Box Variations Says:

[...] The “Whitney Music Box” (flash, sound) is described as “a musical realization of the motion graphics of John Whitney as described in his book Digital Harmony.” [...]

22. pictosonic Says:

This is really interesting. I made an attempt in Java, probably less aesthetic, based on spectral analysis.

I have problems determining accuratly notes from frequencies using FFT) extracted from .WAV files.

23. alwaysBETA » Beta Bites: Issue 13 Says:

[...] Grant – whitney music box – Inspired by experimental film pioneer John Whitney, Jim Bumgardner made this mix of animation, mathematics and animation that helps show how harmonics and chords are related. You might remember this guy from his Flickr Colr Pickr. There’s more information available at this blog post. [...]

24. Articles and Texticles / Whitney Music Box on Coverpop.com Says:

[...] (Quote:) A musical realization of the motion graphics of john whitney as described in his book “digital harmony”. Here’s a description. [...]

25. Bibi's box Says:

Whitney Music Box

Whitney Music Box: a musical realization of the motion graphics of John Whitney as described in his book “Digital Harmony”.The thirteen variations of the music box are the result of Whitney’s idea of harmonic relationships, based on audio, graphics…

26. plastron Says:

Very nice work! Having read Whitney’s book, it’s great to be able to see his ideas fleshed out, especially since his films are hard to find and often are not as ‘smooth’ looking as this. Do you think you will ever make a standlone, downloadable version of these examples (or is that possible)? I would love to have a “hard copy” so to speak, especially for showing to students of abstract film.

27. jbum Says:

Plastron: If you right-click the ‘swf’ link at the bottom right of the Whitney Music Box page, you can download the individual flash movies.

28. The Rare/Often » Blog Archive » Whitney Music Box Says:

29. KrazyDad » Blog Archive » Favorite Posts Says:

[...] Visual Harmony [...]

30. misterwarwick Says:

cool stuff!

What is the tuning?

It should be pythagorean. Unfortunately, it sounds like standard even tempered tuning.

But it’s still cool beans.

best,

HW

31. Multiple Extase » Visuelle Harmonie Says:

[...] Jim Bumgardner morpht eine Hauptszene aus dem Experimentalfilm “Digital Harmony” von John Whitney zurÃ¼ck zur Musik. [...]

32. Whitney Music Box at Junk Musik Says:

33. Electric Spork Says:

Whitney Music Box, quite possibly the coolest use of Flash to date…

Strangely hypnotic, fascinating, and even a little scary all at the same time, the Whitney Music Box is described as:
A musical realization of the motion graphics of John Whitney as described in his book “Digital Harmony”.
It’s a bit …

34. roclar.net Says:

[...] As jbum explains on his blog KrazyDad: “This weekend Iâ€™ve been playing, once again, with the ideas of experimental film pioneer John Whitney, using both graphics and audio. While Whitney was interested in turning musical ideas into motion graphics, Iâ€™m doing the inverse â€” turning one of his key animation ideas back into music.” The result is curiously intriguing. Watching the multi-colored balls going around a center point playing music while creating interesting patterns can be very soothing. [...]

35. ChrisPhoenix Says:

This is very cool. Very, very cool.

I believe it’s impossible for any artist in any medium to do a picture of the Mandelbrot set that’s as good as the original. Likewise, I suspect that atonal and other modern “classical” composers cannot do as well as this music. The Minute Waltz (var. 11) reminds me strongly of Mahler after the first few seconds–but I hate Mahler and I like the Minute Waltz. In fact, atonal and modern “classical” music has always sounded to me exactly like an orchestra tuning up, but I like almost all your variations. (I was raised on Bach &c, taken to concerts before I could talk–and stayed silent for hours, listening. I love classical, just not “classical.”)

I find myself wanting to rotate the white line a bit, so that it would catch the dots before or after they line up. Likewise, on the stereo versions, the odd-numbered lineups hit the right-hand line but never the left-hand line; there’s an imbalance there but I don’t know what to do about it.

Have you tried moving the dots at speeds corresponding to irrational numbers, e.g. the square roots of primes? That should produce music that never repeats, and never forms exactly simultaneous chords, and has no sensible meter. You can get approximately-linear multiples from ~2 to ~48 by taking the square roots of the first 48 primes, subtracting 0.8, and multiplying by 3.4. See e.g. http://primes.utm.edu/lists/small/10000.txt for primes. You can get more-linear multiples by taking primes from the middle of the list and likewise adjusting the range to 2-48.

If you do this, please email me–I’d love to know how it sounds.

Thanks,
Chris

36. jbum Says:

Hi Chris.

I used to have a composition teacher, (Alan Chaplin at CalArts) who disliked
it when his students would fill their pieces with long chromatic runs. He disliked it because the runs are not really composed – there is no choice
involved by the composer – it is just a kind of automatic fill of notes.

So, although I like the music produced by the Whitney Music Box, at some
level, I can hear Alan Chaplin behind my shoulder clicking his tongue in
annoyance, because these pieces are not really composed – they are
fully automatic.

At the same time, the content of the pieces is not as overly familiar to us as a chromatic scale, so perhaps that is some small consolation.

You suggested some interesting modifications. Thus far, I have found that the more I perturb the rules (trying to get more complexity out of the system), the music tends to get less interesting, although visually, the dots get more interesting. I believe this is because our optical circuitry is more equipped to handle new information. “The eye likes surprise, and the ear likes comfort.”

You’ll find some additional mods here:

http://www.coverpop.com/whitney/MIDI/

In which I’ve simply changed the scales, and here:

http://www.coverpop.com/whitney/index.php?var=v18

Where’ve I’ve altered the ratios of the speeds of the dots to get a non
repeating pattern. The speed ratios are based on divisions of powers of 2
by 12 – the same ratios that appear between pitches in a chromatic scale.

Alan Chaplin clucks his tongue yet again!

I’ll try some of your other ideas when I get a chance…

37. whitney music box « Ballinger’s Professional Blog Says:

[...] whitney music box A very neat site demonstrating interesting visual and musical ideas. Fun to play with! From the author: “This weekend Iâ€™ve been playing, once again, with the ideas of experimental film pioneer John Whitney, using both graphics and audio. While Whitney was interested in turning musical ideas into motion graphics, Iâ€™m doing the inverse â€” turning one of his key animation ideas back into music.” His blog (http://www.krazydad.com/blog/2006/04/23/visual-harmony/) has more information and more examples. [...]

38. mathseive Says:

This is a completely amazingly little project. I am fascinated. I’d like to have a go at doing something similar myself in Max/MSP (whether i do get round to it is another matter!).
Do you have any links to videos of John Whitney’s works? i’d really like to see some more!
Thanks!

39. jbum Says:

I’ve saw some of his films when I attended CalArts, and I saw some good ones at the “Visual Music” exhibition which ran recently in D.C. and L.A.

Unfortunately, these films are not available on DVD. You can see a few poor quality stills of them online if you google “Whitney” and “Visual Music”.

40. Fiat Lux! » Archivo del weblog » Whitney Music Box, Musica con colores Says:

[...] Cada vez que un punto de color pasa por la linea del medio reproduce un tono dependiendo del color que sea el punto. Al mezclarse el movimiento de los puntos que giran en espiral se produce musica. Hay diferentes variaciones y tipos de notas para disfrutar. En la pagina de Krazydad podemos ver como surgio esta idea. Un buen alucin para disfrutarlo en pantalla completa. [...]

41. fake artist » Whitney music box Says:

[...] La petite fascination de la matinÃ©e. des petits films en flash inspirÃ© du travail de John Whitney. Explication ici en anglais et cliquez sur ce lien pour voir les animations. post-it : pensez Ã  utiliser des sons dans mon futur projet net-art. [...]

42. Whitney music box at monkeyPi Says:

[...] The Whitney Music Box About the WMB [...]

43. Hola, mundo. » Blog Archive » On ChucK, Terry Riley, Jim Bumgardner and emulating old masters through code Says:

[...] Having a look at the programmer’s website (ah, curiosity) I’ve discovered why the url sounded so familiar: He’s also the man behind whitney music box, a well known series of animations inspired by the work of john whitney, as described in his book Digital Harmony: On the Complementarity of Music and Visual Art. The post explaining the work is worth a look. [...]

44. Tai Mingshen » The Whitney Music Box Says:

[...] Jim Bumgardner has created a musical realization of one of experimental film pioneer John Whitney’s ideas on harmonic resonance, as described in his book Digital Harmony. You’ll find the technical details on Jim’s blog KrazyDad. I picked this up from The J-Walk Blog and see that it is also featured on digg. [...]

45. mlarson.org » Says:

[...] I’m pretty much fascinated with the Whitney Music Box, which explores some of the ideas in John Whitney’s 1959 book Digital Harmony. I like the microtonal variation, and the sine wave harmonics are cool because harmonics are inherently cool. Jim Bumgardner wrote more about this project and some of the mathematics of the patterns in his blog. [...]

46. bob Says:

I love what’s been done here. I think it might be interesting to see if this concept could be somehow adapted or integrated into a performance setting, perhaps incorporating live players. It’d also be interesting to try varying other parameters besides pitch. Very amazing, thanks for this.

47. ledhed2222 Says:

This is pretty interesting work, Jim. Can you recommend some further reading for me on related subjects?

48. jbum Says:

Hi Greg,

The book that accompanied the Visual Music exhibition has a pretty good overview of the field…

http://www.amazon.com/Visual-Music-Synaesthesia-Since-1900/dp/0500512175

49. The Ultimate Insult | Blog Archive | Visual Harmony Says:

[...] Visual Harmony The Whitney Music Box is a cool idea that I can’t explain properly. Read more about it here. [...]

50. Dan Says:

Awesome stuff, this inspired me to go back to doing some flash animation.
I had fun creating some other variations:

One with semicircles:

One very colourful one – The complementary colours at the half-way point are nice:

and one where the elements are arrayed on a grid, not overlayed on the same point:

I’ve done something a bit different in the first 2 to continue further out on the circle:
In the originals, as you move out from the centre each layer does one less rotation per cycle.
I’ve continued this with a stationary layer, then layers which rotate in the opposite direction at increasing numbers of turns per cycle.
As you can see the result is a fairly natural looking continuation – I didnt add any sound yet, but It might be interesting.
My own interest is very much on the geometrical side and I can see some very interesting links to inversion, stereographic projection and loxodromes.
I’ll be posting more about these on my website soon

51. jbum Says:

Great stuff Dan! I added a blog entry to highlight your work. Looking forward to seeing more!

52. KrazyDad » Blog Archive » Dan Piker’s Whitney Variations Says:

[...] a recent comment here on KrazyDad, Daniel Piker shared some flash variations he made recently of the John [...]

53. ariyor Says:

In the originals, as you move out from the centre each layer does one less rotation per cycle.

54. Alex Sanchez Says:

Hi, your work is very interesting. I am a music teacher doing research on some math/music ideas. Do you have a program like tone matrix (http://www.hobnox.com/audiotool) where you can customize the scales utilized as well as the metrical grid? That would be very helpful!! Thanks so much!

55. jbum Says:

No, however you’ll find some scripts on the Whitney pages that can be modified to customize scales and so on.

56. Sebastien Cazaudehore Says:

Hello everyone,
today I went on a weird quest. about 12 years ago (first problem) I accidently found a website/blog with real-time interactive music triggered by activating various parts of a fractal setup.
of course, there was almost no words on this page which makes it impossible to track back. I know that htis was a long time ago, and chances are that the site does not existe anymore. but maybe someone heard about it and remember its name. any thoughts on the question and you can email me please. datagaliwabe@hotmail.fr

57. foxx314 Says:

hi there,
damn, i surely spend hours watching these harmonics =) very nice !!
i’m interested in mathematics and noticed that the same patterns occur in the orbits of the mandelbrot set:

click http://math.hws.edu/xJava/MB/ , in the applet choose “tools / mouse shows orbit”, and then click & drag your mouse near the border of the main cardiodid. exciting !! is there some similarity ??

58. Sammy Says:

Wow. That is absolutely brilliant. I’m kind of an aspiring music theory buff, and seeing the different patterns and numbers in this manner was really, REALLY awesome. It’s got a lot to say about resonance and harmonics. Beautiful work. Just plain beautiful.

59. A very interesting experiment in sound & motion – “Visual Harmony” – The Whitney Music Box | The Beat-Play Experiment Says:

[...] Info from creator at KrazyDad.com: [...]

60. SelflessGene Says:

I’d love to see you put out a version where the dots moved based on the laws of gravity.

Have : velocity = sqrt(m^2/((n+m)*r))

where m is the mass of an arbitrarily massive invisible black hole at the center,

n is the size of the planets (as estimated by the size of each circle)

r is the distance between each planet and the center.

Let me know if you plan on doing it!

61. Squirrel Says:

Wow! I love it! I found this form originally as pelotas.swf and got frustrated with not being able to control it…… so I made my own.
http://squirrel.rp83.com/CounterClock.html

I love your addition of the sound. I think I am going to sit here and let these play all afternoon.

Thanks!

62. jbum Says:

Thanks Squirrel!

I’ve also done some Processing versions, which you’ll find here:

http://www.openprocessing.org/portal/?userID=1032