Posted on

Arranging By Color And Objects With t-SNE

For an upcoming Drift Station project, we’ve been considering how to curatorially sort a massive number of images (about 100k) for presentation. Chronological? Random order? Some other logical scheme? But a more computational approach seemed to make sense: some way of parsing the images that took into account a variety of visual factors in each image, something that would be impossible to do manually.

Neural networks are the obvious answer here, and so I found some very helpful sample code from Gene Kogan and Kyle McDonald, and wrote some Python and Processing code that loads up a folder of images and extracts a vector representation from them. Then, using t-SNE and Rasterfairy, the images were organized into a 2D grid.

I’ve spent the last few days playing with settings in the code, and found there is an interesting balance to be struck between locally preserving color similarity and object similarity. (Note: this post is more of a quick note than a deep-dive analysis.)

Above: a version with blurred images, showing a pretty clear separation by color with fairly smooth transitions. Click on images for a higher-res version. Continue reading “Arranging By Color And Objects With t-SNE”

Posted on

Image Captioning from Novels

captioningimagefromtags

Some progress, trying to caption images using sentences from novels. This first step is working ok, next will be to train a neural-network to do this automatically from my captioned set, then output new captions algorithmically.

Part of a short-term collaboration with the Social Dynamics group at Bell Labs, Cambridge.

Posted on

“Preferred Stimuli” for DNNs

FeaturesAtLastLayerOfAlexNet_PreferredStimuliForEachNeuron-web

Not *entirely* sure what’s going on here (paper forthcoming) but I believe this is the equivalent of tickling a deep neural network: “understanding deep neural networks by synthetically generating the preferred stimuli for each of their neurons.” Via Anh Nguyen, Alexey Dosovitskiy, Jason Yosinski, Thomas Brox, and Jeff Clune at EvolvingAI.org.

Posted on

Torch-rnn: Mac Install

TorchRNN_TinyShakespeare-web

Update! For El Capitan and users of newer version of OS X, you may run into issues installing Torch or Lua packages. A fix is included now.

Update number two! Zach in the comments offers a really helpful fix if you’re on Sierra.

Update three! A lot has changed since 2016, so I’ll be posting a new version of this tutorial soon. In the meantime, please see the comments for common sticking points and troubleshooting.

There have been many recent examples of neural networks making interesting content after the algorithm has been fed input data and “learned” about it. Many of these, Google’s Deep Dream being the most well-covered, use and generate images, but what about text? This tutorial will show you how to install Torch-rnn, a set of recurrent neural network tools for character-based (ie: single letter) learning and output – it’s written by Justin Johnson, who deserves a huge “thanks!” for this tool.

The details about how all this works are complex and quite technical, but in short we train our neural network character-by-character, instead of with words like a Markov chain might. It learns what letters are most likely to come after others, and the text is generated the same way. One might think this would output random character soup, but the results are startlingly coherent, even more so than more traditional Markov output.

Torch-rnn is built on Torch, a set of scientific computing tools for the programming language Lua, which lets us take advantage of the GPU, using CUDA or OpenCL to accelerate the training process. Training can take a very long time, especially with large data sets, so the GPU acceleration is a big plus.

Continue reading “Torch-rnn: Mac Install”