Image Captioning from Novels

captioningimagefromtags

Some progress, trying to caption images using sentences from novels. This first step is working ok, next will be to train a neural-network to do this automatically from my captioned set, then output new captions algorithmically.

Part of a short-term collaboration with the Social Dynamics group at Bell Labs, Cambridge.

Inpainting

inpainting

Totally obsessed with the idea of inpainting, used to reconstruct and remove items from images automatically. I’m especially in love with the glitchy, Predator-esque ghosting that remains.

This is very easy to accomplish with OpenCV and a few lines of code – here I’ve run this with a radius of 5px. The results are actually pretty usable, especially in busy or small areas of the image.

telea-5

The very simple code to run everything:

And here’s the same input, but with a really weird, blobby, random mask:

telea-5

h/t Miriam Redi

Grid Filter: WIP

GridFilter-web

Some experiments converting images to a series of differently-sized grids. Not sure where this is headed, but I like the competing patterns and emerging forms. (Click for full size – lots more details.)

WIP: Hard-Drive Visualizations

Some work-in-progress visualizations of the physical location of specific files on my hard-drive, being made as part of my residency at Bell Labs. The above two images are details, showing files (in red) and empty space (in gray); each little square is one 512-byte sector.

This current version is visualizing an 8GB thumb drive. The prints are approximately 36×60″ (91×152 cm) so each individual sector can be seen. The hope is to do a version of my laptop’s hard-drive, which will require a much larger print or set of prints.

The goal of these visualizations is not analytics or troubleshooting – instead, I’m very interested in the abstraction between us, our computers, and our data. We take a photograph of something in the world and store it on a physical drive (even the end-point of the cloud is still a physical object). Once stored on a hard-drive, digital objects, however ephemera-seeming, continue to exist as physical ones and zeroes, magnetic charges of a specific size and intensity. These visualizations are about trying to unpack and see my personal data on that level.

But there is a significant disconnect between myself and my computer, and the actual data on my hard-drive. It turns out that drives contain a specialized chip that handles requests to read and write data; the algorithms that actually retrieve and set data are a closely guarded industrial secret.  These chips act as a black box between the user and their data, rendering a detailed picture of the exact location of the bits that make up our digital lives almost impossible. This obscurity is especially prevalent in SSDs, due to the way they store data. This post on Aleratec puts it a finer point on it:

…unlike traditional hard-drives where the physical location of each bit of data is known and constant, the physical location of data in an SSD is highly abstracted from the outside world.  Whereas each Logical Block Address (LBA) on a [hard-drive] always points to the same physical location, the physical location to which an SSD LBA points changes often.

My images then are a best-guess of where the actual data is stored, extracted using digital forensics tools. Below is the full image of the drive – click on it for a full-sized view.

Deep Storage

IMG_9816

“You don’t want to see that” was the initial response to my question: “is there an equipment graveyard at Bell Labs?” I was told there was lots of new equipment and that the rooms in the basement were just full of useless old equipment, which is, of course, exactly what I wanted 😊. But after explaining why an artist might want to dig through 60 years of cutting-edge R&D research castoffs, I was kindly taken down to the basement.

The deep storage turned out to be several rooms (and probably more that I haven’t seen yet) packed with lots of useful stuff (a dozen cabinets chock-full of electronics components, huge spools of ribbon cable) and shelf after shelf of old equipment from labs that had been upgraded. Basically: it was amazing.

Case in point: the Harris Dracon TS1 Test Set above. It was used by telephone repair-persons to test connections while atop a telephone pole. Little phone keypad, soft cork grip, a heavy-duty lanyard hook, and of course the amazing blue plastic case.

Continue reading “Deep Storage”

Residency at Bell Labs

IMG_9774

Last week, I started as artist-in-residence at Bell Labs. Though they have worked with artists in the past, I’m fairly sure I’m the first long-term artistic collaborator at Bell Labs since the famous Experiments in Art & Technology which brought scientists and engineers together with the likes of John Cage and Robert Rauschenberg. Bell Labs was also the birthplace of the transistor, laser, UNIX, proof of the Big Bang, and the glowstick, among lots of other really cools stuff.

I’m planning to work on several large-scale pieces about wifi, radio, and EMF. Above is a shot of my very large whiteboard in the beautiful studio Bell Labs has set me up with.

IMG_9770

I could only bring a few things that I could carry on the train my first day, but I think I made a good choice: my dancing cat coffee mug and a copy of the Art & Technology book from LACMA’s original program.