Notes on The Machine Is Learning

This is a series of art videos that were generated as a by-product of an ongoing computational media project further described in GVOD, GVOD + Analytics: Star Treks \\///, etc.

The Machine is Learning, v2.2.
12:37 minutes, 960 x 720 pixels.

The Machine is Learning, v2.3 lbp
12:37 minutes, 960 x 720 pixels.

The Machine is Learning, v5.7 multi
12:37 minutes, 960 x 720 pixels.

The Machine is Learning, v5.7 multi ghost
12:37 minutes, 960 x 720 pixels.

GVOD + Analytics: Star Treks \\///



This is a fan studies and media assemblage experiment, loosely associated with Professor Abigail De Kosnik’s Fan Data/Net Difference Project at the Berkeley Center for New Media. It uses technology associated with copyright verification and the surveillance state to desconstruct serial television into a hybrid media form.

The motivating question for this work is simple. How does one quantize serial television? Given a television episode,  such as the third episode of Star Trek, how can it be measured and then compared to other episodes of Star Trek? Can characters of the original Star Trek television series be compared to characters in different Star Trek universes and franchises, such as comparing Kirk/Spock in Star Trek to Janeway/Seven-of-Nine in Star Trek Voyager? Given a media text, how do you tag and score it? If you cannot score the whole text, can you score a character or characters? How do characters or elements of a media text become countable?

Media Texts:

Star Trek (The Original Series), aka TOS, 1966 to 1969. Episodes: 79. Running time each: 50 minutes. English subtitles from subscene for Seasons 1, 2, 3.

Star Trek Voyager, aka VOY, 1995 to 2001. Episodes: S03E26 to S07E26, ie #68 to #172, a total of 104. Running time each varies between 45 and 46 minutes.

Media Focus/Themes:

The pairs of Kirk/Spock in Star Trek the Original Series and Janeway/Seven of Nine in Star Trek Voyager will be compared in a media-analytic fashion.

A popular fanfic genre is called One True Pairing, aka OTP, which is a perceived or invented romantic relationship between two characters. One of the best known examples of OTP is the pair of Kirk and Spock on TOS. Indeed, fanfic involving Kirk and Spock is so popular to have its own nomenclature, and is called slash, or slash fic.

The pair of Janeway and Seven of Nine are comparable to Kirk and Spock as both the Janeway and Kirk characters are captains of space ships, and both the Seven of Nine and Spock characters are presented as “the other” to human characters: both the borg and vulcans are presented as otherworldly, non-human. The two pairs are different in other areas, the most obvious being gender: K/S is male, J/7 is female.

Some edit tapes for K/S can be found on YouTube for Seasons 1, 2, and 3. Some fanvids for J/7.

Open Questions:

This is a meta-vidding tool with an analytic overlay. It takes serial television shows and adds facial recognition to count face time and change the focus of viewing to specific character pairs instead of entire episodes. Developing the technology to answer these analytic questions, answering and understanding the answers, and formulating the next round of questions is the purpose of this project.

1. Should the method be the first 79 episodes that the character-pairs are together? How do you normalize the series and pairs?

Or minute-normalized, after the edits? The current times are:

TOS == 79 x 50 minutes == 3950 “character-pair” minutes total

VOY == 104 x 43 minutes == 4472 “character-pair” minutes total

2. Best method for facial recognition.

One idea is to use openframeworks, and incorporate an addon. Get FaceTracker library. See video explaining it. Get ofxFaceTracker addon for openframeworks.

Another is to use opencv directly.

OpenCv documentation main page.

Tutorial: Object detection with cascade classifiers.

User guide: Cascade Classifier Training.

Contrib/Experimental: Face Recognition with OpenCV. See the cv::FaceRecognizer class.

Many, many variants go into this. Some good links:

Samuel Molinari, People’s Control Over Their Own Image in
Photos on Social Networks, 2012-05-08

Aligning Faces in C++

Tutorial: OpenCV haartraining Naotoshi Seo

Notes on traincascades parameters

Recommended values for detecting



ffmpeg concat

LBP and Facial Recognition Example with Obama

Simple Face recognition using OpenCV, Etienne Membrives, The Pebibyte


IEEE Xplore: Face detection, pose estimation, landmark localization in the wild, 2012

Xiangxin Zhu, Ramanan, D



3. Measuring “character” and “character-pair” screen time. How is this related to the bechdel test? [2+ women, who talk to each other, about something besides a man] Can be this used to visualize it or flaws as currently conceived? What is bechdel version 2.0? [2+ women, who talk to each other, about something besides a man, or kids, or family] Can we use this tool to develop new forms?

4. How to auto-tag? How to populate the details of each scene in a tagged format? If original sources have subtitles, is there a way to dump the subs to SRT, and then populate the body of the wordpress with the transcript? Or, is there a way to use google’s transcription API’s to try and upload/subtitle/rip?

5. Can the netflix taxonomy be replicated? Given the patents, can some other organization scheme be devised?


0. Prerequisites

Software/hardware base is: Linux (Fedora 20) on Intel x86_64, using Nvidia graphics hardware and software. Ie, contemporary rendering and film production workstation.

Additional software is required on top of this base. For instance, g++ development environment, ffmpeg, opencv, openframeworks.080.

Make sure opencv is installed.

yum install -y opencv opencv-core opencv-devel opencv-python opencv-devel-docs

See OpenCV Configuration and Optimization Notes for more information about speeding up OpenCV on fedora.

1. Digitize selected episodes for processing with digital tools

Decrypt via makemkv. Compress to 3k constant rip with HandBrake.

Using 720p version of TOS in matroska media container. Downloaded SRT subtitles from fan sites. Media ends up being: 960×720, 24 frames a second.

2. Quantize each episode to a select number of frames.

Make sure ffmpeg is installed.

yum install -y ffmpeg ffmpeg-devel ffmpeg-libs


Sample math as follows. Assume a fifty minute show has 24 frames a second. That is:

50 minutes x 60 seconds in a minute x 24 frames a second == 72k total frames an episode.

Assuming a one-frame-a-second sample resolution gives 3k frames for the total set of frames in TOS episode one. Use ffmpeg to create a thumbnail image ever X seconds of video. And set to one image every second.


mkdir $TMPDIR;
ffmpeg -i $1 -f image2 -vf fps=fps=1 ${TMPDIR}/out%4d.png;

3. Sort through frames and set aside twelve frames of Kirk faces, twelve frames of Spock faces.

This is used later, to train the facial recognition. Note: you definitely need hundreds and even thousands of positive samples for faces. In the case of faces you should consider all the race and age groups, emotions and perhaps beard styles.

For example, meet the Kirks.

And here are the Spocks.

In addition, this technique requires a negative set of images. These are images that are from the media source, but do not contain any of the faces that are going to be recognized. These are used to train the facial recognizer. Meet the non-K/S-faces.

4. Seed facial recognition with faces to recognize. Scan frames with facial recognition according to some input and expected result algorithm, and come up with edit lists that can be used to frames that are relevant to the character-pair.

Need either timecode or some other measure that can be dumped with an edit decision list or specific timecode marks. Some persistent data structure? Edits made.

5. Decompose episode into character-pair edit vids.

Use edit decision list or specific timecode marks, as above. Automate ffmpeg to make edits.

6. Store in wordpress container, one post per edit vid? Then with another post, tie together all of a single episode edit vids into one linked post?


There are both copyright risks and patent opportunities in this line of inquiry.

Production Notes:

How Netflix Reverse Engineered Hollywood, Alexis Madrigal, The Atlantic, 2014-01-02

Cyperacea Papyrus XVD14

Papyrus Baselines

Papyrus 1 Process

Papyrus 9 Moving

Production Notes

ffmpeg, potrace, scripting inkscape, imagemagic

Asama OG Combinatorics

Production and process notes for the creation of the compositions and plates for Asama OG can be found via Asama OG x Nezu Museum x Birgit Skiold. Some of the finished pieces are detailed in Asama Stone Ensos.

Other pieces are contemplated, and not-yet-actualized. Further explorations, new and different compositions, and the theory of generating and combining these different plates into works on paper is detailed below.

A quick setting of the scene: during the first half of 2013, four compositions designed for direct gravure were developed for rendering to copper plate and paper during two weeks of photogravure workshops at Crown Point Press. In addition, during the evening hours of the workshop, three experimental deboss plates (copper plates with machine-directed incisions) were produced at TechShop. A final layer was composed during the first week of workshop from a reclaimed copper plate, with various neo-drypoint and aquatint flourishes. This reclaimed plate was intended to be inked with one of three colors, all also mixed with silver metallic: orange, pink, light blue.


The 8 Asama OG Plates

So, to recap the math: 4 direct gravure images x 3 deboss images x 3 colors. Doing some simple calculations gives a first-guess edition size of:

4 x 3 x 3 = 36 combinations of (enso x color x deboss) plates

Thirty six combinations of direct gravure, color, and debossing.

During the workshop, all the plates were produced by Ianne Kjorli, and a partial edition was proofed. It was quite hectic, and the development of the ink colors, deboss plate creation, and direct gravure plate-making were all disjoint processes. There were a lot of moving parts and it was unclear to all how the parts would eventually fit together. Most of the time what made it to paper was the combination of the just-finished gravure plate plus the just-finished or clean deboss plate plus the color du jour. Looking at these prints after the fact makes me think: Ianne Kjorli is made of magic, or magic cross unicorn.

The proofs, any works that were outright mistakes or pure experiments, and all plates sat around on drying racks off Harrison street in downtown San Francisco, oil slowly drying. Weeks pass. Fleeting images of these prints, caught in a snapshot as if archived on tumblr, preoccupy my early waking dreams. At least one month. Eventually everything is claimed, and brought back to Devastation Pacific.

Surrounded by the green walls of Devastation Pacific, the prints are carefully unwrapped. The prints are everything hoped for and more. I go through the prints, trying to assess the output but thinking to myself: this cannot be all of them. Sadly…. it is.

It really is.

It is the end of them. I realize that I don’t have all four direct gravure plates in combination with the three possible colors in the reclaimed plate. Ie, of the basic set of 12 (4 x 3), I have maybe six. Eight?

4 x 3 = 12 basic combinations of (enso x color) plates

And, for fun:

12 > (6 || 8)

So ….. six or eight are less then twelve. Why even bother counting? Pause. Look out window. Breathe. Given that, now what. Given what I’ve got, how is this to be hung?

I start sliding these prints around on the magnets, and combine up to four on a horizontal line. Some of the test prints turn out to have real value. In particular, the test prints of the reclaimed plate with the colors blue, pink, or orange are all winners. In numbers:

3 = 3 combinations of (color) plates

Similarly, the test prints of the deboss plates are charmingly sculptural. So:

3 = 3 combinations of (deboss) plates

I recall Ianne Kjorli strongly championing the combination of black direct gravure and debossing, without the background color. Although only a few were prototyped, these prints are studio visitor favorites. Doing a complete set seems like a no-brainer.

4 x 3 = 12 combinations of (enso x debossing) plates

A week later, two tops, everything gets packed up and bundled up for a trip across the bay to meet a photographer.

More time elapses.

I start sliding photographs of the prints around on my computer screen. Weeks pass. A month. Two. During this time, there is more sliding around, and talking, and trying to figure out these combinations. I come up with three, one composed of two plates. One with three, in a triangle shape. And one with four, in a square shape. I realize that the color plates can be printed in two orientations, which causes a combinatoric explosion and will thus be ignored with the utmost precision.

Some time later, I cut down the prints, and double-hang the studio walls. Re-hang, according to my pre-visualizations on the computer. Very satisfactory.

To recap: six months after the production, I am just figuring out what to make of all this. There is an inventory of pattern building blocks, there is this idea of a transformable matrix, re-configurable and active prints. It is only at this point of careful study do I realize that the full set of prints needed is vast.

Vast. A best guess at the full set of required prints:

3 x 3 = 9, 3 sets of all (color) plates
2 x 3 = 6, 2 sets of all (deboss) plates
4 x 3 = 12 combinations of (enso x debossing) plates 
4 x 3 x 3 = 36 combinations of (enso x color x deboss) plates

How vast?

63 prints in one full edition

For reference, a complete edition is depicted in the following graphic.

The 63 Individual Prints in Asama OG Combinatorics 1.2

Some good news: it is possible to create finished compositions lazily, from a partial edition. New compositions like:


Asama Enso Shoebox for Nezu (blue, orange v, mushroom, cornerstone), 2014

Four intaglio prints. Three with debossing, one orange, one blue. Each print is 510 x 660mm (20 x 26in), so composition is 1020 x 1320mm (40 x 52in) as pictured.

Can be created from the following partial edition:

Used Prints in Asama Enso Shoebox for Nezu


Asama Kochi Ensos 1, 2014

One intaglio print. One blue, one black with debossing. The print is 33 x 26 inches, ie two plates with 3.5″ margins, as pictured. Paper is KH-17 Tosa Hanga Natural printed chine colle on Somerset backing.

Can be created from the following partial edition:


Used Prints in Asama Kochi Ensos 1

Asama Bilbao Orange Enso, 2014

Asama Bilbao Pink Enso, 2014

Diptych of two stone ensos. Size of each is 100cm x 133cm KH-17 Tosa Hanga Natural printed chine colle on Somerset backing.

Thinking about:

Andy Warhol, Shadows, Menil and MOCA

Wishful Reading: Art as Art, Electronic Media, Theory, Generatif

Step One: Invent time machine with comfortable reading situation.
Step Two: Acquire the following.

Art and Electronic Media

Walter Benjamin, Art in the Age of Mechanical Reproduction

Walter Benjamin, The Arcades Project

Marshall McLuhan and Quentin Fiore, The Medium is the Massage, 1967

Edward Shanken, Art and Electronic Media, Phaidon, 2009

Edward Shanken, Art and Electronic Media Online Companion, 2013

Lev Manovich, Software Takes Command, 2008-2013

Lev Manovich, Soft Cinema, 2002-2005

Lev Manovich, The Language of New Media, 2001

New Media Art, Mark Tribe, 2006

New Media: A Critical Introduction, 2nd Edition, 2009, Martin Lister

Domenico Quaranta, In Your Computer, 2011

Domenico Quaranta, Beyond New Media Art, 2013

Domenico Quaranta, The F.A.T. Manual, 2013

Expanded Internet Art and Information Milieu, Cici Moss, Rhizome, 2013-12-19

Curating in the Time of Algorithms, Christian Nagler and Joseph del Pesco, Fillip, Issue 15, 2011


Preparation readings from DMA, UCLA and BCNM, UCB


Design By Numbers, John Maeda

Computational Information Design, Ben Fry, 2004, Dissertation, MIT

Form+Code In Design, Art, and Architecture, Casey Reas, Chandler McWilliams

Generative Design; Visualize, Program, and Create with Processing, Hartmut Bohnacker, Benedikt Grob, Julia Laub, 2012

Visual Grammar, Christian Leborg, 2004

Carsten Nicolai, Fades

Carsten Nicolai, Static Fades

Carsten Nicolai, Moire Index

Grapefruit, Yoko Ono, 1964, 1971

Acorn, Yoko Ono, 2013


Earth Wind Interactive Graphic


Inkjet is the New Black Painting

Scott Rothkopf discussing Wade Guyton’s eight-panel painting Untitled (2008), 2013-01-11, WhitneyFocus on YouTube

Andrew Fisher and Christine S. Kim discuss Wade Guyton’s painting, 2013-02-28, WhiteneyFocus on YouTube.

“James Kalm” aka Loren Munk on video walkthrough of Wade Guyton’s OS show at the Whitney Museum, 2012-10-05.

Visual Culture, Aesthetics, Perception

James Turrell: A Retrospective, Michael Govan, Christine Y. Kim, Prestel, 2013

Emblemata, James Turrell, 2000, Segura Publishing Company, Tempe, AZ

Emblemata Sacra: Spe, Fide, Charitate, Guilelmus Hesius, 1636

Heraldry Symbols and Meanings, ThemesAchievement Elements, Animal Attitude, List of Heraldic Charges, Variations of the Field, Heraldic Phantasy Creatures

Light and the Artist, Thomas Wilfred, The Journal of Aesthetics and Art Criticism, June 1947

Marcel Minnaert, Light and Color in the Outdoors, 1995, ISBN 0387979352

Marcel Minnaert, Light and Color in the Open Air, 1954

Maurice Merleau-Ponty, Phenomenology of Perception, 2013, ISBN 1045834333

James J. Gibson, The Ecological Approach to Visual Perception, 1986, ISBN 0898599598

James J. Gibson, The Senses Considered as Perceptual Systems, 1966

ABC Art, Barbara Rose, October 1965, Art in America

Art and Objecthood, Michael Fried, Artforum, Summer 1967

Robert Morris, Notes on Sculpture, Part 3, Artforum, Summer 1967

Rudolf Arnheim, A Plea for Visual Thinking, Critical Inquiry, Vol. 6, No. 3 (Spring, 1980), pp. 489-497

Rudolf Arnheim, The Power of the Center, 1988

Rudolf Arnheim, New Essays on the Psychology of Art, 1986. Esp. “The Double-edged Mind: Intuition and the Intellect”, “The Tools of Art-Old and New”, “Emotion and Feeling in Psychology and Art”

Agnes Martin:Writings, Agnes Martin, 2005, ISBN 3775716114

Agnes Martin: Paintings, Writings, Remembrances, Arne Glimcher, 2012, ISBN 3775716114

Hal Foster, Vision and Visuality, 1988, Dia Art Foundation

Free is the Only Option

The GNU Manifesto, Richard Stallman, 1985-2014

Guerilla Open Access Manifesto, Aaron Swartz, 2008

Copy Culture in the US and Germany, 2013, ISBN 978-1481985925

Piracy is the Future of Television, Abigail De Kosnik, Convergence Culture Consortium, 2010

Online Piracy And China’s American TV Drama Fanatics, Ousha Yang, Disintermediators, 2013-05-01

In Praise of Copying, Marcus Boon, 2010, ISBN 978-0-674-04783-9

Peter Lunenfeld, The Secret War Between Downloading and Uploading, 2011, MIT Press

Copyright, Permissions, and Fair Use among Visual Artists and the Academic and Museum Visual Arts Communities: An Issues Report, A Report to the College Art Association, February 2014

Pew Internet Libraries, Pew Internet & American Life Project. See: The Rise of E-Reading, 2012. E-Reading Rises as Device Ownership Jumps, 2014.  Reading Habits in Different Communities, 2012. Library Services in the Digital Age, 2013. From Distant Admirers to Library Lovers and Beyond: A typology of public library engagement in America, 2014

Simon Parkin, The $5.7 Million Magazine Illustration, The New Yorker, 2014-02-20

Randy Kennedy, Richard Prince Settles Copyright Suit With Patrick Cariou Over Photographs, NYT, 2014-03-18

Randy Kennedy, Court Rules in Artist’s Favor, NYT, 2013-04-13

Anne Burdick, Johanna Drucker, Peter Lunenfeld, Todd Presner, Jeffrey Schnapp, Digital Humanities, 2012, MIT Press

Virtueel Platform Research: Born-Digital Kunstwerken In Nederland, English summary and recommendations, 2012 Project Conservation Media Art Collection, Netherlands.

Institutional Critique, Commodity Fetishism

Institutional Critique: An Anthology of Artists’ Writings, Alexander Alberro and Blake Stimson, 2009, MIT Press

Thoughts on Art Education, Rudolf Arnheim, 1990

Alternatives, Coco Fusco, Brooklyn Rail, Feb 5, 2013

Race in the Land of MFA, Alex Gallo-Brown, May 28, 2013

Art Schools: A Group Crit, AIA, 2007

An Artist’s Education, Thomas Lawson, The Core Program, catalogue, Museum of Fine Arts, Houston, 2008

9.5 Theses on Art and Class, Ben Davis, 2013

After Art, David Joselit, 2013, ISBN 978-0-691-15044-4

What Is a Warhol? The Buried Evidence, Richard Dorment, NYT, June 20, 2013

Red Obsession: The vintage of the century in the year of the Dragon, 2012

The Global Contemporary and the Rise of New Art Worlds, Hans Belting/Andrea Buddensieg/Peter Weibel, 2013, MIT , ISBN 0262518341

Michael H. Miller, Ways Forward: What’s Next for the Dia Art Foundation?, 2014-02-04, galleristny

Michael H. Miller, Philippe Goes to Hollywood: With Vergne In Place as New Director, L.A. MOCA Looks to the Future, 2014-02-05, galleristny

Visual Forms of Gentrification vs. Ruin

Rosalyn Deutsche, Evictions: Art and Spatial Politics, 1996, MIT press

The Fine Art of Gentrification, Rosalyn Deutsche and Cara Gendel Ryan, October, Vol. 31, Winter 1984

Rebecca Solnit, Get Off the Bus, London Review of Books, Feb 20, 2014.

Kim Velsey, It’s Not About Yuppies Anymore: Gentrification Has Changed and So Has New York, New York Observer, 2014-02-06.

How to Write About Tax Havens and the Super-Rich: An Interview with Nicholas Shaxson, Longreads, The Atlantic, 2014

Andy Rachleff, You Need Equity To Live In Silicon Valley,May 22 2013

Mission 20th Street Neighborhood Group, various email correspondence

The Embedded Landscape of Gentrification, Jason Patch, Visual Studies, October, Vol. 19, No. 2, 2004

Residues of a Dream World: The High Line, 2011, Michael Cataldi, David Kelley, Hans Kuzmich, Jens Maier-Rothe and Jeannine Tang, Theory Culture Society, 2011

wizard hear me