Bittorrent Notes

Prerequisites

OS: Assuming Fedora [24-26] or Centos 7
HW: x86/64, i5, 16GB memory
NET: 2Gbytes/month

For AWS, this is m4.xlarge or m3.xlarge with 21 in / 100 out Gbytes per week for a 96-torrent 2-hr scraper. (Aka, got-705). Expect perhaps twice nework capacity this to saturate sample.

Cloud dedicated host or virtual server footprint: HP ProLiant DL120, Centos 7, 10TB/month.

Development machine is intel NUC Kit: Core i5-6260U Slim or Core i5-7260U (7I5BNK) Slim, 16GB, Fedora 25.

Software Setup

Development baseline is gcc-6.4.1, using C++17, boost-1.60, custom libtorrent from 2017-08-01.

To bootstrap gcc, download the gcc-6-branch from gcc.gnu.org, and configure for c/c++. Make, install. Note that “-std=gnu++17” is required as CXXFLAGS. Set CC as gcc-6.4, CXX as g++-64, and CXXFLAGS as “-g -O2 -std=gnu++17” via exports in .bashrc.

To rebuild boost on Centos 7, download the source RPM on F25, transfer it to the Centos 7 machine, and unpack it. Then:

yum install -y cmake bzip2-devel python-devel python3-devel libicu-devel  openmpi-devel  mpich-devel

To rebuild boost, a little bit of fun in the rpmbuild directory’s SPEC subdirectory:

rpmbuild -ba boost.spc --without python3 --without mpich LDFLAGS+=--build-id

Then install the generated RPMS.

Get some systems diagnostic tools for monitoring network load, VPN support, nmcli, etc.

dnf install -y nload NetworkManager NetworkManager-openvpn openvpn

GeoIP installed as

dnf install -y GeoIP GeoIP-devel GeoIP-GeoLite-data GeoIP-GeoLite-data-extra;
dnf install -y libcurl libcurl-devel;
dnf install -y libevent libevent-devel --allowerasing;
dnf install -y intltool;

Test that this is working by using geoiplookup on the command line

%geoiplookup 8.8.8.8
GeoIP Country Edition: US, United States
GeoIP City Edition, Rev 1: US, CA, California, Mountain View, 94035, 37.386002, -122.083801, 807, 650
GeoIP ASNum Edition: AS15169 Google Inc.

Other dependencies: rapidjson, rapidxml for visualization interface.

dnf install -y rapidjson rapidjson-devel rapidxml-devel;

Assuming SSL and crypto support.

dnf install -y openssl openssl-libs openssl-devel;

For the bittorrent protocol, use libtorrent built from source.

In addition to the operating system packages, python support for geolocation is needed. In particular, need bencode, geoip2, requests. So:

pip install GeoIP
pip install GeoIP2
pip install bencode
pip install requests

Required System Configuration

Remove rpcbind for datacenter use as per BUND warning. See disable portmapper notes.

Have to set ulimits to unlimited (min 16k) because of open file limits in seeding mode.

Background

Part one of Bittorrent distribution research was Fall 2015. Part two is Fall 2016. Part three is Spring & Summer 2017.

Bittorrent. Common terms and jargon.

Basic idea as per The BitTorrent Protocol Specification. Of this, of note is tracker and in particular the tracker scraper protocol.

Magnet links wikipedia entry. PEX, DHT, Magnet links all from lifehacker.

Questions

1. Trackers. List of trackers, announcements, UDP, http.
Fetishizing the most current trackers: 1, 2.

2. Public torrents, Private torrents, SSL torrents. Why are some of these able to be scraped, and others, not so much? From transmission-show -s my.torrent, will get status for public and private, but not ssl torrents.

3. Look at transmission dependencies: openssl, libcurl. See libtransmission includes: transmission.h, variant.h, utils.h. See: struct tr_tracker_stat, tr_torrentTrackers, tr_torrentTrackersFree, tr_info, tr_tracker_info, tr_torrent_activity. Of note, it looks like multiple reads of peers from tracker. TR_PEER_FROM_PEX, TR_PEER_FROM_DHT, TR_PEER_FROM_TRACKER.

4. Look at SSL cert example in libtorrent. See libtorrent github source repository.

5. What kind of tree/node visualizations will work? See The Book of Trees.

Deliverables

One. Data set 1 is four torrents from October 11 for the same recent serial television episode, each a different uploading group: LOL, DIMENSION, AFG, mSD.

Two. Data set 2 is three torrents from October 17, over 8 locales, during 3 time periods.

Three: Prepare for TWD 2016-10-23 Season 7 premiere.

  • visualization: three or more approaches, data types, presentation/exhibition,prototype
  • scraper analytics
  • archival data format
  • persistent scraping, archival interface

Conceptual Forms, Conceptual Cocktails

Le Classique, 2010

Photogravure, black ink on Somerset paper with plate mark and tone. 13 x 19″ (329 x 483mm).  Composition includes crop marks for A6 fill-in-the-blanks invitation, located at center-top of the plate.

This invitation reads:

CONCEPTUAL COCKTAIL PARTY

Yes there will be booze.

Details Marked Below.

HOURS

12   11   10   9   8   7   6   5   4   3   2   1

to

12   11   10   9   8   7   6   5   4   3   2   1

LOCATION

——————————————

On the roof ____

With projection _____

C++

San Francisco, CA

USA

Also present in the composition, are registration lines and fades. As part of the registration, centered on the text for the invitation, is the following production metadata:

PRINT THIS FOR FREE ON WHATEVER PRINTER WHEREVER YOU WANT!

Ask me how. Email printer@gnu.org and cc bkoz@gnu.org.

Underneath this in very small type there appears to be a mis-print, or type overlay. It is repeated, with some overprints, and says:

Then tell me how you did it!

—–

This is the first part of what was assumed to be a triptych composed of three etchings, all the same size. These were done half-way through the July 2010 Photogravure workshop at Crown Point Press, and etched and printed later in the summer and early autumn by Emily York. The workshop instructor was Emily York, with Asa Muir-Harmony. Attendees were Benjamin De Kosnik, Chayo de Chavez, Jay Dee Dearness, and Carolyn Dodds.

The three etchings are: Le Classique, Conceptual Cocktail, and a yet-unfinished piece that I call List Form. Plus one idea for another form, called SHOUT! All these engravings are arranged somehow on a wall, and also some kind of archival media that contains all the digital files used in the production of the plates is either attached to the back of the prints or otherwise incorporated into the display. In the summer of 2010, the archive consisted of a data DVD with outlines in Inkscape SVG files and TIFF files for photos, along with detailed system documentation for a linux print production workstation.  I did some cool drips on the DVDs with metallic ink to signify that they were ART, not DATA. That was what was attempted, at least. Now this part would probably be a folder on Dropbox and some links.

This is how I’d been looking at it for most of last year:

Le Classique, 2010

Conceptual Cocktail, 2010

List Form Plans, 2010

SHOUT! Idea, 2011

Production files for the third conceptual form, List Form, exist but have not yet been printed. This is in collaboration with Tomiyoshi Tsukada and is photogravure, guilloche, scanner-art with a three-layer silver overlay of text and fades.

There was another version of these prints conceptualized. It involved engraving a version of this with kif ink, presenting said version to John Gilmore in collaboration with Roland McGrath, and getting invited to James Turrell‘s Roden Crater. I’d like to think this version is still in development, but is admittedly a long shot.

These prints are very dear to me. But have largely defied any coherent explanation. Below is my latest attempt. Hopefully the passing of time has made it easier to explain what I was going for here.

This project is an investigation into printing forms and methodologies.

For the form experiments, I thought of printed matter, and tried to categorize them into distinct types or forms.

The first two etchings, Le Classique and Conceptual Cocktail, are examples of one of the most endearing and essential forms of printed communication, the invitation. This invitation is to an imaginary party. The third etching, List Form, is an example of another classic form of printed communication, the list. Grocery lists, to-do-lists, etc. The forth etching, SHOUT!, is an example of protest communication, the classic form for getting attention.

For the methodology experiments, I was trying to get at the essence of printed matter through the centuries, and the specialized knowledge and tools necessary to produce fine works on paper. Often, this knowledge is lost. Exact descriptions of some of the early photographic processes are lost forever. Hot type, ditched. Film pre-press, gone. Distressingly, digital files used in the production of works less than ten years old can become obsolete and rendered useless by proprietary format, abandoned products, and the inability to transfer/convert to newer production methods.

Influenced by the production methodologies of the free software movement, (started by Richard Stallman trying to freely communicate with a printer), I created a reference platform for the archiving of print production, based on the free/open source linux operating system and applicable tools, transforming proprietary curves used in the production of photogravure with free intellectual property. And a documentation system for production notes. Then, I used these files to create a reference print. The final step was to then attempt to dethrone this canonized static object, by inviting other participating printers to use these files to create this or altered works, and send me a copy, with documentation on how they got the print so that I could incorporate any new techniques.

Chayo Chavez indicated that this is a way to explain the free-software movement to art people, using their terminology. I considered this a pleasant by-product of the experiment.

As such, I expect this project to be ever-unfinished. But always worthwhile.

As Shown

Conceptual Cocktail, as above but in a swank frame, will be shown as part of “Traces, Marks, Fragments” at the San Luis Obispo Museum of Art (SLOMA) January 20th-February 26, 2012. This is a show juried by Sandow Birk.

Of Interest

Hiroshi Sugimoto, Conceptual Forms, as seen at Altalier Brancusi, Paris, 2006
Man Ray, Objet Mathematique, 1934-36
Edward Ruscha, Stains, 1971-75
Richard Stallman, The GNU Manifesto, 1985