Bittorrent Notes

Prerequisites

OS: Assuming Fedora [24-26] or Centos 7
HW: x86/64, i5, 16GB memory
NET: 2Gbytes/month

For AWS, this is m4.xlarge or m3.xlarge with 21 in / 100 out Gbytes per week for a 96-torrent 2-hr scraper. (Aka, got-705). Expect perhaps twice nework capacity this to saturate sample.

Cloud dedicated host or virtual server footprint: HP ProLiant DL120, Centos 7, 10TB/month.

Development machine is intel NUC Kit: Core i5-6260U Slim or Core i5-7260U (7I5BNK) Slim, 16GB, Fedora 25.

Software Setup

Development baseline is gcc-6.4.1, using C++17, boost-1.60, custom libtorrent from 2017-08-01.

To bootstrap gcc, download the gcc-6-branch from gcc.gnu.org, and configure for c/c++. Make, install. Note that “-std=gnu++17” is required as CXXFLAGS. Set CC as gcc-6.4, CXX as g++-64, and CXXFLAGS as “-g -O2 -std=gnu++17” via exports in .bashrc.

To rebuild boost on Centos 7, download the source RPM on F25, transfer it to the Centos 7 machine, and unpack it. Then:

yum install -y cmake bzip2-devel python-devel python3-devel libicu-devel  openmpi-devel  mpich-devel

To rebuild boost, a little bit of fun in the rpmbuild directory’s SPEC subdirectory:

rpmbuild -ba boost.spc --without python3 --without mpich LDFLAGS+=--build-id

Then install the generated RPMS.

Get some systems diagnostic tools for monitoring network load, VPN support, nmcli, etc.

dnf install -y nload NetworkManager NetworkManager-openvpn openvpn

GeoIP installed as

dnf install -y GeoIP GeoIP-devel GeoIP-GeoLite-data GeoIP-GeoLite-data-extra;
dnf install -y libcurl libcurl-devel;
dnf install -y libevent libevent-devel --allowerasing;
dnf install -y intltool;

Test that this is working by using geoiplookup on the command line

%geoiplookup 8.8.8.8
GeoIP Country Edition: US, United States
GeoIP City Edition, Rev 1: US, CA, California, Mountain View, 94035, 37.386002, -122.083801, 807, 650
GeoIP ASNum Edition: AS15169 Google Inc.

Other dependencies: rapidjson, rapidxml for visualization interface.

dnf install -y rapidjson rapidjson-devel rapidxml-devel;

Assuming SSL and crypto support.

dnf install -y openssl openssl-libs openssl-devel;

For the bittorrent protocol, use libtorrent built from source.

In addition to the operating system packages, python support for geolocation is needed. In particular, need bencode, geoip2, requests. So:

pip install GeoIP
pip install GeoIP2
pip install bencode
pip install requests

Required System Configuration

Remove rpcbind for datacenter use as per BUND warning. See disable portmapper notes.

Have to set ulimits to unlimited (min 16k) because of open file limits in seeding mode.

Background

Part one of Bittorrent distribution research was Fall 2015. Part two is Fall 2016. Part three is Spring & Summer 2017.

Bittorrent. Common terms and jargon.

Basic idea as per The BitTorrent Protocol Specification. Of this, of note is tracker and in particular the tracker scraper protocol.

Magnet links wikipedia entry. PEX, DHT, Magnet links all from lifehacker.

Questions

1. Trackers. List of trackers, announcements, UDP, http.
Fetishizing the most current trackers: 1, 2.

2. Public torrents, Private torrents, SSL torrents. Why are some of these able to be scraped, and others, not so much? From transmission-show -s my.torrent, will get status for public and private, but not ssl torrents.

3. Look at transmission dependencies: openssl, libcurl. See libtransmission includes: transmission.h, variant.h, utils.h. See: struct tr_tracker_stat, tr_torrentTrackers, tr_torrentTrackersFree, tr_info, tr_tracker_info, tr_torrent_activity. Of note, it looks like multiple reads of peers from tracker. TR_PEER_FROM_PEX, TR_PEER_FROM_DHT, TR_PEER_FROM_TRACKER.

4. Look at SSL cert example in libtorrent. See libtorrent github source repository.

5. What kind of tree/node visualizations will work? See The Book of Trees.

Deliverables

One. Data set 1 is four torrents from October 11 for the same recent serial television episode, each a different uploading group: LOL, DIMENSION, AFG, mSD.

Two. Data set 2 is three torrents from October 17, over 8 locales, during 3 time periods.

Three: Prepare for TWD 2016-10-23 Season 7 premiere.

  • visualization: three or more approaches, data types, presentation/exhibition,prototype
  • scraper analytics
  • archival data format
  • persistent scraping, archival interface

Fedora 20 Notes

Install:

San Francisco, California USA

yum install -y fedup
fedup --network 20

This chugs for a bit, and then on restart the screen may go black or appear hung on firstboot for
some time before booting into a desktop splash screen. One may need to wait for as long as 45 minutes here, so be chill. Grab a cocktail or go to lunch.

For the mac, you’ll need to create the usual EFI boot disk (netinst and use –efi in iso-usb-to-disk).

Update/Post-Install:

Install KDE and MATE

yum install -y @mate-desktop @kde-desktop

Install GNOME classic and tweak tool.

yum install -y gnome-classic-session gnome-tweak-tool

Google earth install via this answer. YMMV.

For USB microscope, software choice is GXSM, GNOME X Scanning Microscopy. Hardware recommends GTK+ UVC Viewer.

yum install -y guvcview

For rdate, install openrdate

yum install -y openrdate

Running openframeworks.080.

Get FaceTracker library. See video explaining it. Get ofxFaceTracker addon for openframeworks.

Things that Don’t Work:

  • bluetooth broken or missing when using MATE desktop. Frankly, bluetooth in *any* desktop is pretty flaky on current linux, but when using MATE it’s missing. This is a known issue with the bluez-libs upgrade to the 5.x version. In ye old days of previous Fedora’s, one could download the srpm for bluetooth-mate and rebuild it, solving everything. This approach doesn’t work with f20, and the suggestion to install bluedevil requires the invocation of the following as root to connect to the previously paired keyboard.
    bluedevil-wizard

    And then selecting the bluetooth device in the GUI and explicitly connecting.

    Other issues are 1) booting with encrypted drive renders decryption with bluetooth keyboard impossible (Bug 863883) 2) booting to GNOME login screen with bluetooth keyboard impossible (Bug 1056682)
    . Another solution: try KDE.

  • awful default font rendering. See known issues, and switch to a decent default font for system typography.

Previously broken things that may now be fixed, at least on first inspection:

  • high-resolution printing from large format Epson printers. Broken post Fedora 14, initial testing shows progress. But is it now fixed? More as this develops.
  • yum install -y libbdplus
  • wget http://vlc-bluray.whoknowsmy.name/files/KEYDB.cfg ~/.config/aacs/

Previous install: Fedora 19

Fedora 19 Notes

Install:

San Francisco, California USA

yum install -y fedup
fedup --network 19

This chugs for a bit, and then on restart the screen may go black or appear hung on firstboot for
some time before booting into a desktop splash screen. One may need to wait for as long as 45 minutes here, so be chill. Grab a cocktail or go to lunch.

For the mac, you’ll need to create the usual EFI boot disk (netinst and use –efi in iso-usb-to-disk).

Update/Post-Install:

Mostly the same as Fedora 18.

Current desktop is MATE, via

yum install -y @mate-desktop

Install nightingale, the linux fork of Mozilla.org’s Songbird music player.

Install simple-mtpfs, so that linux can play nice with android devices, and natively mount the
phone filesystem and copy files directly to it.

yum install -y simple-mtpfs

Install inkscape-pages,so that inkscape multi-page pdfs can be created.

Current issues:

0) printing with Samsung laser printers

No foo2qpdl package in Fedora 19, which defeats printing setup with certain Samsung printers. Instead, go to the homepage. Download foo2qpdl, compile, install. Then try to set it up with the usual admin tools on Fedora.

1) printing with Epson Large Format inkjets

Unknown if the print quality issues that surfaced > Fedora 14 are fixed or debug-gable. All output still stuck on Fedora 14 for the time being.

2) mtp support for android devices.

Good background page from archlinux. And for Fedora 19, here.

yum install -y simple-mtpfs

Then, assuming device is attached with a USB cable to the powered-on linux workstation:

simple-mtpfs -l

shows something like this for a Nexus 4

Device 0 (VID=18d1 and PID=4ee2) is a Google Inc (for LG Electronics/Samsung) Nexus 4/10 (MTP+ADB).
1: Google Inc (for LG Electronics/Samsung)Nexus 4/10 (MTP+ADB)

Create a mount point on the linux system, here /mnt/drive/1, and mount via

simple-mtpfs /mnt/drive/1

Checking with df -h

df -h

says:

simple-mtpfs             13G  4.6G  8.5G  35% /mnt/drive/1

3) Apple EFI macs and dual booting. Sigh, oh this again.

Previous install: Fedora 18