Fedora 25 Notes

Install:
San Francisco, California USA

Using x86_64 netinstall images as install media.

Update/Post-Install:

Use RPM Fusion Repositories.

Some basics

dnf install -y tcsh gnome-tweak-tool livecd-tools ncftp lftp

Install KDE, Razor-qt, and MATE Desktops

	 	 
dnf install -y lightdm-gtk-greeter-settings mate-desktop mate-applets network-manager-applet caja mate-themes bluecurve-gtk-themes gnome-themes-legacy;
dnf groupinstall mate-desktop;  
dnf groupinstall kde-desktop;
dnf install -y gnome-themes;

Still a little sketchy, but optionally mate-compiz can be installed:

dnf groupinstall mate-compiz;

Install the basics for VLC

dnf install -y vlc vlc-core vlc-extras;
dnf install -y gaupol mediainfo

Install media xtra codecs.

dnf install -y ffmpeg ffmpeg-devel ffmpeg-compat-devel gstreamer-plugins-bad gstreamer-plugins-ugly gstreamer-plugins-good gstreamer-plugins-good-extras gstreamer-plugins-bad-free gstreamer-plugins-bad-free-extras gstreamer-plugins-bad-nonfree gstreamer-ffmpeg;
dnf install -y k3b-extras-freeworld;

Install negativo.
Use negativo17 repositories for HandBrake, volia.

dnf config-manager --add-repo=http://negativo17.org/repos/fedora-handbrake.repo;
dnf install -y HandBrake-gui HandBrake-cli;
dnf install -y makemkv mkvtoolnix mkvtoolnix-gui;
dnf install -y libdvdcss;

Use makemkv for Bluray.

Install some editors, utilities, and basics

dnf install -y abiword emacs tcsh

Install devel 1

dnf install -y openni openni-devel openni-doc  tbb tbb-devel tbb-doc eigen3-devel eigen3-doc openrdate simple-mtpfs emacs python-BeautifulSoup

Install devel 2

dnf groupinstall -y "Development Tools";
dnf install -y gcc-c++ libstdc++-devel libstdc++-docs boost boost-devel boost-doc valgrind gdb elfutils nemiver dwarves dejagnu emacs expect flex bison cmake git-svn mpfr-devel libmpc-devel lsb;
dnf debuginfo-install glibc;

Install database

dnf install -y glom libpqxx-devel libpqxx postgresql-devel

Install computer vision (hand-build clandmarks)

dnf install -y CImg-devel rapidxml-devel opencv opencv-devel

Install graphic

dnf install -y freetype-freeworld inkscape* ufraw ufraw-gimp 

Install technical documentation toolchain

dnf install -y pdfedit doxygen doxygen-latex graphviz  docbook5-style-xsl docbook5-schemas docbook2X docbook-style-xsl dblatex texinfo-tex dblatex docbook-utils-pdf  graphviz-python graphviz-R graphviz-graphs graphviz-doc R-core

Install sysadmin

sudo dnf install -y system-config-network system-config-samba system-config-services system-config-users system-config-firewall system-config-httpd

Ben Fry Geospatial

Processing started with a map of zip codes. Since then, Ben Fy has made some extraordinary maps of the United States, a few of which are listed below.

zipdecode
1. zipdecode. How does the postal code numbering work in the United States?

allstreets
2. allstreets.

Bittorrent Notes

Prerequisites

Assuming Fedora 24, with GeoIP installed as

dnf install -y GeoIP GeoIP-devel GeoIP-GeoLite-data GeoIP-GeoLite-data-extra;
dnf install -y libcurl libcurl-devel;
dnf install -y libevent libevent-devel --allowerasing;
dnf install -y intltool;

Test that this is working by using geoiplookup on the command line

%geoiplookup 8.8.8.8
GeoIP Country Edition: US, United States
GeoIP City Edition, Rev 1: US, CA, California, Mountain View, 94035, 37.386002, -122.083801, 807, 650
GeoIP ASNum Edition: AS15169 Google Inc.

Other dependencies: rapidjson, rapidxml for visualization interface.

dnf install -y rapidjson rapidjson-devel rapidxml-devel;

Assuming SSL and crypto support.

dnf install -y openssl openssl-libs openssl-devel;

For the bittorrent protocol, use libtorrent built from source.

In addition to the operating system packages, python support for geolocation is needed. In particular, need bencode, geoip2, requests. So:

pip install GeoIP
pip install GeoIP2
pip install bencode
pip install requests

Background

Part one of Bittorrent distribution research was Fall 2015. Part two is Fall 2016.

Bittorrent. Common terms and jargon.

Basic idea as per The BitTorrent Protocol Specification. Of this, of note is tracker and in particular the tracker scraper protocol.

Magnet links wikipedia entry. PEX, DHT, Magnet links all from lifehacker.

Questions

1. Trackers. List of trackers, announcements, UDP, http.
Fetishizing the most current trackers: 1, 2.

2. Public torrents, Private torrents, SSL torrents. Why are some of these able to be scraped, and others, not so much? From transmission-show -s my.torrent, will get status for public and private, but not ssl torrents.

3. Look at transmission dependencies: openssl, libcurl. See libtransmission includes: transmission.h, variant.h, utils.h. See: struct tr_tracker_stat, tr_torrentTrackers, tr_torrentTrackersFree, tr_info, tr_tracker_info, tr_torrent_activity. Of note, it looks like multiple reads of peers from tracker. TR_PEER_FROM_PEX, TR_PEER_FROM_DHT, TR_PEER_FROM_TRACKER.

4. Look at SSL cert example in libtorrent. See libtorrent github source repository.

5. What kind of tree/node visualizations will work? See The Book of Trees.

Deliverables

One. Data set 1 is four torrents from October 11 for the same recent serial television episode, each a different uploading group: LOL, DIMENSION, AFG, mSD.

Two. Data set 2 is three torrents from October 17, over 8 locales, during 3 time periods.

Three: Prepare for TWD 2016-10-23 Season 7 premiere.

  • visualization: three or more approaches, data types, presentation/exhibition,prototype
  • scraper analytics
  • archival data format
  • persistent scraping, archival interface

Ghostscript on the Mac

Current ghostscript binaries can be found here.  Download and install them on the mac in question to enable Terminal commands that look like:

gs -q -dNOPAUSE -sDEVICE=tiff24nc -r300 -sCompression=lzw -dUseCropBox -sOutputFile=./data/tiff/"$filestem"-%03d.tiff "$IFILE" -c quit;

Postgres 9.4 to 9.5 database migration x Fedora 24

Fedora changed major versions of the postgresql database package, meaning that all user databases have to migrated from the 9.4.x series that used to be the default, to the 9.5.x series that is the current upstream stable package.

Fedora 24 -> 9.5.3
Fedora 23 -> 9.4.5
Fedora 22 -> 9.4.1

The general plan is to use the postgres tool pg_upgrade, ie take a empty database created with the new database, and populate it with the data from the old database.

So, the first step is to install both the current postgres package, and the previous version of the postgres package from the older Fedora release. For this purposes, the 9.4.x series.

Use the PostgreSQL Yum repository. From that index, select PostgreSQL 9.4 running on Fedora 24 – x86_64. The download file is: pgdg-fedora94-9.4-4.noarch.rpm.

rpm -ivh pgdg-fedora94-9.4-4.noarch.rpm

Refresh dnf cache and install postgres 9.4 package.

dnf list | grep postgresql;
dnf install postgresql94-server postgresql94-contrib;

This installs postgres version 9.4 into /usr/pgsql-9.4/bin/postgres.

Via the information page for the repository referenced above:

/usr/pgsql-9.4/bin/postgresql94-setup initdb
systemctl enable postgresql-9.4.service
systemctl start postgresql-9.4.service 

After this, finished with setup. Use this excellent blog post about upgrading, starting with item 3.

Fedora 24 Notes

Install:
San Francisco, California USA

Using x86_64 netinstall images as install media.

Update/Post-Install:

Use RPM Fusion Repositories.

Some basics

dnf install -y unrar tcsh gnome-tweak-tool livecd-tools ncftp lftp

Install KDE, Razor-qt, and MATE Desktops

	 	 
dnf install -y lightdm-gtk-greeter-settings mate-desktop mate-applets network-manager-applet caja mate-themes bluecurve-gtk-themes gnome-themes-legacy;
dnf groupinstall mate-desktop;  
dnf groupinstall kde-desktop;
dnf install -y gnome-themes;

Still a little sketchy, but optionally mate-compiz can be installed:

dnf groupinstall mate-compiz;

Install the basics for VLC

dnf install -y vlc vlc-core vlc-extras;
dnf install -y gaupol mediainfo

Install media xtra codecs.

dnf install -y ffmpeg ffmpeg-devel ffmpeg-compat-devel gstreamer-plugins-bad gstreamer-plugins-ugly gstreamer-plugins-good gstreamer-plugins-good-extras gstreamer-plugins-bad-free gstreamer-plugins-bad-free-extras gstreamer-plugins-bad-nonfree gstreamer-ffmpeg;
dnf install -y k3b-extras-freeworld;

Install negativo.
Use negativo17 repositories for HandBrake, volia.

dnf config-manager --add-repo=http://negativo17.org/repos/fedora-handbrake.repo;
dnf install -y HandBrake-gui HandBrake-cli;
dnf install -y makemkv mkvtoolnix mkvtoolnix-gui;
dnf install -y libdvdcss;

Use makemkv for Bluray.

Install some editors, utilities, and basics

dnf install -y abiword emacs tcsh

Install devel 1

dnf install -y openni openni-devel openni-doc  tbb tbb-devel tbb-doc eigen3-devel eigen3-doc openrdate simple-mtpfs emacs python-BeautifulSoup

Install devel 2

dnf groupinstall -y "Development Tools";
dnf install -y gcc-c++ libstdc++-devel libstdc++-docs boost boost-devel boost-doc valgrind gdb elfutils nemiver dwarves dejagnu emacs expect flex bison cmake git-svn mpfr-devel libmpc-devel lsb;
dnf debuginfo-install glibc;

Install database

dnf install -y glom libpqxx-devel libpqxx postgresql-devel

Install computer vision (hand-build clandmarks)

dnf install -y CImg-devel rapidxml-devel opencv opencv-devel

Install graphic

dnf install -y freetype-freeworld inkscape* ufraw ufraw-gimp 

Install technical documentation toolchain

dnf install -y pdfedit doxygen doxygen-latex graphviz  docbook5-style-xsl docbook5-schemas docbook2X docbook-style-xsl dblatex texinfo-tex dblatex docbook-utils-pdf  graphviz-python graphviz-R graphviz-graphs graphviz-doc R-core

Install sysadmin

sudo dnf install -y system-config-network system-config-samba system-config-services system-config-users system-config-firewall system-config-httpd

Multi Channel Display [2,3,4] vs. NUC Notes

Hardware

Intel NUC 6i5SYK (i5-6260U 1.8GHz, Mini DisplayPort 1.2, HDMI 1.4b, Iris 540)
Samsung 850 EVO 250 GB M.2 SSD
Kingston HyperX 16GB DDR4

 

Display
1 x Sony VPL-FHZ55 projector
1 x 1920 x 1080 display AOC
2 x 1920 x 1080 Samsung TV
2 x 1920 x 1600 Dell 2407 LCD
Club 3d Multi Stream Transport Hub
1 x HDMI M to DVI M cable
1 x HDMI M to HDMI M 3M
1 x DisplayPort M to Mini DisplayPort M
2 x DisplayPort M to HDMI M

 

Software

Fedora 23 as per previous notes.

Then, either update xorg-x11-drv-intel to 2016 drop by cannibalizing F24 development, or install via Intel Graphics Installer for Linux. See Intel Open Source site, ie 01.org.

Some useful background information from the Fedora Project on How to Debug Xorg Problems.

To probe the current graphics situation, run

grep " (--) " /var/log/Xorg.0.log

Other useful queries

grep " (II) " /var/log/Xorg.0.log;
grep " (II) " /var/log/Xorg.0.log | grep connected;
grep " (II) " /var/log/Xorg.0.log | grep drm;
grep " (II) " /var/log/Xorg.0.log | grep EDID;

 

On the NUC, the above gives the following output

[bkoz@otis log]$ grep " (II) " /var/log/Xorg.0.log | grep EDID;
[    17.236] (II) VESA(0): EDID Version: 1.3
[    17.237] (II) VESA(0): EDID (in hex):
[    17.237] (II) VESA(0): EDID vendor "AOC", prod id 9063
[    17.237] (II) VESA(0): Using EDID range info for horizontal sync
[    17.237] (II) VESA(0): Using EDID range info for vertical refresh

[bkoz@otis log]$ grep " (--) " /var/log/Xorg.0.log
[    17.157] Markers: (--) probed, (**) from config file, (==) default setting,
[    17.165] (--) PCI:*(0:0:2:0) 8086:1926:8086:2063 rev 10, Mem @ 0xde000000/16777216, 0xc0000000/268435456, I/O @ 0x0000f000/64, BIOS @ 0x????????/131072
[    17.223] (==) VESA(0): Depth 24, (--) framebuffer bpp 32
[    17.284] (--) VESA(0): Virtual size is 1920x1080 (pitch 1920)

 

Test matrix

SW Connection-Display-Driver Status
F23 1 x HDMI 1080p vesa OK
F23 1 x Mini DisplayPort 1080p vesa visible but clipped

 

Future Attempts

Hand-rolled xorg.conf files. See How to create xorg.conf.

Windows 10 is supported.

Can MacOS + mini drive 4 x displays?