technology | sunglint

Protected: Mozilla Android 2024

2024/05/07

Filed under technology Tagged with android

Protected: LoRA Notes

2023/12/21

Filed under art, research, technology Tagged with genai, lora

Deeper Still: Stable Diffusion Notes

2023/12/01

Run Stable Diffusion on linux, use Fedora 37 baseline.

Installation notes. Nothing to it.

Prompt creation is the creative zone right now.

Midjourney

See Nick St. Pierre (on twitter as nickfloats) for some prompt hacking for midjourney. His midjourney prompts look like:

cinematic, cowboy, hillside, high noon, gritty, grainy film negative, high contrast, burnt orange earth & periwinkle skies --ar 4:5 --style raw --stylize 420 --weird 69 --chaos 13 --turbo --repeat 20

Stable Diffusion

A couple of different UIs for this.

easy diffusion
automatic1111, then http://127.0.0.1:7860/

Basic models and key ideas, List of different custom stable diffusion models, including Hugging Face’s stable-diffusion-2, and sdxl. The AUTOMATIC1111 model page is perhaps the most useful, especially the parts about yaml config files for the 2x models.

Huggingface.co, prompt generator,

Prompt guide, fotor, beginners guide, art guide,

underwater temple, (ancient city of atlantis), high detail, majestic city, dramatic scale, 8k, blue, fish, (coral reef), Greek temple, Greek architecture, godrays, cinematic lighting, concept art, distinct, in the style of studio ghibli

Use negative prompts to remove unwanted things. So, if you want to remove “mustache” in the image, set the negative prompt to “mustache.”

Use (increase strength) and [reduce strength]

hyperrealistic

resolution = unreal engine, sharp focus, 8k, vray

dramatic, expansive, low angle, psychedelic

subject == scene description

medium == “digital painting”, “portrait”, “concept art”, “ultra realistic illustration”, underwater portrait, underwater steampunk

artist == by NAME

details chain with , DETAIL

Playground

for making prompts for multiple AI generators

RunwayML

web-ui for inpainting, etc.

civitai

Library of models and prompts

To make your own models, see this workflow post.

LLAMAFILE

here is a product from Moz that lets you use “executable” LLM files that launches a chat interface on a local webbrowser session to run jobs.

Filed under art, technology Tagged with midjourney, stability, stable diffusion

Notes on Large Format Laser

2023/10/26

Approximating the width of a copper plate means the canonical metal type is 18 gauge mirror-finish copper (aka 1.02mm), closest available acrylic that works is 1/16th inch (aka 1.5mm).

Vector cut lines are 100% R, and 0.025mm stroke (0.066mm works in practice, which is .18pt).

There are a variety of acrylics

1.5mm transparent clear gloss (2022)
1.5mm silver mirror with mirror side semi-matte (2023)
1.5mm frosted/matte
1.2mm silicone in grades medium to hard. ~~Get medium-soft~~. Get hard.

Remember, images (text especially) have to be reversed. (If the object is going to be displayed with the laser-cut side face-down).

Star-X v11, 2022

This was two horizontal halves of acrylic, a top half that was 22×30 and a bottom half that was 22×30, that when put together formed a 44×30 inch work. There is a vector cut overlay for the center star. And two 300 dpi PNG images that have not been flattened, with red vector lines representing vector cuts, and black lines part of it that would be rasterized.

fiber linesize: .50pt, 100% black line
satellite triangle radius: 4.6px, red cut line
tor exit nodes: .2pt, 55% black
ix: .188 pt, 100% black

Star-X Southern, v12-13, summer 2023

But there is also the art of getting a whole mess of grouped vector paths to print out out a laser printer. First, ungroup into. This attempt uses paths directly, instead of PNG groups. It is a layered SVG with each layer having some impact.

fiber linesize: .55pt 100% black line
satellite triangle radius: 4.2px, red cut line
tor exit nodes: .263pt, 25% black
ix: .63pt, 100% black
graticule inner: .35mm / 1pt, dashes pattern 13
graticule edge: .71mm / 2pt pattern 14

Antarctica imaged as 50% gray, which was not enough. Should this and the north pole/star be cut?

If this is to be a woodcut, then the base station triangles should not be cutouts. This is more problematic this year, as there are more ground-stations in the Americas, and if they are cut out then they will make the whole plate unstable. Should they be smaller?

Star-X v14.8, winter 2023

This is for laser etching (reverse) on the back of a mirrored acrylic (1.5mm) at 44 x 30 size. It’s broken into two halves vertically, a top half and a bottom half. These are tile-able with a 180 degree turn, and (a stylized) cabling route runs through the Pacific and Atlantic oceans.

fiber linesize: .2pt (.071mm), 100% black line
satellite triangle radius: 3.6px, 100% black fill
tor exit nodes: .263pt, 25% black
ix: .63pt, 100% black
graticule inner: .35mm / 1pt, pattern 16
graticule outer: .71mm / 2pt pattern 14

Star-X v14.9, valentine 2024

This is for laser etching (reverse) on the back of a silicone hard square (1.5mm) at 6×6 size.

fiber linesize: .55pt, 100% black line
satellite triangle radius: 3.6px, 100% black fill
tor exit nodes: .263pt, 25% black
ix: .63pt, 100% black
graticule inner: 0.75pt, pattern 12
graticule outer: .71mm / 2pt pattern 14

See setup hints from Pagoda Arts. Cut lines are styled as Red, 0.025mm stroke or smaller, aka 0.02.

Final production file format is PDF. Created via rasterizing vector layers (black/white) at 600 dpi, and then slotting raster layers in to new svg file as imported PNG. This is the production file SVG, with rasterized layers for cable, satellite, tor exit nodes, and ix. Cut layers are vector. Export the production file SVG raster and vector layers as a single pdf, run as a raster job on the laser cutters. Basically, this is pre-rasterizing the vector files on the beefy linux workstations so that what is presented to the laser imaging doesn’t blow them up with (linux-workstation run by madman) complexity.

NB: This is the thinnest vector line possible. On some angles and bezier curves, this width line may not cut through all of the mirror backing. Probably .25 is better, which means that inkjet resolution for visualization remains the highest-resolution.

NB: Maybe also vector grouping for drop-outs? Ungroup, convert? Or just hitting a machine complexity limit?

Filed under art, research, technology Tagged with acrylic, laser, laser cutting, laser etching, silicone, vector

robots.denai

2023/10/24

Background

Crawling and scraping networks have a long history that precedes the web. For example, modems connected to telephone networks scanned a range of numbers and marked which ones were fax machines. This mapping of the network is the essence of crawling: automated systems set up by humans that attempt to measure (aka “crawl” from the top of a list of addresses to the bottom) the web-based on some per-determined criteria. Scraping is just crawling and then saving the information.

Google has crawled the public web since 1998. Facebook does not allow scraping of its own sites, but contracts out scraping to others. The practice remains controversial. At the technical level, there is a tension between providing an API for structured data querying, and trying to restrict users who want to answer questions outside of the API or prefer the direct efficiency of scraping. Post scraping, the outputs are used to build indexes or summarize news headlines in ways that may violate copyright or terms of service.

While these legal disagreements continue to play out, on the technical side system administrators developed a solution (starting in 1994) using a simple file called robots.txt, and developed into a web standard called Robots Exclusion Protocol (IETF 9309). This is a file at the root level of a website that asks machines reading the site to please not read particular directories, or perhaps more sternly, asks machines not to read the site at all. A better definition of machines would be computers running software to copy or detail the contents and structure of the website for analysis by search engines, or as use as inputs for generative ai, or for any of a million reasons. Robots, say. Scrapers, say. People want to know more about the internet, or a particular site.

The contents of this file are in public, and don’t need logins or any special access for humans or robots to read.

Example robots.txt files

Example Generative Scraper Opt-Outs

Open AI, GPTBot, since 2023-08-07

CCBot, Common Crawl, List of Naughty Obscene… Words

Protected: Moz Browsertime (desktop linux a11y test host)

2023/09/29

Filed under technology

Fedora 38 Notes

2023/09/28

Install:
San Francisco, California USA

Using x86_64 netinstall images as install media, workstation iso to usb.

Update/Post-Install:

Some basics

dnf install -y tcsh fish emacs livecd-tools nload gparted util-linux-user at
dnf install -y nfsv4-client-utils nfs4-acl-tools nfs-utils-coreos nfs-utils

Install for GNOME, and tweaks

sudo dnf install -y gnome-tweaks gnome-shell-extension-common gnome-shell gnome-shell-extension-user-theme gnome-shell-extension-dash-to-dock gnome-shell-extension-apps-menu gnome-shell-extension-places-menu

Install KDE, and the various GNOME 2ish desktops aka MATE and Cinnamon Desktops

dnf groupinstall "GNOME Desktop Environment" "KDE Plasma Workspaces" "MATE Desktop" "Cinnamon Desktop"

Install tailscale

sudo systemctl enable --now tailscaled sshd;

sudo tailscale up

Set default groups, users, limits.conf settings. Here’s how to set up your ssh keys.

Use RPM Fusion Repositories.

Install the basics for media and VLC

dnf install -y vlc vlc-core vlc-extras;
dnf install -y gaupol mediainfo

Install media xtra codecs.

dnf install -y ffmpeg ffmpeg-devel gstreamer-plugins-bad gstreamer-plugins-ugly gstreamer-plugins-bad-nonfree gstreamer-ffmpeg;

Install negativo.
Use negativo17 repositories for HandBrake or MakeMKV, volia. Use these directions for MakeMKV.

dnf config-manager --add-repo=http://negativo17.org/repos/fedora-handbrake.repo;
dnf install -y qt5-qtbase qt5-qtbase-devel
dnf install -y HandBrake-gui HandBrake-cli;
dnf install -y makemkv mkvtoolnix mkvtoolnix-gui;
dnf install -y libdvdcss;

Use makemkv for Bluray.

Install some editors, utilities, and basics

Install devel 1

dnf install -y openni openni-devel openni-doc  tbb tbb-devel tbb-doc eigen3-devel eigen3-doc openrdate

Install devel 2

dnf groupinstall -y "Development Tools";
dnf install -y python3-beautifulsoup4
dnf install -y gcc-c++ libstdc++-devel libstdc++-docs boost boost-devel boost-doc valgrind gdb elfutils nemiver dwarves dejagnu emacs expect flex bison cmake git-svn mpfr-devel libmpc-devel lsb cmake ninja-build;
dnf debuginfo-install glibc;

Install database

dnf install -y glom libpqxx-devel libpqxx postgresql-devel

Install computer vision (hand-build clandmarks and opencv if optimized)

dnf install -y CImg-devel rapidxml-devel rapidjson-devel jq opencv opencv-devel

Install graphic

dnf install -y freetype-freeworld inkscape* ufraw ufraw-gimp fuse-exfat

Install technical documentation toolchain

dnf install -y pdfedit texlive-pdfjam doxygen doxygen-latex graphviz  docbook5-style-xsl docbook5-schemas docbook2X docbook-style-xsl dblatex texinfo-tex dblatex docbook-utils-pdf  graphviz-python graphviz-R graphviz-graphs graphviz-doc R-core

dnf install -y pdfarranger ghostscript ghostscript-tools-printing ghostscript-tools-fonts

See separate posts on getting Nvidia and TensorFlow optimized CUDA up.

Issues

Set ulimits higher. Modify /etc/security/limits.cof like so:

*** limits.conf.f25	2016-12-11 23:27:46.609458649 -0800
--- limits.conf	2016-12-15 20:16:57.289245497 -0800
***************
*** 52,57 ****
--- 52,58 ----
  
  #*               soft    core            0
  #*               hard    rss             10000
+ 
  #@student        hard    nproc           20
  #@faculty        soft    nproc           20
  #@faculty        hard    nproc           50
***************
*** 59,61 ****
--- 60,67 ----
  #@student        -       maxlogins       4
  
  # End of file
+ 
+ bkoz               hard    nofile	64000
+ bkoz               soft    nofile       64000
+ 
+

Filed under technology Tagged with 38, fedora, linux

Python 3.10/3.11 Side-by-Side Notes

2023/07/24

F38 defaults to Python 3.11.4, so the “python” and the “python3.11” commands are the same.

To use Python 3.10.x on this system:

sudo dnf install -y python3.10-libs python3.10-debug python3.10-devel python3.10

and then

python3.10

Launches a Python 3.10.12 shell.

To use a virtual environment set to this python version,

mkdir moz-python

(make sure you are in the BASH shell, say by typing the command “bash“)

python3.10 -m venv moz-python

source moz-python/bin/activate

Filed under technology Tagged with python, runtime versioning

GeoIP2 expanded fields: continents and country registered

2023/07/17

Base platform is fedora 38

sudo dnf install -y libmaxminddb libmaxminddb-devel

sudo dnf install -y geolite2-city geolite2-country geolite2-asn

download geolite2++, unpack

ccmake . to configure

make

Here are the various APIs to GeoIP2. And some idea of the continental slices. Background IP Geolocation Data for schema fields and metadata.

For the C++ API, use GeoLite2++ and here are the docs. Just use the pre-built binary from rizal.

To purchase a one-time GeoIP2 City-level database in binary format from MaxMind, go here. Stash the updated files in /usr/share/GeoIP.

The source interface is in the alpha60 file “a60-geo-maxmind-2.h”, and uses the following databases directly:

“/usr/share/GeoIP/GeoLite2-Country.mmdb”

“/usr/share/GeoIP/GeoLite2-City.mmdb”

“/usr/share/GeoIP/GeoLite2-ASN.mmdb”

Filed under research, technology Tagged with c++, geo, ip, linux

Mermaid x Markdown

2023/03/31

Doxygen integration, maybe?

But Markdown with Mermaid, structured in pages, is perfect and an easy way to make code flow diagrams.

Some Mermaid basics:

This gets translated into markdown like:

  ```mermaid
graph TD;
      A-->B;
      A-->C;
      B-->D;
      C-->D;
```

Here’s an example for doxygen.

Filed under technology Tagged with documentation, mermaid

chrome/chromium CLI

2022/11/21

The list of chrome CLI settings

Filed under technology Tagged with chrome, web automation

Protected: Influx/Grafana Docker Setup

2022/10/04

Filed under technology Tagged with graphana, influx, influx v2

Cahill Keyes Star-X

2022/06/27

The data sources and iconography used are as follows:

Fiber nodes, submarine cables, from TeleGeography (thin black lines)
Starlink Satellite Gateways (black triangles)
Internet Exchanges (purple x)
Tor Exit Nodes (blue ray clusters)
Tor Exit Nodes With Obfuscated Geolocation (larger green ray clusters)

The Star-X projection is based on the Cahill-Keyes Butterfly M projection as implemented in the D3.js carto package, as show in the Visionscarto base map. This projection geometry is mostly accurate with limited distortion, and has been associated with decades of anti-nuclear and utopian thinking. The Cahill-Keyes Butterfly M projection is landscape (horizontal) in orientation, with four horizontal octets or sections – quarto one is a north and south octet paired– that starts in the Pacific Ocean by New Zealand to the coast of North America as the first quarto, and then the second quarto is the rest of of the Americas, the third quarto is the Atlantic Americas to Africa and Europe, and the fourth quarto is Asia to Oceania.

The Star-X projection is to cut up the four quartos as above and stack them on top of each other. Spatially, this is a look “down” at the globe, as if from space but over the North Pole. Take the first two quartos as the southern vertical half of the projection, where the northern vertical half is composed of rotating the third and fourth quartos about the North Pole, such that they are upside-down and on top of the southern half. In the center of the page is a star geometric shape that marks the position of the North Pole.

Here’s a smaller map that shows what this rotation and tessellation looks like when the continents and land masses are obvious.

Besides this obsessive geometric symmetry, the fiber routes run off the globe, falling off the cartography and being pulled to the sides as if from a force much greater than known gravity.

The size of this map almost exactly matches the poster given away at SFMOMA by Felix Gonzalez-Torres “Untitled”, 1992/1993, 29 x 44

The Star X projection as manifest in 2022 is made of 4 components, each 22×17 inches portrait. The total image size is 44×34 inches, and with a 1 inch matt, 46 x 36.

See the following diagram of plate location and orientation, with plate numbers in red.

—

Filed under art, research, technology

Protected: Grafana Subqueries and Transforms

2021/06/10

Filed under research, technology Tagged with grafana

Protected: Grafana Annotations

2021/06/04

Filed under technology

Mozilla Android Testing Workflow 2022

2021/02/16

Workflow Outline

Use the same host, devices, device setup as Moz FNPRMS Notes, binaries. Through trial and error, have found that Fenix nightly builds are most reliably fetched after 10am PST, so adjust the schedule to reflect this.

Get daily test data: schedule daily runs from chron
- FNPRMS x 4 devices x 2 browsers (Fenix Nightly and Chrome)
  - 10:00 Pixel 4xl
  - 11:00 Pixel 2
  - 12:00 Samsung S10
  - 13:00 Samsung S7
  - 14:00 process results
- Browsertime x 4 devices x 2 browsers (Fenix Nightly and Chrome)
  - 15:00 Pixel 4xl
  - 17:00 Pixel 2
  - 19:00 Samsung S10
  - 21:00 Samsung S7
  - 23:00 process results
FNPRMS
- download current fenix nightly 64 bit from treehearder
- convert apk to debug build
- remove old binary, install new fenix nightly debug binary
- run through tests per device
- collate data
  - check in to github
  - insert into influx db for grafana dashboard
Browsertime
- /opt/mozilla/browsertime.git/run-through-sites.sh fenix-nightly.sh sites.txt
- /opt/mozilla/browsertime.git/run-through-sites.sh chrome.sh sites.txt
- output is:
  - - browsertime-results-2020-06-22-fenix-nightly
      - browsertime-fenix-nightly-cnn.com.json
      - browsertime-fenix-nightly-en.m.wikipedia.org.json
      - browsertime-fenix-nightly-m.imdb.com.json
      - browsertime-fenix-nightly-stackoverflow.com.json
      - browsertime-fenix-nightly-support.microsoft.com.json
      - browsertime-fenix-nightly-tripadvisor.com.json
      - browsertime-fenix-nightly-www.amazon.com.json
      - browsertime-fenix-nightly-www.youtube.com.json
    - browsertime-results-2020-06-18-chrome
      - browsertime-fenix-nightly-cnn.com.json
      - browsertime-fenix-nightly-en.m.wikipedia.org.json
      - browsertime-fenix-nightly-m.imdb.com.json
      - browsertime-fenix-nightly-stackoverflow.com.json
      - browsertime-fenix-nightly-support.microsoft.com.json
      - browsertime-fenix-nightly-tripadvisor.com.json
      - browsertime-fenix-nightly-www.amazon.com.json
      - browsertime-fenix-nightly-www.youtube.com.json
- Extract metrics from json each day, from each .json file above
- Convert to from json to csv
- Convert from 2 browsers, 2 csv files to svg glyph for each of 8 above
- Add metrics to generate SVG glyph to influx
- Display glyph with SVG custom grafana plugin

Notes

Extract stack:

browsertime-to-influx-with-1-metric-list.sh 1 fenix-nightly (2)
transform-1-metric-cosmology-to-influx.sh 1 fenix-nightly RDIR XTRACT_LIST (4)
copy-json-files-to-one-dir.sh
(for each browsertime JSONFILE)
moz-perf-x-extract.browsertime_url.exe JSONFILE
moz-perf-x-extract.browsertime_url.exe JSONFILE XTRACT_LIST
(for each metric in extracted CSV file)
csv-index-by-line-and-field.sh

Filed under technology

Protected: geckodriver Notes

2021/02/16

Filed under technology

Fedora 32 Notes

2020/09/16

Install:
San Francisco, California USA

Using x86_64 netinstall images as install media.

Update/Post-Install:

Use RPM Fusion Repositories.

Some basics

Install the Material Shell for GNOME, and tweaks

sudo dnf install -y gnome-tweak-tool chrome-gnome-shell-10.1-12.fc34.x86_64 gnome-shell-extension-background-logo-40.0~rc-2.fc34.noarch gnome-shell-extension-material-shell-12-2.fc34.noarch gnome-shell-theme-flat-remix-0.0.20210623-1.fc34.noarch gnome-shell-40.3-1.fc34.x86_64 gnome-shell-extension-common-40.3-1.fc34.noarch gnome-shell-extension-apps-menu-40.3-1.fc34.noarch gnome-shell-extension-launch-new-instance-40.3-1.fc34.noarch gnome-shell-extension-places-menu-40.3-1.fc34.noarch gnome-shell-extension-window-list-40.3-1.fc34.noarch gnome-shell-extension-user-theme-40.3-1.fc34.noarch gnome-shell-extension-windowsNavigator-40.3-1.fc34.noarch gnome-shell-extension-workspace-indicator-40.3-1.fc34.noarch gnome-shell-extension-dash-to-dock-69-8.2021706gite4beec8.fc34.noarch

dnf install -y tcsh emacs livecd-tools ncftp lftp

Install KDE, and the various GNOME 2ish desktops aka MATE and Cinnamon Desktops

dnf groupinstall mate-desktop; dnf groupinstall kde-desktop; sudo dnf install @cinnamon-desktop-environment dnf install -y gnome-themes;

Optionally mate-compiz can be installed:

dnf groupinstall mate-compiz;

Install the basics for media and VLC

dnf install -y vlc vlc-core vlc-extras;
dnf install -y gaupol mediainfo

Install media xtra codecs.

dnf install -y ffmpeg ffmpeg-devel ffmpeg-compat-devel gstreamer-plugins-bad gstreamer-plugins-ugly gstreamer-plugins-good gstreamer-plugins-good-extras gstreamer-plugins-bad-free gstreamer-plugins-bad-free-extras gstreamer-plugins-bad-nonfree gstreamer-ffmpeg;
dnf install -y k3b-extras-freeworld;

Install negativo.
Use negativo17 repositories for HandBrake or MakeMKV, volia. Use these directions for MakeMKV.

dnf config-manager --add-repo=http://negativo17.org/repos/fedora-handbrake.repo;
dnf install -y qt5-qtbase qt5-qtbase-devel
dnf install -y HandBrake-gui HandBrake-cli;
dnf install -y makemkv mkvtoolnix mkvtoolnix-gui;
dnf install -y libdvdcss;

Use makemkv for Bluray.

Install some editors, utilities, and basics

dnf install -y abiword

Install devel 1

dnf install -y openni openni-devel openni-doc  tbb tbb-devel tbb-doc eigen3-devel eigen3-doc openrdate simple-mtpfs emacs python-BeautifulSoup

Install devel 2

dnf groupinstall -y "Development Tools";
dnf install -y gcc-c++ libstdc++-devel libstdc++-docs boost boost-devel boost-doc valgrind gdb elfutils nemiver dwarves dejagnu emacs expect flex bison cmake git-svn mpfr-devel libmpc-devel lsb;
dnf debuginfo-install glibc;

Install database

dnf install -y glom libpqxx-devel libpqxx postgresql-devel

Install computer vision (hand-build clandmarks and opencv if optimized)

dnf install -y CImg-devel rapidxml-devel opencv opencv-devel

Install graphic

dnf install -y freetype-freeworld inkscape* ufraw ufraw-gimp fuse-exfat

Install technical documentation toolchain

dnf install -y pdfedit texlive-pdfjam doxygen doxygen-latex graphviz  docbook5-style-xsl docbook5-schemas docbook2X docbook-style-xsl dblatex texinfo-tex dblatex docbook-utils-pdf  graphviz-python graphviz-R graphviz-graphs graphviz-doc R-core

dnf install -y pdfarranger

See separate posts on getting Nvidia and TensorFlow optimized CUDA up.

For rendering svg, use both inkscape and cairosvg. To install the latter, do

pip3 install cairosvg

Issues

Something wrong with file indexing, it is consuming gigabytes of memory. Turn off search in the Gnome menu, but this is insufficient. Instead, use

balooctl disable

Set ulimits higher. Modify /etc/security/limits.cof like so:

*** limits.conf.f25	2016-12-11 23:27:46.609458649 -0800
--- limits.conf	2016-12-15 20:16:57.289245497 -0800
***************
*** 52,57 ****
--- 52,58 ----
  
  #*               soft    core            0
  #*               hard    rss             10000
+ 
  #@student        hard    nproc           20
  #@faculty        soft    nproc           20
  #@faculty        hard    nproc           50
***************
*** 59,61 ****
--- 60,67 ----
  #@student        -       maxlogins       4
  
  # End of file
+ 
+ bkoz               hard    nofile	8092
+ bkoz               soft    nofile       8092
+ 
+

In addition, it seems as if there is a bug with nmcli when using the Mate window manager. Mostly, it doesn’t work (along with nm-applet when used in a programmatic fashion from C/C++ code. So, use GNOME, as much as it hurts.

Filed under technology Tagged with f32, fedora, fedora 32, linux

Bittorrent Notes v2

2020/08/28

Prerequisites

OS: Assuming Fedora [32] or Centos 8
HW: x86/64, i5, 16GB memory
NET: 10TB/month

For AWS, this is m4.xlarge or m3.xlarge with 21 in / 100 out Gbytes per week for a 96-torrent 2-hr scraper. (aka, got-705). Expect perhaps twice nework capacity this to saturate sample.

Cloud dedicated host or virtual server footprint: HP ProLiant DL120, Centos8, 10TB/month.

Development machine is intel NUC Kit: Intel NUC NUC10i7FNH, 64GB, Fedora 32.

Software Setup

Development baseline is Fedora 32, targeting Centos 8. To make sure that the base software packages are the same for both platforms, here are the updated C++ packages that need to be installed for Centos 8.

- g++: 10.2
- boost: 1.69
- libtorrent: 2.0.0 RC3

To bootstrap gcc, download the gcc-10.2 release from gcc.gnu.org, and configure for c/c++. Install prequisites. Use this to configure, then make, install:

sudo dnf install -y mpfr-devel gmp-devel
wget config.http://repo.okay.com.mx/centos/8/x86_64/release/libmpc-devel-1.0.2-9.el8.x86_64.rpm
sudo dnf localinstall ./libmpc-devel-1.0.2-9.el8.x86_64.rpm

../gcc-10.2.0/configure --verbose --prefix=/home/bkoz/bin/H-x-gcc-10.2 --enable-languages=c,c++,lto --enable-shared --disable-multilib --enable-threads=posix --enable-checking=release --with-system-z-lib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --with-gcc-major-version-only --with-linker-hash-style=gnu --enable-plugin  --enable-gnu-indirect-function --enable-cet --with-tune=native

Set this install prefix in the active shell’s PATH (bash/tcsh), and same for the library path and LD_LIBRARY_PATH. Source that, and make sure that

which g++

points at the right compiler.

To rebuild boost 1.69.x on Centos 8, download the source RPM on F32, transfer it to the Centos 8 machine, and unpack it. Something like this:

wget https://download-ib01.fedoraproject.org/pub/fedora/linux/updates/testing/32/Everything/SRPMS/Packages/b/boost-1.69.0-15.fc32.src.rpm

Install some prerequisites:

dnf install -y cmake bzip2-devel python3-devel libicu-devel  openmpi-devel  mpich-devel python3-numpy

To rebuild boost, a little bit of fun in the rpmbuild directory’s SPEC subdirectory. First, edit the boost.spec file to work around the plugin flags for annobin that only work with the system compiler. Easy as:

diff --git a/boost.spec.orig b/boost.spec
index 6f9f3d9..303a31a 100644
--- a/boost.spec.orig
+++ b/boost.spec
@@ -6,6 +6,7 @@
 # We should be able to install directly.
 %global boost_docdir __tmp_docdir
 %global boost_examplesdir __tmp_examplesdir
+%undefine _annotated_build
 
 %if 0%{?flatpak}
 # For bundling in Flatpak, currently build without mpich and openmpi,
@@ -138,7 +139,7 @@ Patch65: boost-1.66.0-build-optflags.patch
 Patch82: boost-1.66.0-no-rpath.patch
 
 # https://bugzilla.redhat.com/show_bug.cgi?id=1541035
-Patch83: boost-1.66.0-bjam-build-flags.patch
+#Patch83: boost-1.66.0-bjam-build-flags.patch
 
 # https://bugzilla.redhat.com/show_bug.cgi?id=1673669
 Patch84: boost-1.69-random.patch
@@ -650,7 +651,7 @@ find ./boost -name '*.hpp' -perm /111 | xargs chmod a-x
 %patch62 -p1
 %patch65 -p1
 %patch82 -p1
-%patch83 -p1
+#%patch83 -p1
 %patch84 -p2
 %patch85 -p2
 %patch86 -p1
@@ -666,7 +667,7 @@ PYTHON3_ABIFLAGS=$(/usr/bin/python3-config --abiflags)
 # There are many strict aliasing warnings, and it's not feasible to go
 # through them all at this time.
 # There are also lots of noisy but harmless unused local typedef warnings.
-export RPM_OPT_FLAGS="$RPM_OPT_FLAGS -fno-strict-aliasing -Wno-unused-local-typedefs -Wno-deprecated-declarations"
+export RPM_OPT_FLAGS="-O2 -g -fexceptions -fno-strict-aliasing -Wno-unused-local-typedefs -Wno-deprecated-declarations"
 export RPM_LD_FLAGS
 
 cat > ./tools/build/src/user-config.jam << "EOF"

Then build.

Note that this is easier for boost-1.73.x versions, which can be done with just the RPM_OPT_FLAGS bits above (Some fun with: RPM_OPT_FLAGS=”-std=gnu++17 -O2 -g -march=native -pthread -fPIC -fexceptions). Holla!

rpmbuild -ba boost.spec

Then install the generated RPMS.

Get some systems diagnostic tools for monitoring network load, VPN support, nmcli, etc.

dnf install -y nload NetworkManager NetworkManager-openvpn openvpn

GeoIP installed as

dnf install -y GeoIP GeoIP-devel GeoIP-GeoLite-data GeoIP-GeoLite-data-extra;
dnf install -y libcurl libcurl-devel;
dnf install -y libevent libevent-devel --allowerasing;
dnf install -y intltool;

Test that this is working by using geoiplookup on the command line

%geoiplookup 8.8.8.8
GeoIP Country Edition: US, United States
GeoIP City Edition, Rev 1: US, CA, California, Mountain View, 94035, 37.386002, -122.083801, 807, 650
GeoIP ASNum Edition: AS15169 Google Inc.

For output, one will need a copy of the submarine JSON data files, and some specialized packages for converting from markdown format to svg, pdf:

dnf install -y pandoc pandoc-common pandoc-pdf

Other dependencies: rapidjson, rapidxml for visualization interface.

dnf install -y rapidjson rapidjson-devel rapidxml-devel;

Assuming SSL and crypto support.

dnf install -y openssl openssl-libs openssl-devel;

For the bittorrent protocol, use libtorrent built from source.

Install data science command line: json2csv, jq. NB need GOPATH set in the environment and GOPATH/bin set in PATH.

dnf install -y golang golang-github-bmizerany-assert-devel.noarch;
go get github.com/jehiah/json2csv;
dnf install -y jq;

Install fonts for display of a wide variety of character sets.

# sans
dnf install -y adobe-source-han-sans-cn-fonts adobe-source-han-sans-tw-fonts adobe-source-han-sans-jp-fonts adobe-source-han-sans-kr-fonts

# western
dnf install -y adobe-source-code-pro-fonts adobe-source-sans-pro-fonts adobe-source-serif-pro-fonts

Also, install Apercu.

Finally, install inkscape and cairosvg, as primary and backup SVG render engines, for transforming to PNG and PDF. And ffmpeg, for PNG to MKV and MP4. And ghostscript, for transforming PDF to PDF.

dnf install -y inkscape;
dnf install -y ffmpeg-libs ffmpeg;
dnf install -y ghostscript;
pip install cairosvg;

Required System Configuration

Remove rpcbind for datacenter use as per BUND warning. See disable portmapper notes.

Have to set ulimits to unlimited (min 16k) because of open file limits in seeding mode.

Standard timezone is Berlin, Germany, CEST.

timedatectl set-timezone Europe/Berlin

Install at, so that chron can be a bit more liquid.

yum install -y at
systemctl enable atd.service
systemctl start atd.service

Background

Part one of Bittorrent distribution research was Fall 2015. Part two is Fall 2016. Part three is Spring & Summer 2017. Part four is 2018.

Bittorrent. Common terms and jargon.

Basic idea as per The BitTorrent Protocol Specification. Of this, of note is tracker and in particular the tracker scraper protocol.

Magnet links wikipedia entry. PEX, DHT, Magnet links all from lifehacker.

Questions

1. Trackers. List of trackers, announcements, UDP, http.
Fetishizing the most current trackers: 1, 2.

2. What kind of tree/node visualizations will work? See The Book of Trees.

Filed under technology Tagged with bittorrent, c++17, c++20, dht, documentation, libcurl, libtransmission, node, pex, private trackers, radial, scrape, ssl torrents, system administration, torrent, tracker, tree, visualization

Moz Browsertime Notes

2020/06/09

Browsertime is a multi-vendor test harness for testing web browser performance. Mozilla uses it to gauge desktop and mobile performance. This post details running browsertime on a linux host, targeting desktop browsers. The URL site list is derived from TP6. Should be edited and observed every 3-6 months or so to make sure no popups, no logins, etc.

Setup

Four options, two to use the top-level sources, and two to use what’s in Mozilla’s tree.

ONE (local js harness)

2023 F38 base

checkout git repo

cd /home/bkoz/src/browsertime.git
sudo npm install -g

This installs an older geckodriver, so instead

cd /home/bkoz/src/browsertime.git/node_modules/@sitespeed.io/geckodriver/vendor
cp geckodriver-v0.29 .
mv geckdriver geckodriver-v0.27
ln -s geckodriver-v0.29 geckodriver

volia.

To check visual metrics are working:

/home/bkoz/src/mozilla-gecko.git/obj-x86_64-pc-linux-gnu/_virtualenvs/init_py3/bin/python /home/bkoz/src/mozilla-gecko.git/tools/browsertime/node_modules/browsertime/browsertime/visualmetrics.py --check

Notably the python PIL and SSIM_PIL modules are required.

sudo pip3 install pyssim

and later versions use execa, so npm install -g execa versions over 117 need bidi

npm install ff-test-bidi-har-export

When updating between versions of browsertime, make sure to re-build the node dependencies. Aka, in the browsertime source dir:

rm -rf node_modules package_lock.json
npm cache clean –force
npm i –force

TWO (local docker) Use local docker setup.

Change to the browsertime source directory, and verify that the Dockerfile exists.

build current docker image, use the script:
$MOZPERFBT/docker-build-mozilla-browsertime.sh
get fresh firefox-nightly binaries daily, and unpack in /opt/mozilla/bin/firefox-nightly with an alias of firefox-nightly.exe pointing to the actual binary.
This is a cron script and runs once a day at 10am PST: download-linux-firefox-nightly-url.sh
Do 15 iteration test run

run-once-per-day.linux.sh, uses common_browsertime_options.sh
run chrome with specified site list and browsertime options
run firefox with specified site list and browsertime options
crontab entry

Many of the dashboards require a HAR file and post-processing with pagexray. So, make sure that the browsertime options include the --har command-line option.

archive results to /opt/mozilla/data/browsertime-results/2022-0m-0d-platform

THREE (local gecko build directory with mach)

In a configured/built mozilla central repository

./mach browsertime --setup

Gives:

Installing browsertime for mach using "/home/bkoz/.mozbuild/node/bin/node /home/bkoz/.mozbuild/node/bin/npm ci --loglevel=error --scripts-prepend-node-path"...

browsertime installed successfully!

browsertime binary is at 
(m-c)/tools/browsertime/node_modules/.bin/browsertime

On Fedora 33, as of 2021-01-07, there is no check for libjpeg.h, so this may fail. To work around do

dnf install -y libjpeg-turbo-devel

FOUR (CI from local gecko build)

And finally, to run with mach use raptor-browsertime look at the docs.

./mach raptor --browsertime

And to edit options, look at

(gecko src)/testing/raptor/raptor/browsertime/base.py

Some raptor command lines look like

./mach raptor --print-tests

./mach raptor -t browsertime-live --browsertime
(other live site tests: expedia, nytimes, wikia-marvel, stackoverflow-question)

./mach raptor-test -t amazon --browsertime --cold --browser-cycles 2

./mach raptor-test -t expedia --browsertime --cold --live-sites --browser-cycles 1 

./mach try perf --browsertime

Filed under technology

← Older posts

sunglint

Protected: Mozilla Android 2024

Protected: LoRA Notes

Deeper Still: Stable Diffusion Notes

Notes on Large Format Laser

robots.denai

Protected: Moz Browsertime (desktop linux a11y test host)

Fedora 38 Notes

Python 3.10/3.11 Side-by-Side Notes

GeoIP2 expanded fields: continents and country registered

Mermaid x Markdown

chrome/chromium CLI

Protected: Influx/Grafana Docker Setup

Cahill Keyes Star-X

Protected: Grafana Subqueries and Transforms

Protected: Grafana Annotations

Mozilla Android Testing Workflow 2022

Protected: geckodriver Notes

Fedora 32 Notes

Bittorrent Notes v2

Moz Browsertime Notes

Tags

Archives