IPFS Background


Some info:
Stanford Seminar, IPFS and the Permanent Web, October 22, 2015.

Main problem points with current web: offline, distributed, permanence, security, speed, lack of richer communication protocols.

Merkle Web, Merkle Tree, Merkle Links: protocol to upgrade how the web works. Related to git’s merkledag. Instead of using a centralized address to locate, use a combination of data x hash function as the address, that way it can be shared.

A quick summary:

Taking a higher-level approach to network design, like Paul Baran and some of the earliest categorization of networks, as featured in 1964’s On Distributed Communications. See network designs for centralized, distributed, peer, isolated, etc.

There is also no ‘presence linking’ in the status quo, meaning that there isn’t a notion for a peer to announce itself in several transports, so that other peers can guarantee that it is always the same peer.

This is especially problematic when sharing media. Any file, any media: when shared on a centralized network, the total amount of data required to share is the size of the original media times multiplied by the number of shares (modulo caching, if be). Ten or 20x multiples are common.

The solution is a distributed web with content uniquely identified, so it can be properly served by multiple peers in a resilient manner. In addition, this works for disconnected, offline, and slow networks.

Instead, social platforms are throwing tons of capital to build walled gardens on centralized networks. Apps continue to dominate the mobile web.

Some hype:
TEDxSanFrancisco, The Next Internet Revolution, December, 2016.

There is documentation for the distributed web as a free gitbook, The Decentralized Web Primer.

Sources, Protocol Specifications

InterPlanetary File System wikipedia.

The IPFS Project, a hypermedia transport protocol project. Based on a network stack encoded in libp2p. See specs documentation here. The naming system is IPNS.

Online community is found here.

Part of Protocol Labs

what about other peer to peer networking libraries, namely libtorrent? libp2p design compare?

language bindings for libp2p are javascript, go, rust, python. What else is in development?

See Also


Internet Life, Social Media, Backup and Archiving


There is a plugin called “Give Me My Data” which, when installed, can be used to export the wall data from Facebook, along with friend and mutual friend graph information in DOT and XML formats.

Google, g+, gmail, youtube.

One intriguing option is google takeout, which will package up all your metadata into one (or more) files.  In addition, gmail can be archived via POP to local mailclient and then exported to mbox or other. Chrome bookmarks can be exported, Google maps can be exported, etc. Google plus data can be shaped by user-configurable settings.

Data liberation front, nice job!


This one is a bit complicated.There is no tumblr export or backup capability provided by tumblr.com, so instead a variety of other options exist. The most reliable is to import your tumblr blog into WordPress, and then export it from WordPress in an XML form.

To do this, see the tumblr.com post.

After import has completed, all the tumbler posts will be imported into the target wordpress.com blog as individual posts. Some of these posts will have titles, some will have the tumbler post id number as the title. All posts are imported with with visibility set to “Public.” Tags in tumblr posts are converted to wordpress tags, and all tumblr posts will have the initial category of “Uncategorized.”

Apply the following transformations to the imported tumblr posts:

1. Create a new “tumblr” category and tag, and batch process the imported posts categorized as “Uncategorized” with it.

2. Set visibility to “Private” 

3. Turn off comments, pings, etc.

These are all easier if the default 20 posts per page is set to a larger number, ie to 999. To do this, Enter the “Posts” menu, and select the top right “Screen Options” button. Change 20 posts to 100. Technically, one can go up to 999 posts per page, but then the query generated by the large bulk transformations makes hosted wordpress/client browsers fail.YMMV. The 100 to 200 range seems to work well for my situation.

To help organize the imported tumblr posts, I create tags for “tumblr”, “import,” and the name of the tumblr blog imported. A tumblr category is made. Then, these are bulk applied and the posts marked “Private” and categories set to “tumblr.”

The tumblr import is remarkably intact. Tags, posts, titles, etc. Photos, videos are imported (although multi-photo arrangements in tumblr may be imported with different orientation.) I’m starting to like the wordpress archive better than the new tumblr dashboard, to be quite honest.

Adding to the oddity that is import/export from tumblr, see wp2tumblr on github. This was the opposite idea: take wordpress content and then import it into tumblr.


Here is a post from twitter about your twitter archive.


Sign in to the Dashboard, and go to the “Tools” option on the left menu. Select export. Volia. Easy peasy.

Fiber Near Futures


San Francisco Fiber Internet Map
last updated: 2013-09-01

A map of recently permitted infrastructure improvements in San Francisco that may point to enhanced networking capabilities, along with current service or pilot project areas. Click on the map above to move to google maps, and then click on the individual layers to see current permits for Sonic.net and for the ATT’s Uverse pilot.

Data Sources

1. SF DPW/Surface Mounted Facilities, Active Permit Map

2. SF DPW Surface Mounted Facilities Permits

3. Search for the following vendors


Using a combination of the above public data sources, and perhaps selective manual use of the FIOS/uverse/(input vendor here) web app, metro San Francisco is able to be data-mapped for service.

Vendor Notes

Some notes on ATT’s Infrastructure Improvement Plan.

All evidence points to the sunset district as the earliest-adopter neighborhood for ATT’s Uverse build-out. Specifically, the outer sunset from 45th to about 15th, centered on Kirkham should be active by now. Permits have been approved as per the map above for decent coverage of the sunset district. As of summer, 2013, permits are in-process for much of the inner mission, mission, bernal heights, and sunnyside.

Notice for permits for the new Uverse cabinets are placed on utility poles by proposed locations, with residents given ample time to comment.

The speed of the newer, post-buildout uverse service is unknown. The most useful data points would be a selection along Kirkham, as these seemed to be the first upgrades to be completed.

Speeds along Mission Street in the core business district of the Mission District, ie Mission Street between 16th and 24th street are currently max 12Mbps/line. For sonic.net bonded, that is doubled to max 24Mbps.

After the infrastructure upgrade, speeds are rumored to double to 24Mpbs single, 48Mpbs bonded. This would be more in-line with Comcast business service speeds, the usual at-home internet service provider for local tech companies like Google, Facebook, and Twitter. ETA: end of 2013.

Political Notes

Of concern is that in the recent municipal elections, a credible plan to improve SF infrastructure was not mentioned once, in any race, by any candidate. This is not even an issue that candidates for the Board of Supervisors have to address! Instead, San Franciscans have to endure more fretting about public nudity and dogs off leash.

More concerning: the 5-year and 10-year infrastructure plans for the city do not even mention telecommunications, networking, wireless coverage, fiber, or use of the existing “dark” fiber network.

People, wake up. San Francisco’s status as an alpha-city in the internet economy is at-risk. San Francisco is not competitive with other high-tech locales for individual network infrastructure. The continuing lack of leadership by Mayor Ed Lee and all the current members of the Board of Supervisors may be relegating San Francisco to last-off-the-block in the US municipal fiber race. Sign the petition to make fiber broadband a priority for San Francisco!

The only action this year has been David Chiu‘s plan to allow for installation of fiber when the streets are ripped up for repair. Given his previous opposition to any gig fiber buildout, this is slightly encouraging. (However obvious.) Sadly, this is two years too late for the massive, city-wide sewer repair, and far too weak.

Chicago has a plan.

Post-Sandy New York has a plan. The massive infrastructure replacement required after “Superstorm” Sandy in 2012 mandates a fiber future.

Seattle has a plan, and an activist mayor.

Los Angeles? Sure.

Kansas City already has a plan, a working implementation, and many many many delighted residents. Right now, that mayor is sitting pretty.

The saga so far:

Fiber Last Century!

Fiber Internet Wins!!

San Francisco Supervisors vs. Fiber Internet

Fiber Last Century!

(This is a recent photo of items found while doing a once-in-a-century telecommunications infrastructure upgrade on a 100 year old SF building. Given the MPOE choices for this locale, the shaky aerials climbed, there is no doubt that the dark corners of the SF telecom system has parts that look like this or worse.)

Was: Fiber Now!

Then: Fiber Last Year!

Now: Partial Fiber Soon Maybe.

So what’s up with high speed internet in San Francisco? Wasn’t there some big hullabaloo about this last year?

In fact, there is some good news. San Francisco Beautiful’s (SFB) latest delay tactic for the all-city fiber internet rollout by ATT has been defeated. Sadly, this is not a chance to rejoice and carouse through the streets, singing so long, suckers. They will no doubt appeal, as haters always hate. And SFB has been hating for five long years and counting….

So what’s going on here?

Let’s break it down into three groups: San Francisco Beautiful, leading the opposition to a once-in-a-century telecommunications upgrade for San Francisco. The Supervisors of San Francisco. And San Francisco citizens who want broadband/home networking that is competitive with other top-tier, high-speed venues like Paris, Bucharest, Tokyo, Hong Kong, and Seoul.

News on the high-speed front: in the report above, is there a glimmer of hope for people who want gig internet in SF’s mission/media gulch/north/beach/haight/castro/noe/soma/etc neighborhoods? Is Sonic.net gonna put 200 boxes in? YES PLEASE ACCORDING TO THESE COMMENTS. Is Verizon looking to put FiOS in SF, for real? Pinch me. Again, the primary goal for SF should be to have multiple vendors providing high-speed internet services, and not limit it to one vendor. This kind of competition means lower prices and faster bandwidth. A secondary, hail-mary type goal should be to convince sonic or google to bring gig-ethernet service to the city, and stop messing around. Note, some expedited processing by the part of city government certainly helped Kansas City. Would the same help SF?

Of note: Seattle just allowed private licensing of the (public) dark fiber ring: SF has this same capability, but is not even considering these solutions. The lack of use of SF’s fiber network is even more distressing when one considers all the recent SFDPW water/sewage improvements, which ripped up streets all over the city: with effective leadership, a full fiber to the home solution could have been implemented simultaneously with the water conservation work. Wiff…. missed opportunity.

News on the supervisor front: Mad Rad Supervisor and proven Internet Champion Carmen Chu and the FUF are planning on planting a record number of trees in the Sunset district in a couple of weeks. Come on people, sign up for the “Outer Sunset” plant and help to make San Francisco REALLY Beautiful. Oh, look who helped pay for some of this beautification: ATT. LOL. Now, I will go cry over a glass of wine….

And bringing up the rear: San Francisco Beautiful. INTERNET DEATH EATER. Stick to the Mission Mural Restoration project and give up on attacking internet infrastructure already. What has to be done to get you to stop harassing San Francisco residents who want better internet? What has to be done to get you to live up to the “livability” part of your mission statement?

The saga so far:

Fiber Near Futures

Fiber Internet Wins!!

San Francisco Supervisors vs. Fiber Internet

Fiber Internet Wins!!!

At approximately 4:45PST, the San Francisco Board of Supervisors voted to affirm the exemption determination for AT&T’s once-in-a-century telephone infrastructure upgrade project, aka “Project Lightspeed.” This means that AT&T can now start to implement their plan to upgrade the telecom infrastructure in the city, with the goal of delivering higher-speed 24Mbs DSL service within all San Francisco districts. AT&T also calls this as “Uverse” which can be any combination of television, telephone, and internet.

The final proposal seems to allow 495 cabinets to be installed throughout the city of San Francisco by AT&T. Multiple potential sites for each cabinet location are to be evaluated. When AT&T seeks an excavation permit from the DPW, besides doing the usual permit stuff, DPW will also notify that site’s supervisor. Then, that supervisor signs an MOU (memorandum of understanding) with AT&T regarding the site, or one of the alternative sightings are considered. Something like that: the specific details seem fuzzy. There seems to be a formal and sane DPW appeals process, and a  my-supervisor-is-crazy-curve-ball appeals process.

Voting against were the following sorry lot of supervisors, with their comments below.

District 1: Eric Mar

District 3: David Chiu

District 5: Ross Mirkarimi

District 6: Jane Kim

District 11: John Avalos

Note to the supervisors listed above: I will be voting against you in future elections, geography permitting, and consider your conduct in this matter anti-internet, anti-technology, anti-jobs: death eaters, all! Here are specific comments.

Mirkarimi: Against this for 2+ years. I don’t understand the process outlined in the MOU. Is this a back-door in the city planning process? Voting against, and in addition tried to defer decision for a fourth time to September 6th.

Mar: AT&T community engagement much improved. They scaled back from 726 to 495 locations, shows improvement. AT&T pays 5% tax on video profits to the city for the right to have boxes taking up city space providing the city/state a utility like telecom. Should that be increased for more boxes? Decreased for fewer boxes? Protect public space. Voting against.

Kim: Confused here. Starts with: “this is not about competition.” Then mentions that many constituents in SoMa want more companies providing internet connectivity and television. Unlike the present situation, with a Comcast monopoly.  Then she gets a couple of thank-yous in, and gets less confused. AT&T did great community outreach. Then she gets confused again: At what number is there a significant impact? Pedestrian safety! Pedestrians complain about the palm trees on 6th street. Voting against.

Chiu: AT&T phase one is 495 cabinets, and then have to confer for up to 231 additional cabinets in any phase two. He wonders about the six years the board of supervisors have been discussing this, and the way technology has changed. (Implying that the technology can be re-engineered by our now-smarter future selves?) Also brings up placement of the boxes on private property. Voting against.

Shouts out to Elsbernd for getting real about this and his unequivocal support from the beginning, and for Wiener for voting yes and the dialogue with AT&T, even if he did delay this vote three times and three months. All good in the end! Scott Wiener’s note explaining his vote is an excellent read, and clearly states the issues: recommended.

This marks the end of a nearly 2.5 year saga whereby AT&T tries to do a network upgrade and the SF Board of Supervisors resists. See San Francisco Board of Supervisors vs. Fiber Internet for some background.

Addendum. Not so fast, apparently. San Francisco Beautiful strikes again, and launches a new legal initiative, which will no doubt delay this project once again. The situation is a bit more nuanced, with ATT on the attack on another front: wireless antenna improvements, aka Project Snail-speed. This second front is also under attack by SF Beautiful and the likely battleground is the Board of Supes at some point, so stay tuned. In addition, ATT is making the tie between good wired infrastructure and better wireless infrastructure explicit with this microcell product. This kind of do-it-yourself-with-our-help-and-network is common in the southland to fix up cellular network holes and dead spots.

When will San Francisco have 100Mbit or gig fiber internet available to the majority of residents, with a choice of providers? Sebastopol, CA, anybody? Canarsie, NY, anybody? Kansas City? Hello?

We built this shit.

The saga so far:

Fiber Near Futures

Fiber Last Century

San Francisco Supervisors vs. Fiber Internet