Sprint Aug 24

[ghost]

Sprint Goals

• IPLD/records, whyrusleeping's PRs, rht's PRs, and bughunting. More details here ff.: Sprint Aug 17 #28 (comment)
• everybody able to deploy our infrastructure

Sprint Discussions

See https://github.com/ipfs/pm#sprint-discussion-schedule

• Infrastructure discussion: https://etherpad.mozilla.org/SHYH0M9LvG

Sprint Deliverables

• Add yourself below, that way we only have people listed who are really in the sprint
• Add links to issues down here. Only add things you can finish this sprint.

@krl

• make ipfs-api webcomponent part of node-ipfs-api build process
• editing of starlog entries

@rht

• ipfs add breaks somewhere in the middle of upload kubo#1072
• memory fail when adding >200GB kubo#1584
• dag archive

@chriscool

• Binary data and json test kubo#1599

@lgierth

• enable CORS for API gateway api doesn't resolve ipns entries kubo#1586 ipfs: enable CORS for API infra#68
• Infrastructure bus factor
  • split up Split up solarnet.yml playbook infra#74
• Bad blocks / denylist
  • review why's PR implement access control lists for gateway kubo#1551
  • add reason field (dmca vs. code-of-conduct)
    • i.e. 451 Unavailable For Legal Reasons vs. 452 Violation of Code of Conduct, or so
  • denylist.ipfs.io Deploy denylist infra#59
• TLS/SSL for gateways SSL/TLS Cert for gateway infra#6
• Static pages on gateway
  • migrate gateway.ipfs.io to ipfs.io Move gateway to ipfs.io infra#4
  • inline directory-listing stylesheets gateway: embed directory-listing styles kubo#1615 (from http://gateway.ipfs.io/ipfs/QmWxrgcm6nTNbRB7S2ugq7Wfw3pD1FVBTN6WMtkHiD396u/bundle.css)
  • make /ipfs and /ipns work for (gateway.)ipfs.io Make /ipfs and /ipns work with self-hosted ipfs.io infra#72
• giving people non-root access

@diasdavid

• Continue working on libp2p-spec
• Implement Record, following IPRS
• Make abstract-record-store
• Implement kad-record-store, that follows abstract-record-store interface
• Implement distributed-record-store
• node-ipld https://github.com/diasdavid/node-ipld
• Make the Node.js implementation more welcoming by:
  • Adding CI
  • Adding Code Coverage reference
  • Adding Badges
    • david-dm
    • codestyle
    • build

@jbenet

• doc for ipfs-api implementors https://github.com/ipfs/go-ipfs/blob/master/docs/implement-api-bindings.md
  • implement --api <multiaddr> option --api <address> support for both client and daemon kubo#1598
  • tests for --api <multiaddr> option (can someone pick this up?)
• batch 1 of go-ipfs Sprint Aug 17 #28 (comment)
  • multicodec protobuf support protobuf support (WIP) multiformats/go-multicodec#1
  • go-ipld module use multicodec
  • extract all the merkle dag tooling into own module
• dist.ipfs.io
  • screenshot for ipfs-app
  • make sure ipfs-app dist works
  • run a full dist in a storage node
• statship
  • figure out docker reqs
  • detailed spec (inc UX and what's needed)
  • prototype it
• eth
  • c++ api bindings? (req ipfs-api impls)
  • patricia tree stuff?
• outreach
  • organize IPFS Berlin meetup
  • ipfs meetup
  • talk with ascribe.io
  • talk with alexandria
  • prepare Zurich talks

@whyrusleeping

• finish mfs PR
• finish lists PR
• land node constructor PR
• land blocklists PR
• investigate/fix cbor encoding issue
• fix protobuf multicodec

[jbenet]

added my tasks above. @whyrusleeping yours?

[jbenet]

@lgierth maybe deprecate the etherpad when making issues: https://etherpad.mozilla.org/CFlfCtFXnB that way people who are late know to post it here

[ghost]

maybe deprecate the etherpad when making issues: https://etherpad.mozilla.org/CFlfCtFXnB that way people who are late know to post it here

yep good call

[jbenet]

• @whyrusleeping your list looks sparse-- are you getting through all of Batch 1 + 4 stuff? Sprint Aug 17 #28 (comment)
• @whyrusleeping @rht how are we handling batch 8 ?
• @rht i think dag-archive is way under specified atm. land batch 8 instead?
• @rht what is next step on Add fixed-period and conditional repo GC kubo#1495 ?

[rht]

batch 8:
"progress" on ipfs/kubo#1072 if @whyrusleeping can take a head shot on it.

There are 3 cases of the bug:

1. the original issue, which is too big for me to download and to reproduce. It was said there is no symlink.
2. @davidar's, also too big. Can't tell if there is a symlink.
3. my case, which can be reproduced with ipfs add -r Godeps/_workspace/src/github.com/jbenet/go-multihash. This is fixed with removing the broken symlink in go-multihash: test/sharness/bin.

Still can't tell why the symlink err doesn't appear in the daemon or ipfs add.
The symlink err appears (when daemon is running), however, when I forced to Stat() all the fileinfo right after https://github.com/ipfs/go-ipfs/blob/master/commands/files/serialfile.go#L58

[davidar]

@rht No symlinks. I can try to make a test case that doesn't require downloading anything if that helps

[rht]

@davidar It will definitely help. It is likely any file that is not regular file or directory?

One culprit of the not returning err is likely in one of os.Open.
Currently pulling things from ipfs/kubo#1413 (but with Stat replaced by Lstat).

[davidar]

@rht No, just regular files. Test case:

mkdir foo
for i in `seq 999`; do dd if=/dev/urandom of=foo/$i bs=1M count=1; done
ipfs add -r foo

[ghost]

I'll likely be on the train tomorrow CEST evening, can someone head the sprint sync for me? That involves:

• announce at 5pm UTC
• post new etherpad here
• role call and start at 5:15pm UTC
• discuss sprint goals

[whyrusleeping]

@lgierth I got it.

[ghost]

Thanks @whyrusleeping, I'll materialize the sprint issue on tuesday

[ghost]

My sprint update, I likely won't be around for sync.

done

• enable CORS for API -- this was blocking krl's work
• TLS/SSL for gateways
  • our setup is graded A by SSL Labs :) https://www.ssllabs.com/ssltest/analyze.html?d=gateway.ipfs.io&latest
• Infrastructure bus factor
  • split up the monolithic ansible playbook -- this will make deploys faster since we don't need to run all the tasks, but only the ones we want
• hellabot CR
• hetzner netscan alert, once more

pending CR/merge/deploy

• Static pages on gateway
  • inline directory-listing stylesheets gateway: embed directory-listing styles kubo#1615 (from http://gateway.ipfs.io/ipfs/QmWxrgcm6nTNbRB7S2ugq7Wfw3pD1FVBTN6WMtkHiD396u/bundle.css)
  • make /ipfs and /ipns work for (gateway.)ipfs.io Make /ipfs and /ipns work with self-hosted ipfs.io infra#72

little/no progress

• Bad blocks / denylist
  • not a lot of progress, direction changed a bit and there was fruitful discussion, i held back from taking further steps though while it wasn't clear which direction we're taking
• giving people access to our boxes
  • just didn't get to it

[whyrusleeping]

New etherpad: https://etherpad.mozilla.org/ijvTxGAhOP

[daviddias]

SPRINT CHECKIN

This week's primary goal, pushing a full working version of a libp2p, was majorly delayed due to the current challenges when implementing IPLD (with the discovered json-ld limitations), which lead to a lot of discussion in what is the best approach from here forward when it comes to Linked Data. IPLD is necessary for creating MerkleDAG obj/links, which is the data structure of IPRS.

• Worked in the Node.js implementation https://github.com/diasdavid/node-ipld
• Listed current components of the IPFS Node.js implementation in the node-ipfs repo, so other people could follow their state.
• Added some convenient features for contributors to the set of node modules we have for libp2p, so other people feel more encourage to contribute following things like codestyle, passing tests, coverage, etc.
• Edited Kyle's talk on "Making the Web Permanent"
• Started writing an article (currently on: https://github.com/diasdavid/blog.daviddias.me/master/drafts/2015-09-XY-real-time-communications-the-challenges-solved-with-webrtc.md) to document how WebRTC solves the transversal network problems for P2P applications
• Continued working and reviewing WebRTC specs, sent out some request for clarifications and editorial mistake fixes. (It is kind of refreshing seeing how some of WebRTC specs have been broken down into smaller, with a single focus, specs)
• Also took some time to look more in depth to React stuff for the electron app

[jbenet]

@jbenet's update

I'm at an IPFS dinner meetup organized by IPFSers in Basel (switzerland). will be giving a talk tomorrow at ETH, and talks to other groups here. Will look through updates + sync up with people a bit later tonight.

• got about half of what i wanted to get done :/
• The critical stuff is IPLD + Starship.
• Starship on track
• Ran into issues with IPLD's model, discussing with mildred and daviddias
• multicodec improvements took a bit longer, but it's in great shape now.
• below is the checkbox listing:
• doc for ipfs-api implementors https://github.com/ipfs/go-ipfs/blob/master/docs/implement-api-bindings.md
  • implement --api <multiaddr> option --api <address> support for both client and daemon kubo#1598
  • tests for --api <multiaddr> option (can someone pick this up?)
• batch 1 of go-ipfs Sprint Aug 17 #28 (comment)
  • multicodec protobuf support protobuf support (WIP) multiformats/go-multicodec#1
  • go-ipld module use multicodec
  • extract all the merkle dag tooling into own module
• dist.ipfs.io
  • screenshot for ipfs-app
  • make sure ipfs-app dist works
  • run a full dist in a storage node
• statship
  • figure out docker reqs
  • detailed spec (inc UX and what's needed)
  • prototype it
• eth
  • c++ api bindings? (req ipfs-api impls)
  • patricia tree stuff?
• outreach
  • organize IPFS Berlin meetup
  • ipfs meetup
  • talk with ascribe.io
  • talk with alexandria
  • prepare Zurich talks
• added
  • ipld discussions with json-ld folks, mildred, daviddias
  • multicodec binary to test out the interface
  • tons of CR

[whyrusleeping]

@whyrusleeping's update

• finish mfs PR
  • switched from 'session' flag to pathing
  • ready for final CR
• finish lists PR
  • still unclear on what steps to take
• land node constructor PR
• investigate/fix cbor encoding issue
• fix protobuf multicodec
• rewrite trailers code for go1.5 (not merged yet)
  • should discuss when to switch to 1.5
• symlinks PR is up, implements basic support for adding symlinks, viewing them in fuse, and writing them via ipfs get
• rm blockservice workers
  • bitswap doesnt block, provides collector makes sure of that.
  • still have a couple TODO's on that
• worked on gx
• wrote https://github.com/whyrusleeping/gx-go-import to make importing packages easy