Sprint Aug 17

[ghost]

Sprint Goals

...

Sprint Discussions

See https://github.com/ipfs/pm#sprint-discussion-schedule

Sprint Deliverables

• Add yourself below, that way we only have people listed who are really in the sprint
• Add links to issues down here. Only add things you can finish this sprint.

@lgierth

• Bad blocks
  • deploy gateway-dmca-denylist Deploy denylist infra#59
• cjdns
  • land Discovery.Cjdns WIP: discover peers from cjdns kubo#1316
• Static pages on the gateways
  • migrate gateway.ipfs.io to ipfs.io Move gateway to ipfs.io infra#4
  • dist.ipfs.io and ipfs.io/dist https://github.com/ipfs/distributions
• Hosts
  • rename willie and scruffy Rename scruffy and willie infra#58
  • storage hosts sirius and pollux solarnet: add storage hosts sirius and pollux infra#62
  • access to storage host for @davidar
• SSL/TLS on the gateways
  • StartSSL cert SSL/TLS Cert for gateway infra#6
  • erase plaintext secrest from history
• downloads.openwrt.org in IPFS
  • investigate ipfs add bottleneck Slow ipfs add on storage host infra#64

@diasdavid

• Add more tests to kad-router
• Review and provide feedback on HTTP-P2P https://datatracker.ietf.org/doc/draft-benfield-http2-p2p/
• Revamp network spec -> make a libp2p spec
• "hopefully" have something like a record store
• Catch up on W3C WebRTC WG, document railing process
• update multistream + multicodec bring updates on updated multistream multiformats/multicodec#2 Call multicodecto multicodec :) multiformats/multistream-select#3 rename to length-prefixed-message multiformats/multistream-select#2

[ghost]

@jbenet @whyrusleeping @krl @cryptix @diasdavid @mappum please add your tasks

[jbenet]

@lgierth thank you. sorry i keep failing at this-- (ton of lose endes). Actually, if someone wants to take over the materializing of the etherpad + sprint issues from me, would be much appreciated

[jbenet]

@whyrusleeping and i discussed go-ipfs over

• https://etherpad.mozilla.org/Qn2JQa4eXx
• https://botbot.me/freenode/ipfs/2015-08-19/?msg=47633855&page=3
• https://botbot.me/freenode/ipfs/2015-08-22/?msg=47880245&page=4

Summary is:

• this week we land batches: 1, 4, 7, 8
• next are: 2, 5 > 3, 9 > 6

all the Go help we can get is useful.

[jbenet]

IPLD / Records

Batch 1:

• implement protobuf support (missing) @jbenet
• solve cbor encoding fails to produce equal buffers (with canonical = true)? ugorji/go#93 @whyrusleeping
• go-ipld module (done) (needs a tweak to just use go-multicodec) @jbenet
• extract all the merkle dag tooling into own module(s), maybe @jbenet
  under go-ipld, or maybe under go-merkledag or go-mdag

Batch 2:

• implement go-iprs on top of go-ipld // records + ipld @jbenet
• keystore (@whyrusleeping)

Batch 3:

• dht / naming fixes @whyrusleeping
• migration! \o/ (let's do this last, because records may need it too) @whyrusleeping

other batches

this doesn't make much sense, as they're not ordered. just some "coarser units of work".

Batch 4 -- landing @whyrusleeping's PRs

• pub/priv blockstore WIP: separate private and public blocks kubo#1538 @whyrusleeping
• nodebuilder replace nodebuilder with a nicer interface kubo#1572 @whyrusleeping
• blocklists implement access control lists for gateway kubo#1551 @whyrusleeping
• mfs Implement mfs API kubo#1550 @whyrusleeping
• Editor more efficient an attempt at making the editor more efficient kubo#1567 @whyrusleeping

Batch 5 -- symlinks!

• symlink support!!! @whyrusleeping

Batch 6 -- ipget + ipfs-update

• ipget
• ipfs-update (using new dist.ipfs.io)

Batch 7 -- landing @rht's PRs

• gc Add fixed-period and conditional repo GC kubo#1495 @rht there is filesize bug so this is pending
• ctx Make sure ctx in commands are derived from req.Context kubo#1585 @rht Merged
• get Refactor ipfs get kubo#1558 @rht Merged
• notify Notify when daemon is ready kubo#1569 @rht Merged

Batch 8 -- bughunting

• memory leak/fail when >20GB memory fail when adding >200GB kubo#1584
• add abruptly dies Add abruptly dies when daemon running kubo#1591
• UTF8 corruption JSON encoding corrupts non-UTF-8 binary data kubo#1582

Batch 9 -- bughunting less important

• pin path ancestors? pin path ancestors? kubo#1580
• datastore keys cant contain '/' some local hash is not listed by ipfs refs local kubo#1554

[jbenet]

@whyrusleeping I've CRed all your Batch 4. (still need to play more with mfs. im not super sold on the UX. maybe it could be another tool entirely, but consider the local (/myfs/... at $repo/local/myfs) idea i suggested there.

[jbenet]

@rht i merged ctx, get, and notify! :) If you can tackle some of the things above, maybe some of the bugs, or even symlinks, would be great.

[rht]

symlink support!!! @whyrusleeping

what about unixfs permission?

[jbenet]

@rht yeah, we should add these as well.

[jbenet]

• My sprint update for tonight.
• sorry for failing again on the etherpad-- @lgierth has picked this up instead
• whyrusleeping and i charted a roadmap for go-ipfs dev 0.4.0 in writing. it was good way to list out all the things we need to do. may be useful to do similar for other things.
• we're behind on many things, let's pick it up!

---

• ship dist.ipfs.io (not done)
  • improved mkdist.js for more dists / versions
  • ipfs-app mkdist.js
  • need ipfs-app gif
  • first build of dist.ipfs.io
• ipld / go-multicodec -- implemented + shipped
  • needs cbor ordering fix + protobuf test fixes
• upcoming go-ipfs things + dev 0.4.0 roadmap
• fix https://github.com/ipfs/infrastructure/blob/master/seeding/refs-to-seed
  • ensure everything is seeded
  • seed on {sirius,pollux}.i.ipfs.io (not done)
• outreach
  • berlinjs meetup talk
  • podcast invterview (out tonight)
  • caught up on mailpile
  • caught up on ext discussions
• catch up on CR (anything left for me?)

---

• I may not be back in time for IRC checkin. if i'm not, someone (maybe @lgierth or @diasdavid) go through everyone?
• @whyrusleeping may be out today.

[daviddias]

SPRINT UPDATE

This past week was +- productive. Coming back home after 53 of travelling meant that I had to take care of a lot of stuff in the first couple of days.

• Wrote more tests to the kad-router (which covered some more edge cases)
• Reviewed HTTP-P2P proposal
• Worked a lot of revamping the network spec, now known as libp2p. It still requires more though, specially after having a good conversation today with @jbenet where we learned that we weren't completely in sync of how the Record Store would work, did some progress, hopefully it will be nailed down by today node-ipfs meeting
• continue pursuing my role in the W3C WebRTC WG, documented railing process and still digesting all of the info available
• update multistream + multicodec specs bring updates on updated multistream multiformats/multicodec#2 Call multicodecto multicodec :) multiformats/multistream-select#3 rename to length-prefixed-message multiformats/multistream-select#2

[ghost]

Pad for Sprint Aug 24: https://etherpad.mozilla.org/CFlfCtFXnB

[krl]

Starlog can now do infinite scroll, publishing and viewing of entries. Webcomponents turn out to be a really nice match to ipfs, from starlog repo you can now do 'npm run local', and 'npm run gateway' to use vulcanize to pack everything up into one html file, and view it on 5001 or gateway (still fails on CORS though)

I would like to take next week to properly package the ipfs-api webcomponent, and make it part of the build-step of node-ipfs-api (maybe rename it too?). Things left to be done for starlog is editing entries.

have been bitten by untested edge cases in the append only log, and spent quite some time fixing that and covering all things we need.

[ghost]

hosts

• applied proper naming to the utility hosts scruffy and willie, they're now deimos and phobos
• built out the storage hosts sirius and pollux
• gave davidar a login for sirius and pollux
• mirrored downloads.openwrt.org to pollux
• investigating ipfs add bottlenecks

gateways

• still using a self-signed cert, waiting for startssl.com's process
• made IPNSHostname work (kind of), for self-hosting ipfs.io
  • needs a bit more work regarding stylesheets in directory listings
• ongoing fixes to the readonly api, we need CORS there
• erased secrets from the infrastructure.git history (i.e. rewrite history)

not-done

• dist.ipfs.io deployment
• gateway-dmca-denylist
• discovery.cjdns

general things

• picked up sprint sync
• badly need to increase infrastructure bus factor, i'm the only one that can deploy at the moment and that sucks. it makes me a bottleneck, and keeps me from working on go-ipfs and cjdns stuff

[chriscool]

My sprint update (sorry for being late):

• gave my talk at LinuxCon NA where I talked about Sharness and a bit about IPFS
• met some people who already know about IPFS like Alexis Rossi of the Internet Archive
• changed the GitCop message a bit and responded to Gary (the GitCop guy)
• started Binary data and json test kubo#1599 and sent email to Markus