[WIP] Move to pull-streams

[dignifiedquire]

No description provided.

[daviddias]

• change before merge

[daviddias]

Have you run the https://github.com/libp2p/interface-stream-muxer tests? I would love to see a time benchmark between using regular streams and pull streams

Let's also have tests for:

• destroy underlying pull stream propagates the close and error to all the streams above

const socket = tcp.connect
// do the spdy thing, open a bunch of streams (leave them open)
socket.destroy()
// asses that all the streams, including the libp2p-spdy instance emits an error

• close one of the muxed streams doesn't close others that are open

a) open 5 muxed streams,
b) close one, make sure the others don't close

• close on spdy doesn't close until all the streams that are being muxed are closed too

a) open 5 muxed streams
b) call close/end on the spdy instance, make sure it waits for all of the muxed streams to finish

[dignifiedquire]

Have you run the https://github.com/libp2p/interface-stream-muxer tests? I would love to see a time benchmark between using regular streams and pull streams

I haven't run them yet, I will go and update them + move them to mocha as that way they will be always run with the main test suite.

[daviddias]

they will be always run with the main test suite.

They can be run on release, but not every single as they take some quite of time to finish.

[daviddias]

Getting consistent:

messages 1446
endStreams 14
streamsCreated 100

on the:

    it.only('100 stream with 100 msg', (done) => {
      spawn(muxer, 100, 100, done)
    })

[daviddias]

More detailed:

  # nStreams: 1 nMsg 1
streamsCreated 1
endStreams 1
collectStreams 1
msgSent 1
msgReceived 1

 # nStreams: 1 nMsg 10
streamsCreated 1
endStreams 1
collectStreams 1
msgSent 10
msgReceived 10

 # nStreams: 1 nMsg 100
streamsCreated 1
endStreams 1
collectStreams 1
msgSent 100
msgReceived 100

 # nStreams: 10 nMsg 1
streamsCreated 10
endStreams 10
collectStreams 10
msgSent 10
msgReceived 10

 # nStreams: 10 nMsg 10
streamsCreated 10
endStreams 10
collectStreams 10
msgSent 100
msgReceived 100

 # nStreams: 10 nMsg 100
streamsCreated 10
endStreams 10
collectStreams 10
msgSent 1000
msgReceived 1000

 # nStreams: 100 nMsg 1
streamsCreated 100
endStreams 100
collectStreams 100
msgSent 100
msgReceived 100

 # nStreams: 100 nMsg 10
streamsCreated 100
endStreams 100
collectStreams 100
msgSent 1000
msgReceived 1000

 # nStreams: 100 nMsg 100
streamsCreated 100
endStreams 14
collectStreams 14
msgSent 10000
msgReceived 1446

[daviddias]

@dignifiedquire could you push an update to this branch with the remove of the throttle on stress tests and the pointer to the forked version of spdy?

[dignifiedquire]

@diasdavid done

[daviddias]

@dignifiedquire still missing these tests:

#16 (comment)

[dignifiedquire]

Have you run the https://github.com/libp2p/interface-stream-muxer tests? I would love to see a time benchmark between using regular streams and pull streams

It's not using the same test system anymore, but here are the before and after times

# current master
total:     130127
passing:   130127
duration:  11.4s
npm run compliance  14.52s user 1.42s system 131% cpu 12.108 total

# pull
npm run test:node  5.70s user 0.55s system 107% cpu 5.824 total

[dignifiedquire]

@diasdavid could you write some pseudo code for the tests you are suggesting, I'm not entirely sure what you want.

[daviddias]

@dignifiedquire added more info there - #16 (comment)

[daviddias]

should be to a tcp socket

[daviddias]

Last things for the merge

• [WIP] Move to pull-streams #16 (comment)
• [WIP] Move to pull-streams #16 (comment)
• [WIP] Move to pull-streams #16 (comment)
• Update readme with documentation about pull-streams

[dignifiedquire]

close on spdy doesn't close until all the streams that are being muxed are closed too

test exists, but fails, not sure why though :(

• 4 out of 5 streams are closed
• the last one is closed on the listener side, but not on the dialer side
• all data is received and sent
• spdy-transport does not emit the close event on the last one on the raw node.js stream

[daviddias]

@dignifiedquire you mentioned that spdy is build to destroy all the muxedConns if we call it to end. If that is the case, then it should close all the streams and not wait for them to finish.

what is the raw Node.js stream? the one spdy is mounted on top?

One more thing:

• let's add to the list the same test, but make it happen where data is still being sent.

[dignifiedquire]

you mentioned that spdy is build to destroy all the muxedConns if we call it to end

This was a misunderstanding from my side. Calling .end will

• Send goaway frame
• Error all newly created streams with New stream after goaway
• Attach an optional close handler, which is triggered once all open streams are closed

[daviddias]

awesome, so it doesn't abruptly all flowing streams, that is what I thought :)

[dignifiedquire]

let's add to the list the same test, but make it happen where data is still being sent.

that's what the test currently does, send data on 5 streams delayed, and trigger .end before they are finished