net/mock: support ConnectionGater in MockNet

[ajsutton]

Adds support for enforcing ConnectionGater checks to MockNet. This enables integration tests to test connection gating policies such as peer blocking policies.

MockNet doesn't actually use security handshakes or a connection upgrade process, so the whole cycle is simulated with calls to InterceptSecured and InterceptUpgraded being done when the connection is opened.

[ajsutton]

I should note as well - the local and remote checks are run on the same thread as part of connecting which I believe matches the real behaviour where if the security handshake or protocol negotiation fails Dial will return an error, but if it should be async they could be moved to peerNet.remoteOpenedConn and just close the connection if the checks fail.

[ajsutton]

Updated the public interface to support passing in PeerOptions so both the peer store and connection gater can (optionally) be specified when generating or adding a peer. Mocknet is a little more complex but:

1. This provides a lot more flexibility for callers
2. The API is now extensible for any other inputs that need to be added without having to add an ever growing range of WithXAndYAndZ type methods.

[marten-seemann]

What's the motivation behind this PR? mocknet is deprecated and will be replaced with an in-memory-transport soon.

[ajsutton]

We're using mocknet in our testing but ran into limitations in our ability to test things because the connection gater wasn't in use - in particular we're unable to test that peers that a downscored past a certain point are then disconnected and banned from reconnecting.

A more realistic in-memory-transport sounds excellent but if that's not likely to be ready in the next release it would be nice to get this in if possible to unblock our testing until the in memory transport is available. Long term it sounds like in-memory transport would let our test setup be much closer to the production setup which would be a big win so I'd definitely be keen to switch over to it.

[marten-seemann]

There's a flaky mocknet test:

  === RUN   TestEventBus
      mock_test.go:672: wrong connectedness type
  --- FAIL: TestEventBus (0.01s)

Can you please debug and fix?

[ajsutton]

Yep I'll see what I can find.

[Wondertan]

I made that test and can help debugging it

[ajsutton]

@Wondertan some debugging help would be useful. Even on the current master branch I get failures when running the test multiple times. I added:

func TestLoop(t *testing.T) {
	for i := 0; i < 1000; i++ {
		t.Run("", func(t *testing.T) {
			TestEventBus(t)
		})
	}
}

to mock_test.go and get 10-20 failures on each run locally.

It looks like when it fails it's because it gets an additional connected event when it's expecting disconnected events. Throwing a bunch of printlns and a debug.PrintStackTrace() in when it's adding a connection it looks like sometimes the peers are disconnected before the identify process has completed and so it reconnects. Stack trace of the second connection is:

runtime/debug.Stack()
	/opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/debug/stack.go:24 +0x64
runtime/debug.PrintStack()
	/opt/homebrew/Cellar/go/1.20.4/libexec/src/runtime/debug/stack.go:16 +0x1c
github.com/libp2p/go-libp2p/p2p/net/mock.(*peernet).openConn(0x140002cad00, {0xff?, 0x1031e0b3f?}, 0x19?)
	/Users/aj/Documents/code/go-libp2p/p2p/net/mock/mock_peernet.go:154 +0x1a4
github.com/libp2p/go-libp2p/p2p/net/mock.(*peernet).connect(0x140002cad00, {0x1400019b080, 0x22})
	/Users/aj/Documents/code/go-libp2p/p2p/net/mock/mock_peernet.go:143 +0x3a8
github.com/libp2p/go-libp2p/p2p/net/mock.(*peernet).DialPeer(0x14000080820?, {0xff?, 0x10317e249?}, {0x1400019b080?, 0x1400094fcb8?})
	/Users/aj/Documents/code/go-libp2p/p2p/net/mock/mock_peernet.go:106 +0x28
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).dialPeer(0x140006be180, {0x103506930, 0x14000526d20}, {0x1400019b080, 0x22})
	/Users/aj/Documents/code/go-libp2p/p2p/host/basic/basic_host.go:739 +0xe8
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).Connect(0x140006be180, {0x103506930, 0x14000526d20}, {{0x1400019b080, 0x22}, {0x0, 0x0, 0x0}})
	/Users/aj/Documents/code/go-libp2p/p2p/host/basic/basic_host.go:732 +0x10c
github.com/libp2p/go-libp2p/p2p/host/basic.(*BasicHost).NewStream(0x140006be180, {0x103506930, 0x14000526d20}, {0x1400019b080, 0x22}, {0x14000518290, 0x1, 0x1})
	/Users/aj/Documents/code/go-libp2p/p2p/host/basic/basic_host.go:635 +0xb0
github.com/libp2p/go-libp2p/p2p/protocol/identify.(*idService).sendPushes.func1({0x10350dc40, 0x14000114360})
	/Users/aj/Documents/code/go-libp2p/p2p/protocol/identify/id.go:304 +0x138
created by github.com/libp2p/go-libp2p/p2p/protocol/identify.(*idService).sendPushes
	/Users/aj/Documents/code/go-libp2p/p2p/protocol/identify/id.go:299 +0x6c0

[Wondertan]

@ajsutton, I just tried and cannot reproduce this flake on master, however, I am able to reproduce this on mocknet-gater branch with GOMAXPOCS set to 1,2,3
I tried your snippet(with GOMAXPROCS) and GOMAXPROCS=1 go test ./p2p/net/mock -run TestEventBus -count 100 -v -failfast on both versions 3 times and the results were the same.
I am not sure how gater changes can contribute to flakes

[ajsutton]

ah, looks like it got fixed on master at some point since I branched this off.

$ git checkout 8719fc49446e488621d8bdee7d0fbdaef3f1591a
Previous HEAD position was 9370213e net/mock: support ConnectionGater in MockNet
HEAD is now at 8719fc49 net/mock: mimic Swarm's event and notification behavior in MockNet (#2287)
 aj@Busy  ~/Documents/code/go-libp2p  ➦ 8719fc49  go test ./p2p/net/mock -run TestEventBus -count 10000 -v -failfast
=== RUN   TestEventBus
    mock_test.go:669: wrong connectedness type
--- FAIL: TestEventBus (0.00s)
FAIL
FAIL	github.com/libp2p/go-libp2p/p2p/net/mock	0.397s
FAIL

but on current master it passes consistently for me.

So I've rebased this on master and now it's passing reliably on this branch too.

[ajsutton]

ok, something different this time:

  === CONT  TestSimultOpen
      simul_test.go:32: error swarm dialing to peer failed to dial 12D3KooWCaRagxPWMBeyU8eHbzvvJrmmc2ympA6bngPuPfBmXNvR:
            * [/ip4/127.0.0.1/udp/59707/quic] INTERNAL_ERROR (local): write udp4 127.0.0.1:48990->127.0.0.1:59707: sendto: operation not permitted
  --- FAIL: TestSimultOpen (0.00s)

If I'm reading that test right its using a real swarm host, not the mock net though.

[MarcoPolo]

Thank you, this looks good. Could you try rerunning CI again? I think you hit a flaky test not related to these changes. (I would, but GH doesn't let me)

Apologies for the delay to review

[ajsutton]

I couldn't rerun the test in GH either but have rebased this on top of latest master which got them to run again.

[ajsutton]

@MarcoPolo Looks happy now.

[MarcoPolo]

Great! Thanks for contribution!