Daemon ignores filters and trigger NetScan abuse #8585
Comments
syntafin
added
kind/bug
|
Thank you for submitting your first issue to this repository! A maintainer will be here shortly to triage and review.
Finally, remember to use https://discuss.ipfs.io if you just need general support. |
Have you directly seen this, e.g. noticing Wireshark outputs that you are dialing one of the addresses that should be filtered? Could you get any information from the provider as to why you got triggered for abuse? |
|
Yes, we can only run the node if we block the IP ranges via Firewall, what will kill the function of ipfs (other nodes can't connect) |
|
@syntafin mind sending Wireshark dumps that confirm this? |
lidel
added
need/author-input
|
Resolving since haven't received more info. Feel free to reopen if you're able to provide it. |
This comment was marked as spam.
This comment was marked as spam.
|
@syntafin : thanks for the details. To move this forward, we'll need an easier repro case. Can you confirm this on a clean installation, with the latest release, and get a packet capture (tcpdump, wireshark, tshark). It would help too to know whether this is SYN or SYN ACK packets. |
|
I will check this out on weekend. |
|
I uploaded the log here, because of the size (I didnt do any filtering): |
|
2022-06-10 conversation: @Jorropo will do a timebox investigation to see if we can identify next steps. |
|
@syntafin could you please send a file that doesn't have reverse dns names resolved ? Could I pls get a dump like this: This will records all the packets and their content to one of the banned range and that is comming or going to the port 4001 to a dump file in Let it run until it captures a few packet ~100, it will print the current packet count to stderr. |
|
Update, I have setupped a new server with the server profile and a capture running. |
|
Update: my capture didn't got any packet after 5 days. |
|
After 10 days I have captured 0 wrongfully sent packet. |
|
Due to lack of reproduction I'm gonna close this, if anyone sees this again pls reopen. |
|
@Jorropo |
|
@ROBERT-MCDOWELL have you tried |
|
yes, it's the same. for now I just set iptables to drop all private ip range going out. |
|
The same way we did for our side too. |
|
Reopening given continued reports. Is anyone able to provide a repo case like @Jorropo request? Also, is everyone reporting this using the latest Kubo release? |
|
FWIW this is holding me back from using IPFS seriously, as the amount of data i'd like to share is not workable with a home router (it crashes from all the incoming connections), and this issue makes it impossible to host on hetzner without them frowning at all the outgoing connections. |
|
This is a real pain - ran into multiple reports with hetzner this way |
|
I've previously made a fork of go that will panic when dialling one of the restricted IP address with the goal of revealing the offending stacktrace. I fixed it and it's now working (it correctly crashes). If peoples want to help please run this binary as daemon ( |
|
@BigLep Shouldn't we consider to use the Server profile as the default for Kobo? Is there many usage of local discovery? I guess a bunch of IPFS nodes on the same network segment will be part of an IPFS cluster, or? |
|
@rvalle I belive the issue here is that Kubo is trying to dial addresses in private ranges even tho
A few conferences I went had a few IPFS nodes running but we didn't all shared an IPFS cluster and my phone isn't on my ipfs cluster because I don't want it to sync GiBs of data. One smarter thing to do would be to list what are the IP addresses on the machine, and then enable the server profile by default if a public IP is found (wont cover everything but should help in most cases). |
|
@Jorropo yes, I know. I have been kicked out of Hetzner now twice because of this. I understand the issue is now resolved. My additional question, since dialing local addresses is a dangerous feature, why is it enabled by default. |
|
It's not dangerous Hetzner has pointless rules. If you have a private network with other machines on it, then you could argue that maybe this could be used for attacks where someone advertise a private IP and Port which is used by let's say your printer and now your printer is printing (note I don't know of any printer that implements remote printing by just printing whatever is sent on a TCP stream, this would have been how might have done things about 45 to 30 years ago, but now all printers I know use more advanced protocols that give meta options (such as sending images, pdfs, ...)): (the start of a TCP libp2p handshake) If the attacker could precisely control what your node send when doing this first exchange, this would be extremely bad (the attacker could spoof an http request to some internal API, or some SQL / Redis connection (spoofing redis requests to insert tasks in the scheduler queue and gain RCE is a commun CTF challenge)). But it can't, so nothing intresting can be done with this. Given the attacker do not control what is sent, you would need some really dumb services that make sense of garbage inputs like this for anything interesting to happen, most services will understand that The other option is that the attacker points it to some other libp2p node on your network, the first part of the handshake will work, however because all libp2p connections are encrypted, extremely quickly you are gonna reach the security handshake, which is gonna fail (because the peer ids don't match). |
|
I have setupped a v0.19.0-dev build (this one #8585 (comment)) on a machine at hetzner because the deployements I did on my machines and at scaleway are working fine (using |
|
fwiw, i had a lot of data stored on my instance, and it triggered primarily because it was dialing-out to other nodes to refresh the keys in the DHT my lowwater was around 800, and highwater around 2000 |
|
@ShadowJonathan are you running with the server profile ? Because my tests instance also have lots of data (I joined them to my ipfs cluster because why not ?) and it's doing DHT publishes yet its doing fine (after a few days for not hetzner, and 2 hours for the hetzner one). |
|
Yes, though I also wanna say that testing on hetzner and expecting a turnaround in a day won't really work, usually it's around a week or more |
|
@rvalle |
|
I have now upgraded to 0.17 and cleared a firewall rule that I had to stop private traffic out of the interface in our hetzner production environment. A quick tcpdump shows no longer traffic to 192.168/16 on port 4001, so this looks good. When I upgraded to 0.16.0 I was notified within hours of network scanning abuse, It has been few hours already and no notification... looks good to me. |
|
I'll make an effort to port my old patches to 0.17 then, and get it running, I'll report back after 😃 |
|
It has been two weeks now since I created: I'm giving up on this for now (the server will continue to run for a few months probably and I'll check on it from time to time), I'll pick up this again if someone can show me a panic stack trace found by #8585 (comment) (you have to use the server profile). |
|
I've been running for a few hours now and a glance seems to suggest that kubo behaves much better than my previous assessment, but i'll also keep it running. |
|
@ShadowJonathan |
|
It's been running well since, yes |
|
Started via docker image (kubo/latest) a new one, and... guess what, this arrived 5mins after: Local IP dials Logs
|
|
@syntafin AFAIT:
You did not configured your node. You have to pass the Or do |
Yeah, on initial start it ignored the environment variable, we restarted it and from Hetzner's perspective it does work now (again). |
|
is it also a solution for normal deamon too? or should I give up IPFS at alll and find another technology? |
As I said, it ignored the environment variable on first launch, but now it works. |
|
|
Closing as original issue is resolved.
If you still have a netscan-related problem, please open a new issue 🙏 |
|
To make it clear, I had a missunderstood what triggered the last netscan incident! Teammate forgot on first launch to set the server environment, so it started as desktop/client daemon |
Checklist
Installation method
ipfs-update or dist.ipfs.io
Version
Config
{ "API": { "HTTPHeaders": {} }, "Addresses": { "API": "/ip4/127.0.0.1/tcp/5001", "Announce": [], "Gateway": "/ip4/127.0.0.1/tcp/8080", "NoAnnounce": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "Swarm": [ "/ip4/0.0.0.0/tcp/4001", "/ip6/::/tcp/4001", "/ip4/0.0.0.0/udp/4001/quic", "/ip6/::/udp/4001/quic" ] }, "AutoNAT": {}, "Bootstrap": [ "/dnsaddr/bootstrap.libp2p.io/p2p/QmNnooDu7bfjPFoTZYxMNLWUQJyrVwtbZg5gBMjTezGAJN", "/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa", "/dnsaddr/bootstrap.libp2p.io/p2p/QmbLHAnMoJPWSCR5Zhtx6BHJX9KiKNN6tpvbUcqanj75Nb", "/dnsaddr/bootstrap.libp2p.io/p2p/QmcZf59bWwK5XFi76CZX8cbJ4BhTzzA3gU1ZjYZcYW3dwt", "/ip4/104.131.131.82/tcp/4001/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ", "/ip4/104.131.131.82/udp/4001/quic/p2p/QmaCpDMGvV2BGHeYERUEnRQAwe3N8SzbUtfsmvsqQLuvuJ" ], "DNS": { "Resolvers": {} }, "Datastore": { "BloomFilterSize": 0, "GCPeriod": "1h", "HashOnRead": false, "Spec": { "mounts": [ { "child": { "path": "blocks", "shardFunc": "/repo/flatfs/shard/v1/next-to-last/2", "sync": true, "type": "flatfs" }, "mountpoint": "/blocks", "prefix": "flatfs.datastore", "type": "measure" }, { "child": { "compression": "none", "path": "datastore", "type": "levelds" }, "mountpoint": "/", "prefix": "leveldb.datastore", "type": "measure" } ], "type": "mount" }, "StorageGCWatermark": 90, "StorageMax": "10GB" }, "Discovery": { "MDNS": { "Enabled": false, "Interval": 10 } }, "Experimental": { "AcceleratedDHTClient": false, "FilestoreEnabled": false, "GraphsyncEnabled": false, "Libp2pStreamMounting": false, "P2pHttpProxy": false, "ShardingEnabled": false, "StrategicProviding": false, "UrlstoreEnabled": false }, "Gateway": { "APICommands": [], "HTTPHeaders": { "Access-Control-Allow-Headers": [ "X-Requested-With", "Range", "User-Agent" ], "Access-Control-Allow-Methods": [ "GET" ], "Access-Control-Allow-Origin": [ "*" ] }, "NoDNSLink": false, "NoFetch": false, "PathPrefixes": [], "PublicGateways": null, "RootRedirect": "", "Writable": false }, "Identity": { "PeerID": "12D3KooWHqc31RKhjpRe8DcF4KsJSBS3qX3SqG9AByJrbYBnidzf" }, "Internal": {}, "Ipns": { "RecordLifetime": "", "RepublishPeriod": "", "ResolveCacheSize": 128 }, "Migration": { "DownloadSources": [], "Keep": "" }, "Mounts": { "FuseAllowOther": false, "IPFS": "/ipfs", "IPNS": "/ipns" }, "Peering": { "Peers": null }, "Pinning": { "RemoteServices": {} }, "Plugins": { "Plugins": null }, "Provider": { "Strategy": "" }, "Pubsub": { "DisableSigning": false, "Router": "" }, "Reprovider": { "Interval": "12h", "Strategy": "all" }, "Routing": { "Type": "dht" }, "Swarm": { "AddrFilters": [ "/ip4/10.0.0.0/ipcidr/8", "/ip4/100.64.0.0/ipcidr/10", "/ip4/169.254.0.0/ipcidr/16", "/ip4/172.16.0.0/ipcidr/12", "/ip4/192.0.0.0/ipcidr/24", "/ip4/192.0.2.0/ipcidr/24", "/ip4/192.168.0.0/ipcidr/16", "/ip4/198.18.0.0/ipcidr/15", "/ip4/198.51.100.0/ipcidr/24", "/ip4/203.0.113.0/ipcidr/24", "/ip4/240.0.0.0/ipcidr/4", "/ip6/100::/ipcidr/64", "/ip6/2001:2::/ipcidr/48", "/ip6/2001:db8::/ipcidr/32", "/ip6/fc00::/ipcidr/7", "/ip6/fe80::/ipcidr/10" ], "ConnMgr": { "GracePeriod": "20s", "HighWater": 900, "LowWater": 600, "Type": "basic" }, "DisableBandwidthMetrics": false, "DisableNatPortMap": true, "EnableAutoRelay": false, "EnableRelayHop": false, "Transports": { "Multiplexers": {}, "Network": {}, "Security": {} } } }Description
We got a Abuse Report from our ISP because they detected NetScan/PortScan on our machine, when go-ipfs is running.
Following up the comment in #1226 (comment) we checked our configuration and saw that the filter rules are in there.
First we ran the systemd service, and now via
ipfs daemonwith the same issue.For information, we initialised our service with the server profile.
It seems that the active configuration is ignored and the local lookup is still active.
The text was updated successfully, but these errors were encountered: