Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Don't assume memory layout of std::net::SocketAddr (master) #122

Merged
merged 2 commits into from
Nov 9, 2020
Merged

Don't assume memory layout of std::net::SocketAddr (master) #122

merged 2 commits into from
Nov 9, 2020

Conversation

faern
Copy link
Contributor

@faern faern commented Nov 8, 2020 •
edited
Loading

Porting #120 from the 0.3.x branch to master.

Fixes #121

faern
faern commented Nov 8, 2020
View reviewed changes
src/sys/unix.rs
fn from_s_addr(in_addr: libc::in_addr_t) -> Ipv4Addr {
in_addr.to_be().into()
pub(crate) fn from_in_addr(in_addr: in_addr) -> Ipv4Addr {
Ipv4Addr::from(in_addr.s_addr.to_ne_bytes())
Copy link
Contributor Author

@faern faern Nov 8, 2020 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These helper methods were really handy. Removed a lot of code from the sockaddr.rs file and got rid of a number of #[cfg(...)]s. So I made them pub(crate) so I could use them in the other module.

I also changed the abstraction one level. So instead of converting to/from the only field in the in_addr/in6_addr structs I made these helpers convert directly into that struct. This in turn helped remove some repeated code here in the two sys files as well.

faern
faern commented Nov 8, 2020
View reviewed changes
src/sys/unix.rs
}

fn from_s_addr(in_addr: libc::in_addr_t) -> Ipv4Addr {
in_addr.to_be().into()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

.to_be() here was not correct really... Well it yielded the same result, but it was not optimally from an understanding point of view IMO. in_addr is stored in BE. So u32::from_be would be more correct. But both of them just swap the bytes on a LE machine and does nothing on a BE machine, so it does not matter in practice.

Anyway. My solution uses to_ne_bytes to avoid a double swap (since the From<u32> impl for Ipv4Addr swaps the bytes back again).

@faern faern mentioned this pull request Nov 9, 2020
Merged
34 tasks
@Thomasdezeeuw Thomasdezeeuw merged commit 02cebb5 into rust-lang:master Nov 9, 2020
@Thomasdezeeuw
Copy link
Collaborator

Thanks @faern.

bors added a commit to rust-lang-ci/rust that referenced this pull request Jul 31, 2022
@bors
Auto merge of rust-lang#78802 - faern:simplify-socketaddr, r=joshtrip…
3405e40 
…lett

Implement network primitives with ideal Rust layout, not C system layout

This PR is the result of this internals forum thread: https://internals.rust-lang.org/t/why-are-socketaddrv4-socketaddrv6-based-on-low-level-sockaddr-in-6/13321.

Instead of basing `std:::net::{Ipv4Addr, Ipv6Addr, SocketAddrV4, SocketAddrV6}` on system (C) structs, they are encoded in a more optimal and idiomatic Rust way.

This changes the public API of std by introducing structural equality impls for all four types here, which means that `match ipv4addr { SOME_CONSTANT => ... }` will now compile, whereas previously this was an error. No other intentional changes are introduced to public API.

It's possible to observe the current layout of these types (e.g., by pointer casting); most but not all libraries which were found by Crater to do this have had updates issued and affected versions yanked. See report below.

### Benefits of this change

- It will become possible to move these fundamental network types from `std` into `core` ([RFC](rust-lang/rfcs#2832)).
- Some methods that can't be made `const fn`s today can be made `const fn`s with this change.
- `SocketAddrV4` only occupies 6 bytes instead of 16 bytes.
- These simple primitives become easier to read and uses less `unsafe`.
- Makes these types support structural equality, which means you can now (for instance) match an `Ipv4Addr` against a constant

### ~Remaining~ Previous problems

This change obviously changes the memory layout of the types. And it turns out some libraries invalidly assumes the memory layout and does very dangerous pointer casts to convert them. These libraries will have undefined behaviour and perform invalid memory access until patched.

- [x] - `mio` - Issue: tokio-rs/mio#1386.
  - [x] `0.7` branch tokio-rs/mio#1388
  - [x] `0.7.6` published tokio-rs/mio#1398
  - [x] Yank all `0.7` versions older than `0.7.6`
  - [x] Report `<0.7.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0081.html
- [x] - `socket2` - Issue: rust-lang/socket2#119.
  - [x] `0.3.x` branch rust-lang/socket2#120
  - [x] `0.3.16` published
  - [x] `master` branch rust-lang/socket2#122
  - [x] Yank all `0.3` versions older than `0.3.16`
  - [x] Report `<0.3.16` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0079.html
- [x] - `net2` - Issue: deprecrated/net2-rs#105
  - [x] deprecrated/net2-rs#106
  - [x] `0.2.36` published
  - [x] Yank all `0.2` versions older than `0.2.36`
  - [x] Report `<0.2.36` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0078.html
- [x] - `miow` - Issue: yoshuawuyts/miow#38
  - [x] `0.3.x` - yoshuawuyts/miow#39
  - [x] `0.3.6` published
  - [x] `0.2.x` - yoshuawuyts/miow#40
  - [x] `0.2.2` published
  - [x] Yanked all `0.2` versions older than `0.2.2`
  - [x] Yanked all `0.3` versions older than `0.3.6`
  - [x] Report `<0.2.2` and `<0.3.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0080.html
- [x] - `quinn master` (aka what became 0.7) - quinn-rs/quinn#968 quinn-rs/quinn#987
  - [x] - `quinn 0.6` - quinn-rs/quinn#1045
  - [x] - `quinn 0.5` - quinn-rs/quinn#1046
  - [x] - Release `0.7.0`, `0.6.2` and `0.5.4`
- [x] - `nb-connect` - smol-rs/nb-connect#1
  - [x] - Release `1.0.3`
  - [x] - Yank all versions older than `1.0.3`
- [x] - `shadowsocks-rust` - shadowsocks/shadowsocks-rust#462
- [ ] - `rio` - spacejam/rio#44
- [ ] - `seaslug` - spacejam/seaslug#1

#### Fixed crate versions

All crates I have found that assumed the memory layout have been fixed and published. The crates and versions that will continue working even as/if this PR is merged is (please upgrade these to help unblock this PR):

* `net2 0.2.36`
* `socket2 0.3.16`
* `miow 0.2.2`
* `miow 0.3.6`
* `mio 0.7.6`
* `mio 0.6.23` - Never had the invalid assumption itself, but has now been bumped to only allow fixed dependencies (`net2` + `miow`)
* `nb-connect 1.0.3`
* `quinn 0.5.4`
* `quinn 0.6.2`

### Release notes draft

This release changes the memory layout of `Ipv4Addr`, `Ipv6Addr`, `SocketAddrV4` and `SocketAddrV6`. The standard library no longer implements these as the corresponding `libc` structs (`sockaddr_in`, `sockaddr_in6` etc.). This internal representation was never exposed, but some crates relied on it anyway by unsafely transmuting. This change will cause those crates to make invalid memory accesses. Notably `net2 <0.2.36`, `socket2 <0.3.16`, `mio <0.7.6`, `miow <0.3.6` and a few other crates are affected. All known affected crates have been patched and have had fixed versions published over a year ago. If any affected crate is still in your dependency tree, you need to upgrade them before using this version of Rust.
@Thomasdezeeuw Thomasdezeeuw mentioned this pull request Aug 8, 2022
Open
workingjubilee pushed a commit to tcdi/postgrestd that referenced this pull request Sep 15, 2022
@bors
Auto merge of #78802 - faern:simplify-socketaddr, r=joshtriplett
06e77d2 
Implement network primitives with ideal Rust layout, not C system layout

This PR is the result of this internals forum thread: https://internals.rust-lang.org/t/why-are-socketaddrv4-socketaddrv6-based-on-low-level-sockaddr-in-6/13321.

Instead of basing `std:::net::{Ipv4Addr, Ipv6Addr, SocketAddrV4, SocketAddrV6}` on system (C) structs, they are encoded in a more optimal and idiomatic Rust way.

This changes the public API of std by introducing structural equality impls for all four types here, which means that `match ipv4addr { SOME_CONSTANT => ... }` will now compile, whereas previously this was an error. No other intentional changes are introduced to public API.

It's possible to observe the current layout of these types (e.g., by pointer casting); most but not all libraries which were found by Crater to do this have had updates issued and affected versions yanked. See report below.

### Benefits of this change

- It will become possible to move these fundamental network types from `std` into `core` ([RFC](rust-lang/rfcs#2832)).
- Some methods that can't be made `const fn`s today can be made `const fn`s with this change.
- `SocketAddrV4` only occupies 6 bytes instead of 16 bytes.
- These simple primitives become easier to read and uses less `unsafe`.
- Makes these types support structural equality, which means you can now (for instance) match an `Ipv4Addr` against a constant

### ~Remaining~ Previous problems

This change obviously changes the memory layout of the types. And it turns out some libraries invalidly assumes the memory layout and does very dangerous pointer casts to convert them. These libraries will have undefined behaviour and perform invalid memory access until patched.

- [x] - `mio` - Issue: tokio-rs/mio#1386.
  - [x] `0.7` branch tokio-rs/mio#1388
  - [x] `0.7.6` published tokio-rs/mio#1398
  - [x] Yank all `0.7` versions older than `0.7.6`
  - [x] Report `<0.7.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0081.html
- [x] - `socket2` - Issue: rust-lang/socket2#119.
  - [x] `0.3.x` branch rust-lang/socket2#120
  - [x] `0.3.16` published
  - [x] `master` branch rust-lang/socket2#122
  - [x] Yank all `0.3` versions older than `0.3.16`
  - [x] Report `<0.3.16` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0079.html
- [x] - `net2` - Issue: deprecrated/net2-rs#105
  - [x] deprecrated/net2-rs#106
  - [x] `0.2.36` published
  - [x] Yank all `0.2` versions older than `0.2.36`
  - [x] Report `<0.2.36` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0078.html
- [x] - `miow` - Issue: yoshuawuyts/miow#38
  - [x] `0.3.x` - yoshuawuyts/miow#39
  - [x] `0.3.6` published
  - [x] `0.2.x` - yoshuawuyts/miow#40
  - [x] `0.2.2` published
  - [x] Yanked all `0.2` versions older than `0.2.2`
  - [x] Yanked all `0.3` versions older than `0.3.6`
  - [x] Report `<0.2.2` and `<0.3.6` to RustSec Advisory Database https://rustsec.org/advisories/RUSTSEC-2020-0080.html
- [x] - `quinn master` (aka what became 0.7) - quinn-rs/quinn#968 quinn-rs/quinn#987
  - [x] - `quinn 0.6` - quinn-rs/quinn#1045
  - [x] - `quinn 0.5` - quinn-rs/quinn#1046
  - [x] - Release `0.7.0`, `0.6.2` and `0.5.4`
- [x] - `nb-connect` - smol-rs/nb-connect#1
  - [x] - Release `1.0.3`
  - [x] - Yank all versions older than `1.0.3`
- [x] - `shadowsocks-rust` - shadowsocks/shadowsocks-rust#462
- [ ] - `rio` - spacejam/rio#44
- [ ] - `seaslug` - spacejam/seaslug#1

#### Fixed crate versions

All crates I have found that assumed the memory layout have been fixed and published. The crates and versions that will continue working even as/if this PR is merged is (please upgrade these to help unblock this PR):

* `net2 0.2.36`
* `socket2 0.3.16`
* `miow 0.2.2`
* `miow 0.3.6`
* `mio 0.7.6`
* `mio 0.6.23` - Never had the invalid assumption itself, but has now been bumped to only allow fixed dependencies (`net2` + `miow`)
* `nb-connect 1.0.3`
* `quinn 0.5.4`
* `quinn 0.6.2`

### Release notes draft

This release changes the memory layout of `Ipv4Addr`, `Ipv6Addr`, `SocketAddrV4` and `SocketAddrV6`. The standard library no longer implements these as the corresponding `libc` structs (`sockaddr_in`, `sockaddr_in6` etc.). This internal representation was never exposed, but some crates relied on it anyway by unsafely transmuting. This change will cause those crates to make invalid memory accesses. Notably `net2 <0.2.36`, `socket2 <0.3.16`, `mio <0.7.6`, `miow <0.3.6` and a few other crates are affected. All known affected crates have been patched and have had fixed versions published over a year ago. If any affected crate is still in your dependency tree, you need to upgrade them before using this version of Rust.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Thomasdezeeuw Thomasdezeeuw Thomasdezeeuw approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Port pr #120 to master
2 participants
@faern @Thomasdezeeuw