filesystem-based dm-verity

[brauner]

I'm exploring the possibility of making dm-verity usable directly from the filesystem, i.e., to decouple the concept from the block layer and move it into the filesystem layer to supplement fsverity which is very different from dm-verity.

Erofs is a natural target because it allows to mount images directly as files and is used heavily already for container workloads. Being able to use dm-verity without having to use device mapper at all seems quite a natural thing to do.

Points of order:

• I have just started thinking about this and so all thoughts here are very very raw.

• I have not worked with either fsverity or dm-verity in-depth before and my current level of understanding is coarse (think cobblestones, not pebbles). Hence, this idea will be riddled with plot-holes, pot-holes, and inconsistencies.

• The fsverity implementation is custom to each filesystem as it is implemented on a per-file basis. The justification for this is given in [1]:

  It would require defining a filesystem-independent way to store the
  verity metadata. Extended attributes don’t work for this because (a) the
  Merkle tree may be gigabytes, but many filesystems assume that all
  xattrs fit into a single 4K filesystem block, and (b) ext4 and f2fs
  encryption doesn’t encrypt xattrs, yet the Merkle tree must be encrypted
  when the file contents are, because it stores hashes of the plaintext
  file contents.

  So the verity metadata would have to be stored in an actual file. Using
  a separate file would be very ugly, since the metadata is fundamentally
  part of the file to be protected, and it could cause problems where
  users could delete the real file but not the metadata file or vice
  versa. On the other hand, having it be in the same file would break
  applications unless filesystems’ notion of i_size were divorced from the
  VFS’s, which would be complex and require changes to all filesystems.

  That means that fsverity data is very tightly integrated into the on-disk format of the filesystem. That makes the fsverity data part of the filesystem attack vector. In contrast, dm-verity is a trivial format which is its key strength.

• fsverity and dm-verity are really distinct and it's somewhat unfortunate that the naming suggests that fsverity is dm-verity on the filesystem level.

  Whereas dm-verity protects metadata and data, fsverity only protects data and is file-based. So changes to metadata such as owner, mode, timestamps, and xattrs are still allowed, since these are not measured by fs-verity. Verity files can also still be renamed, deleted, and linked to. All of that is undesired.

  Some implementations such as composefs work around this limitation by moving the metadata to a separate (erofs) image and then using fsverity on the image file and ship the data files with fsverity enabled and combine it all with overlayfs.

  For use-cases where images are to be treated as immutable but without involving devices at all this is all unnecessary. The filesystem should just do it itself and it should be as dumb as possible.

• This is mostly regurgitated from the dm-verity documentation (not a 1:1 citation but marking it as such anyway):

  dm-verity supports writing a superblock to disk (verity header). The
  kernel implementation only reads the hash blocks following the header.
  Userspace is responsible to verify the integrity of the verity header.

  Directly following the header (and with sector number padded to the
  next hash block boundary) are the hash blocks which are stored a depth
  at a time (starting from the root), sorted in order of increasing
  index.

  The dm-verity data is located on a separate device or partition which is used to check the integrity of the data, metadata, directory structure etc. It is checked during first access. Once in the pagecache it's not re-verified. IOW, page-cache corruption is still a potential issue.

  The simply format should ideally be ported almost 1:1 to the filesystem itself without having to integrate it deeply into the on-disk format itself.

  For example with erofs, would it be possible to just specify a second file or second device that contains the verity data? erofs already supports secondary devices and files through the "device" mount option.

• If done right this could pave a way for allowing filesystem that implement support for this to be mountable by unprivileged users. And it would also mean that any usage of devices gets out of the way completely.

Would be good to get some input, scolding, ridiculing from:

@josefbacik
@poettering
@cyphar
@ebiggers
@hsiangkao
@amir73il

[hsiangkao]

Hi @brauner! I think the main difference between dmverity and fsverity is that dmverity markle tree can be landed into the image blob(s) itself, but fsverity merkle tree is currently landed in the post-EOF OOB area (I'm not sure if fsverity merkle tree can be imported or exported), so it's hard to (re)distribute.

My own premature brief point is that I wonder if it's possible to have common libverity library APIs in the kernel for both dmverity or fsverity because I think other than merkle tree block sizes or some header format, it should be basically the same.
With the common APIs, filesystems like EROFS can easily integrate the verity functionality (or even reusing dm-verity on-disk tree so users can choose to use the old dm-based dm-verity or use erofs verity itself) without any barrier.

I wrote some RFC for this too: https://lore.kernel.org/r/Y6KqpGscDV6u5AfQ@B-P7TQMD6M-0146.local
Although I'm very interested in this, but I don't have time to actually work on builtin verity support for now...