CVE-2023-53618: Undefined Security Weakness

In the Linux kernel, the following vulnerability has been resolved:

btrfs: reject invalid reloc tree root keys with stack dump

[BUG] Syzbot reported a crash that an ASSERT() got triggered inside prepare_to_merge().

That ASSERT() makes sure the reloc tree is properly pointed back by its subvolume tree.

[CAUSE] After more debugging output, it turns out we had an invalid reloc tree:

BTRFS error (device loop1): reloc tree mismatch, root 8 has no reloc root, expect reloc root key (-8, 132, 8) gen 17

Note the above root key is (TREE_RELOC_OBJECTID, ROOT_ITEM, QUOTA_TREE_OBJECTID), meaning it's a reloc tree for quota tree.

But reloc trees can only exist for subvolumes, as for non-subvolume trees, we just COW the involved tree block, no need to create a reloc tree since those tree blocks won't be shared with other trees.

Only subvolumes tree can share tree blocks with other trees (thus they have BTRFS_ROOT_SHAREABLE flag).

Thus this new debug output proves my previous assumption that corrupted on-disk data can trigger that ASSERT().

[FIX] Besides the dedicated fix and the graceful exit, also let tree-checker to check such root keys, to make sure reloc trees can only exist for subvolumes.