runtime: fix possible deadlock in accounts_db

[BurtonQin]

Problem

Similar to #10466

In runtime/src/accounts_db.rs:

1. conflicting lock order:
   storage is before accounts_index in generate_index()
   

   solana/runtime/src/accounts_db.rs

   Lines 1808 to 1812 in 4cfb758

   	pub fn generate_index(&self) {
   	let storage = self.storage.read().unwrap();
   	let mut slots: Vec<Slot> = storage.0.keys().cloned().collect();
   	slots.sort();
   	let mut accounts_index = self.accounts_index.write().unwrap();

   
   However, the lock order conflicts with any other place involving the two locks, e.g.
   

   solana/runtime/src/accounts_db.rs

   Lines 662 to 667 in 4cfb758

   	let accounts_index = self.accounts_index.read().unwrap();
   	// Calculate store counts as if everything was purged
   	// Then purge if we can
   	let mut store_counts: HashMap<AppendVecId, usize> = HashMap::new();
   	let storage = self.storage.read().unwrap();

   
   The fix is to swap the order of the two locks.
2. double read lock:
   The first read lock is in generate_index()
   

   solana/runtime/src/accounts_db.rs

   Lines 1808 to 1809 in 4cfb758

   	pub fn generate_index(&self) {
   	let storage = self.storage.read().unwrap();

   
   Then scan_account_storage() is called at L1823
   

   solana/runtime/src/accounts_db.rs

   Line 1823 in 4cfb758

   .scan_account_storage(

   
   The second lock is at L969
   

   solana/runtime/src/accounts_db.rs

   Lines 967 to 969 in 4cfb758

   	let storage_maps: Vec<Arc<AccountStorageEntry>> = self
   	.storage
   	.read()

   
   The fix is to use a lock propagation way to avoid recursively acquiring the same lock.
   Separate the lock acquisition from scan_account_storage and pass the ref to read lock guard into the new scan_account_storage_inner.

There remains one possible double read lock that I find really tricky to tackle.

In ledger/src/blockstore.rs:

1. The first read lock is at 1501 in get_confirmed_block():
   

   solana/ledger/src/blockstore.rs

   Line 1501 in 4cfb758

   let lowest_cleanup_slot = self.lowest_cleanup_slot.read().unwrap();

2. Two places where get_slot_entries() is called:
   

   solana/ledger/src/blockstore.rs

   Line 1518 in 4cfb758

   let slot_entries = self.get_slot_entries(slot, 0)?;

   
   

   solana/ledger/src/blockstore.rs

   Lines 1524 to 1525 in 4cfb758

   	let parent_slot_entries = self
   	.get_slot_entries(slot_meta.parent_slot, 0)

3. get_slot_entries_with_shred_info():
   

   solana/ledger/src/blockstore.rs

   Lines 1866 to 1867 in 4cfb758

   	pub fn get_slot_entries(&self, slot: Slot, shred_start_index: u64) -> Result<Vec<Entry>> {
   	self.get_slot_entries_with_shred_info(slot, shred_start_index, false)

4. get_completed_ranges():
   

   solana/ledger/src/blockstore.rs

   Line 1883 in 4cfb758

   let (completed_ranges, slot_meta) = self.get_completed_ranges(slot, start_index)?;

5. The second read lock is at L1915:
   

   solana/ledger/src/blockstore.rs

   Line 1915 in 4cfb758

   let lowest_cleanup_slot = self.lowest_cleanup_slot.read().unwrap();

Summary of Changes

1. Swap the order of storage and accounts_index in generate_index().
2. Add an inner version of scan_account_storage to separate the lock acquisition.

[codecov]

Codecov Report

Merging #10469 into master will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##           master   #10469   +/-   ##
=======================================
  Coverage    81.6%    81.6%           
=======================================
  Files         296      296           
  Lines       69320    69323    +3     
=======================================
+ Hits        56622    56629    +7     
+ Misses      12698    12694    -4     

[ryoqun]

@BurtonQin Thanks for continued contribution! I'll take a look at this and the previous one today. Also, are you planning to submit more PRs? I want to know that, so that I'll be prepared to review other upcoming PRs as well. :)

[BurtonQin]

No, these are all that the current detector can find.

[ryoqun]

Perfect!! I'm merging!

There remains one possible double read lock that I find really tricky to tackle.
...

For this, I'll tackle later. Thanks for reporting all of these!