cluster: do not put ephemeral creds in a snapshot

Previously, the security manager was inserting ephemeral credentials
into the snapshot that is reloaded on controller log replay. This is a
problem because, upon log replay, RP will reload an ephemeral credential
that has a username and password but no principal. This leads to
authorization failures.

Therefore, this commit stops the security manager from insterting
ephemeral credentials into a snapshot.

Fixes redpanda-data#11141

src/v/cluster/security_manager.cc

@@ -153,8 +153,22 @@ security_manager::fill_snapshot(controller_snapshot& controller_snap) const {
 
     for (const auto& cred : _credentials.local()) {
         ss::visit(cred.second, [&](security::scram_credential scram) {
-            snapshot.user_credentials.push_back(user_and_credential{
-              security::credential_user{cred.first}, std::move(scram)});
+            if (scram.principal().has_value()) {
+                // Do not write ephemeral users to disk
+                if (
+                  scram.principal()->type()
+                  != security::principal_type::ephemeral_user) {
+                    snapshot.user_credentials.push_back(user_and_credential{
+                      security::credential_user{cred.first}, std::move(scram)});
+                }
+            } else {
+                // Regular users may not have a defined principal. For example,
+                // those users created from HTTP POST security/users (See
+                // parse_scram_credential() in the Admin server). Therefore, add
+                // these users into the snapshot.
+                snapshot.user_credentials.push_back(user_and_credential{
+                  security::credential_user{cred.first}, std::move(scram)});
+            }
         });
         co_await ss::coroutine::maybe_yield();
     }