nrf_rpc: implement timeout for response

[Damian-Nordic]

Introduce necessary changes to support timing out waiting for a command response:

1. Handle nrf_rpc_os_msg_get() returning null data pointer, indicating that the response has not arrived within the required maximum time.
2. Introduce context mutex to protect all context members. This is necessary because now the response may arrive after the initiating thread has already released the context.

Note that 2 may not sufficient solution to address another scenario where the context is freed by the initiator but is then reused before a delayed response arrives - to solve this, we likely need to introduce the concept of session ID or conversation ID to detect outdated packets.

manifest-pr-skip

[Damian-Nordic]

@doki-nordic @grochu Friendly reminder to take a look.

[grochu]

Sorry for not having reviewed in shorter time, I hoped someone more experienced with nRF RPC could do it. Anyway the changes technically are OK to me, I just have some minor comments from the documentation perspective.

[doki-nordic]

I started reviewing this PR, but since I have limited time to review it, I did not finalized it. I have concerns about mutex locking logic. Do we really need to lock it in context alloc function? When locking and unlocking logic is spread across multiple functions and multiple files, there is higher risk of edge cases or time races that may lead to dead lock. I am concern about context reusing functionality. Can you do simple context reusing test on this PR? See docs for some details about it: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrfxlib/nrf_rpc/doc/architecture.html#id5 . This will happen, for example, when the local side sends a command and waits for a response, next the remote send another command back to local within the command handler context. In this case, context on local side should be reused.

[Damian-Nordic]

I started reviewing this PR, but since I have limited time to review it, I did not finalized it. I have concerns about mutex locking logic. Do we really need to lock it in context alloc function? When locking and unlocking logic is spread across multiple functions and multiple files, there is higher risk of edge cases or time races that may lead to dead lock. I am concern about context reusing functionality. Can you do simple context reusing test on this PR? See docs for some details about it: https://docs.nordicsemi.com/bundle/ncs-latest/page/nrfxlib/nrf_rpc/doc/architecture.html#id5 . This will happen, for example, when the local side sends a command and waits for a response, next the remote send another command back to local within the command handler context. In this case, context on local side should be reused.

cmd_ctx_alloc may seem counterintuitive but I didn't find a better place to do the locking:

1. cmd_ctx_alloc already modifies the context to initialize it, so it must lock the mutex in order to prevent another thread from using a partially initialized context when e.g. a delayed response arrives. If we wanted to do lock/unlock every time in a single function, we would need to do this in many functions, such as cmd_ctx_alloc, cmd_ctx_reserve, nrf_rpc_cmd_common. I thought this would be less efficient.
2. cmd_ctx_alloc is called only when a new context is allocated - in the case of context reusing, this won't be called again. If we moved locking to e.g. nrf_rpc_cmd_common, we might be going to lock an already locked mutex, so we would need to assume the mutex is reentrant or add an extra check.

I'm happy to document it better, or rework this if you prefer locking the mutex in individual functions, but I thought doing this the current way actually simplifies the design - you always run nRF RPC code with the context mutex held and unlock it only when waiting for the peer or when releasing the context.

And yeah, I can run a test like that :)

[Damian-Nordic]

@grochu I addressed your comments, please take another look.
@doki-nordic I tested this code with the context reusing (implemented UDP callback which retrieves extra information about the received UDP packet from the remote device) and it works as expected.
Let me know if you have time for a call so I can provide the rationale for the current design.

[grochu]

Approving to unblock development. Let's discuss the context locking with mutex later in another context.