how to communicate real time?

[ahmad-ahmadi95]

I have a question, I've trained a model in TensorFlow.Keras and now my python editor in the Unreal Engined 4.18.3, have all modules I need (i.e TensorFlow and Keras) but in the case of the driving simulator I need communication between python scripts and the Unreal Environment, how can I do that?

[getnamo]

You can check out the old qlearning branch examples for in the loop communication: https://github.com/getnamo/tensorflow-ue4-examples/blob/qlearn/Content/Scripts/PongAI.py

The basics of it is call a json input with your ai input e.g. wheel location, images as float arrays, etc, then run your python code inside the function https://github.com/getnamo/tensorflow-ue4-examples/blob/qlearn/Content/Scripts/PongAI.py#L63 and return a result. On unreal side use the updated outputs to update your simulation (e.g. steering).

[ahmad-ahmadi95]

thanks for sharing, but you know, using the pip command in the same directory I could update the TensorFlow, ((pip install TensorFlow == 2.6.1)) and now the python editor in the Unreal Engine 4.18.3 recognize (Tensorflow. Keras) but another question has arisen is, how can I use the Unreal output as an Input for python script? for example, I want a way of communication between python and the Unreal engine.
for example, my NN model takes a bunch of inputs and outputs desired steering degree. I require that the car in the Unreal give me its steering degree and compare it with my model output to produce an error.

[getnamo]

Convert your input to json and reshape the inputs into the size/dimensions your network expects.

There's a handy struct to json string converter which can be used to feed the input, see this section of the documentation https://github.com/getnamo/tensorflow-ue4#basic-json-string and https://github.com/getnamo/tensorflow-ue4#basic-json-string

Keep in mind typically NNs expect float array inputs.