InCharacter: Evaluating Personality Fidelity in Role-Playing Agents through Psychological Interviews
- May. 2024: InCharacter got accepted to ACL 2024! See you in Bangkok.
Install necessary dependencies via:
pip install -r requirements.txt
No need to install ChatHaruhi separately; a fixed version of ChatHaruhi is already included in the code/ directory of this repository.
Enter the code folder.
cd code
Set your openai apikey in config.json. You can refer to config_template.json for its format.
To assess the personality of a specific role-playing agent (RPA), use the following commands:
Conduct personality test on a specific character:
python personality_tests.py --questionnaire_name BFI --character hutao --agent_type ChatHaruhi --agent_llm gpt-3.5 --evaluator_llm gpt-4 --eval_method interview_assess_batch_anonymous
Supported choices for eval_method include ['self_report' (SR), 'self_report_cot' (SR-CoT), 'expert_rating' (ER_batch), 'expert_rating_collective' (ER_all), 'option_conversion' (OC),'dimension_option_conversion' (d-OC)].
To reproduce our experiments on the 32 RPAs, please refer to code/run_experiments.py
Radar chart of BFI personalities of state-of-the-art RPAs (yellow) and the characters (blue). O, C, E, A, N stands for openness, consciousness, extroversion, agreeableness and neuroticism in the BFI.
Interview Response:
Result:
Self-report Response:
(May give options inconsistent with character behaviors)
Interview Response:
Result: