Trouble Running Deployed Model: Exception: Argument passed to at() was not in the map.

[mstapelberg]

Hi There,

I am trying to run a model I trained based off 50 structures from an AIMD simulation of Tungsten at 3300K as an example.

I have attached my config.yaml file
example.txt

I followed the instruction on https://github.com/mir-group/pair_allegro/tree/stress to install as I need NPT for my future simulations

I made sure to install pytorch 1.11.0 (cuda 11.3 version) with pip and libtorch with C+11 supporting cuda 11.3 + MKL-include for the pair-allegro commands

I did not install with Kokkos.

I am able to train models, evaluate, and deploy them, however I notice they typically have 2-3x higher force mae and 2x higher E/N mae (another problem for another day perhaps).

I had to write extxyz files myself since ase doesn't seem to be able to read my OUTCAR files, but I made sure to copy the exact format of the tutorial sitraj.xyz.

I wanted to try and test pair_allegro on a simple single point calculation

# ---------- 1. Initialize simulation ---------------------
units metal
atom_style atomic
dimension  3
boundary   p p p
read_data lammps.data

# ---------- 2. Specify interatomic potential ---------------------
#pair_style eam/alloy
#pair_coeff * * /home/bond/Software/lammps-17Nov16/potentials/Al_zhou.eam.alloy  Al
pair_style allegro3232
pair_coeff * * ../../potentials/npt_32_w.pth W 

# pair_style lj/cut 4.5
# pair_coeff 1 1 0.392 2.620 4.5

# ---------- 3. Run single point calculation  ---------------------
thermo_style custom step pe lx ly lz press pxx pyy pzz
run 0

# ---- 4. Define and print useful variables -------------
variable natoms equal "count(all)"
variable totenergy equal "pe"
variable length equal "lx"

print "Total energy (eV) = ${totenergy}"
print "Number of atoms = ${natoms}"
print "Lattice constant (Angstoms) = ${length}"

(I create a supercell with cubic unit cells of tungsten, 10 x 10 x 10 using ase).

However, I get the following error.

[[24369,1],0]: A high-performance Open MPI point-to-point messaging module
was unable to find any relevant network interfaces:

Module: OpenFabrics (openib)
  Host: gpu-rtx6000-02

Another transport will be used instead, although this may result in
lower performance.

NOTE: You can disable this warning by setting the MCA parameter
btl_base_warn_component_unused to 0.
--------------------------------------------------------------------------
LAMMPS (21 Nov 2023)
OMP_NUM_THREADS environment is not set. Defaulting to 1 thread. (src/comm.cpp:98)
  using 1 OpenMP thread(s) per MPI task
Reading data file ...
  orthogonal box = (0 0 0) to (31.652 31.652 31.652)
  1 by 1 by 1 MPI processor grid
  reading atoms ...
  2000 atoms
  read_data CPU = 0.369 seconds
Allegro is using input precision f and output precision f
Allegro is using device cuda
Allegro: Loading model from ../../potentials/npt_32_w.pth
Allegro: Freezing TorchScript model...
Type mapping:
Allegro type | Allegro name | LAMMPS type | LAMMPS name
0 | W | 1 | W
WARNING: No fixes with time integration, atoms won't move (src/verlet.cpp:60)
Neighbor list info ...
  update: every = 1 steps, delay = 0 steps, check = yes
  max neighbors/atom: 2000, page size: 100000
  master list distance cutoff = 6
  ghost atom cutoff = 6
  binsize = 3, bins = 11 11 11
  1 neighbor lists, perpetual/occasional/extra = 1 0 0
  (1) pair allegro3232, perpetual
      attributes: full, newton on, ghost
      pair build: full/bin/ghost
      stencil: full/ghost/bin/3d
      bin: standard
Setting up Verlet run ...
  Unit style    : metal
  Current step  : 0
  Time step     : 0.001
Exception: Argument passed to at() was not in the map.
--------------------------------------------------------------------------
MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD
with errorcode 1.

NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes.
You may or may not see output from other processes, depending on
exactly when Open MPI kills them.

After doing some googling it may be I installed something incorrectly? But I am unsure of how to begin this debugging process and would appreciate any help you can provide!

Thanks,
Myles

[Linux-cpp-lisp]

Hi @mstapelberg ,

Thanks for your interest in our code!

Your network has only ForceOutput and not StressForceOutput, which means there is no stress output from the model when the stress branch pair_allegro (which by the way the multicut branch is latest) tries to access, leading to this error. Here's an older example of how to do this by redeploying your trained model (mir-group/nequip#69 (comment)), for example.

[mstapelberg]

Hi @Linux-cpp-lisp

Thank you very much for your reply and help with this!

I apologize for my slow reply, I have done as you recommended:

nequip-deploy build --model example.yaml deployed-stress-w.pth
INFO:root:Building model from config...
Traceback (most recent call last):
  File "/home/myless/.mambaforge/envs/allegro/bin/nequip-deploy", line 10, in <module>
    sys.exit(main())
  File "/home/myless/.mambaforge/envs/allegro/lib/python3.10/site-packages/nequip/scripts/deploy.py", line 219, in main
    model = model_from_config(config, deploy=True)
  File "/home/myless/.mambaforge/envs/allegro/lib/python3.10/site-packages/nequip/model/_build.py", line 96, in model_from_config
    model = builder(**params)
  File "/home/myless/.mambaforge/envs/allegro/lib/python3.10/site-packages/allegro/model/_allegro.py", line 32, in Allegro
    builder_utils.add_avg_num_neighbors(
  File "/home/myless/.mambaforge/envs/allegro/lib/python3.10/site-packages/nequip/model/builder_utils.py", line 35, in add_avg_num_neighbors
    raise ValueError("avg_num_neighbors = auto but initialize is False")
ValueError: avg_num_neighbors = auto but initialize is False

I have attached my updated example.yaml,
example.txt

I am currently on the main branch for Allegro. Is there a particular branch you recommend I use for Nequip? Noted for using multicut on the pair_allegro repo.

Thanks!
Myles

[Linux-cpp-lisp]

Regarding branches, for now main on Allegro and develop on nequip is good.

Regarding the error, you could either find avg_num_neighbors in config.yaml of your old training and just put that in the redeployment yaml, or you could also make a copy of your old training dir, change ForceOutput to StressForceOutput in config.yaml and just redeploy the --train-dir. Both should work, but let me know if not.