examples: Adding Sugarscape IG - polars with numba vectorized loop

[adamamer20]

This PR introduces a Numba implementation for the loop in SugarscapePolars to optimize agent move calculations. Key points:

1. Implementation: Added a Numba-accelerated function to determine the best moves for agents, replacing the more complex DataFrame-based logic.

2. Validation: Implemented equality checks across different Polars implementations to ensure consistent results.

3. Performance Comparison:

   • Numba-CPU and Numba-Parallel are the fastest.
   • DataFrame looping is now the slowest due to added complexity for correct agent priority handling.
   • Numba GPU implementation is slower than non-vectorized CPU, likely due to data transfer overhead and the nature of the parallelism in this problem.

4. Numba Considerations:

   • Requires explicit type and size specifications for vectors.
   • Input vector must match output size.
   • Limited to 1D vector operations.

5. Performance Bottlenecks:

   • Major time consumption in join operations, particularly in neighborhood calculations and cell updates.
   • Potential for optimization with lazy evaluation and an improved backend.

6. Future Optimizations:

   • Explore lazy evaluation approaches for join operations.
   • Non-vectorized loop directly on Polars DataFrame (without converting to numpy vectors) might offer potential speed improvements.

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

[adamamer20]

This is ready.

[rht]

Why is the previous one, although more concise, not considered favorable?

[adamamer20]

Do you mean the previous implementation?

[rht]

Yes.

[rht]

How did you find out about the necessity of adding this condition? Useful for best practice / how-to guide on how this was overlooked previously.

[adamamer20]

I found that out when testing with many potential collisions.
The problem was the following:

1. Consider two agents targeting the same cell:
   • A mid-priority agent (higher in the agent order)
   • A low-priority agent (lower in the agent order)
2. The mid-priority agent has low preference for the cell
3. The low-priority agent has high preference for the cell
4. Without accounting for priority:
   • The mid-priority agent's best moves kept getting "stolen" by higher priority agents
   • This forced it to resort to lower preference target cells
   • However, these lower preference cells were often already taken by lower priority agents in previous iterations

The best-practice to prevent these race conditions, is to implement a "priority" count to ensure that each action is "legal".

The downside is that the "priority" needs to be recomputed at each iteration and I think that's why is slower than Numba now. After the Ibis refactoring we can check if lazy evaluation can help mitigate the issue.

[rht]

Sounds messy. I hope it is possible to abstract out the move process, so that the user only has to provide the function on how the agent chooses the optimal location to move to.

[rht]

Approved because my comments are questions instead of blocking requests.