Understanding Iterations, Convergence, and Hardware Impact on Robyn Model Performance

[stdp95]

Hello, Robyn Community!

I've been experimenting with Robyn for a while now, and I've observed some intriguing behaviors and outcomes based on iterations, convergence, and hardware configurations. I would love to get insights from the community and developers on a few points to understand the underlying mechanics better and optimize my use of Robyn. Here are my observations and questions:

• Early Convergence in Iterations: When setting up a model with a high number of iterations per trial, say 15k, but the model seems to converge within the first 5k iterations, does Robyn continue to process all 15k iterations, or does it have a mechanism to stop early? For instance, if the hyperparameters are such that optimal convergence is achieved early, what's the behavior here? I'm curious about the pros and cons of letting the model run all iterations versus stopping early, especially regarding computational efficiency and model accuracy.

• Iteration and Trial Dynamics: Suppose I configure Robyn with 30k iterations across 5 trials. Is there a scenario where the model could converge at around 10k iterations but then fail to show improvement or even diverge in subsequent iterations up to 30k? What are the best strategies to identify and understand this early, ensuring computational resources aren't wasted if such a situation is possible?

• Hardware Configuration Anomalies: I've noticed some peculiarities regarding computational times across different hardware setups. For example, a configuration with 64 cores and 120GB RAM showed time savings for 10k iterations compared to a less powerful setup (32 cores, 64GB RAM). However, increasing the iterations to 40k resulted in both configurations taking the same amount of time, which seems counterintuitive. Has anyone else experienced similar behavior? What might be causing this, and how can we optimize hardware configurations for different iteration scales?

I appreciate any insights or experiences you could share regarding these points. Understanding these dynamics could greatly enhance how we utilize Robyn for marketing mix modeling and ensure we're leveraging computational resources efficiently.

Thank you all in advance for your contributions!

[stdp95]

@gufengzhou I'm Just checking to see if you had a chance to look into this.

[gufengzhou]

Hi, sorry for the late reply. The discussion section here is indeed not as closely monitored as issues or the FB group.
For your convergence questions, it's very difficult to predict convergence time, which is not only a Robyn challenge, but Nevergrad and actually many optimisation packages also don't provide convergence prediction. The practical reason is that it varies a lot from dataset to dataset. Therefore, if you see one dataset converges at 5k already over difference trials, you can actually confidently reduce it to around 5k. There's no auto-stop function neither. As of divergence with higher iteration, to be honest I haven't seen this so far and this shouldn't happen, at least not with Nevergrad.
Regarding the computation question, I'm really no expert here so I can't provide a clear answer. Hope someone in the community might? You could try asking it in the FB group too. https://www.facebook.com/groups/robynmmm/

[AdimDrewnik]

My experience is that increasing number of trials and iterations in most cases does very little in finding better fitting models. I have noticed that the biggest factor is seed. So I usually run Robyn with low number of trials and iterations but I run many seeds in order to explore the search space more diversely.