Parametrical Simulation taking too long

Hi,
Retaking pollination … i’m not sure if this thread matches others, so i decided to open it’s own one.
I’ve sent a series of runs (180). It is running 1:15 hours now and so far only one case is completed.
Creating this 180 cases took like 5 hours, so i prefer not having to do that again … :slight_smile:
Some of the cases are “heavy” to run (on local machine) and others should run very quickly.
Do you have any words of wisdom here?
This is the link.

Thanks,
-A.

Some update: Somehow 30 minutes after my previous post, where only one case was completed, now 179 are done.
Around of 2 hours to run all cases is fantastic. My estimate is that running all of them on my laptop will take something like 24 hours!! It is a huge time saver.
The question is it could be done faster? Something happened at the beginning of the job that it took so long?
Thanks,
-A.

3 Likes

Hi @ayezioro :wave:

We have introduced some changes to the CPU quotas we assign to accounts on Pollination recently. I will be writing a more in depth post this week to help users understand how CPU resources are allocated and how parallel run compete for CPU resources.

The short answer to explain why things might have been slower at the start is that our cluster was adding more machines to support your workflow which can take up to 5 minutes.

The reason why each run took longer to execute in parallel is because each task was competing with all the other tasks in the job to use one of the 100 available CPUs to you account. As such they all ran individually slower but much faster overall.

Could it be done faster? ABSOLUTELY! :smile: Everyone’s account is now limited to 100 parallel CPUs so that we can accommodate everyone’s compute needs at the same time. If we were to increase your quota to 200 CPUs I would imagine your job would take 1 hour (300 CPUs 30 mins, 400 CPUs 15 mins etc…).

1 Like

Hi @ayezioro, thank you so much for providing feedback and glad to see that all the 180 jobs are succeeded successfully! We are doing something right here! :sunglasses:

image

I agree that the user experience is not ideal. You probably expect to get each case done one by one before submitting the next case but as @antoinedao mentioned all the runs are submitted at the same time and because of that each run might need to wait for a longer period before it can really start the simulation. As a results you get them all finished almost around the same time. You will still save time as a whole for the job but each run feels to be taking too long to finish as the title of your post is suggesting.

Thanks @antoinedao and @mostapha !!
I understand completely after your explanations. Makes sense.
As i said i’m happy with the simulation time, if i compare the alternative (X12 times, at least).
You are definitely doing something right, @mostapha. Keep doing this and more!

-A.

2 Likes