EnergyPlus simulation runs successfuly but the run stays unfinished for a long time

patryk_wozniczka · August 27, 2021, 4:33am

Hi folks,

My simulations have been extremely slow lately or have been failing again. Is this the same issue (I’ve read in another thread that this has been addressed)?

mostapha · August 27, 2021, 1:34pm

Hey @patryk_wozniczka! It should be the same issue. I don’t see any jobs running on production.

@antoinedao or @tyler can help with updating the UI so you can access the results.

We thought it was resolved but it was deeper than what we thought originally and there were several combinations that could cause the issue of job status not being updated on the UI. The good news is that the jobs run successfully and we need to fix the update status. The other good news is that @tyler has made tremendous progress on this issue and should be able to push an update to production soon.

Sorry that this is taking us longer than ideal and thank you for being patient with us! We will get it fix for you!

patryk_wozniczka · August 27, 2021, 2:02pm

FYI this job for example (from 2 days ago) finished 79 of 144 runs within 1.5 hours (very quickly) but then slowed down and runs started failing. So I cancelled.

patryk_wozniczka · August 27, 2021, 2:04pm

Sounds good, @mostapha. Do you have any sense how long it might take you folks to fix this? The more reliable pollination is the easier it would be to integrate it into our workflows.

mostapha · August 27, 2021, 2:07pm

Thanks! I’m sure @tyler will try to recreate the issue and update you on this. We really appreciate reporting cases like this. The Job ID is 3ec3d6e4-9185-463e-93a3-bbb6e5ed08d8.

Let me give you an estimate after @tyler has a closer look. If this is the same issue then it should be really fast but that’s to be seen.

patryk_wozniczka · August 28, 2021, 3:11pm

433a5b8e-67a8-4592-a8ce-1bca20f0de20

Runs keep failing:

But it says Energy Plus Completed Successfully

mostapha · August 28, 2021, 3:49pm

Is there any errors in the .err file for this run? You can find it as one of the outputs.

tyler · August 31, 2021, 4:07am

Hi @patryk_wozniczka ! My apologies for the bad experience here.

I just merged the updates I’ve been working on to improve the issue we’ve been having with lost run status updates. Would you be able to try to run the job again and let us know if the issue persists?

In the meantime, I will recreate this job on our test cluster and see if I can find anything else.

patryk_wozniczka · August 31, 2021, 12:10pm

that’s good news!

I will run the job again.

@mostapha I did inspect the .err file (only some minor errors) and ran that model on a local machine to double check and it ran successfully.

patryk_wozniczka · September 1, 2021, 4:10pm

I ran the job again but increased the number of runs to 144 (which is what a complete study would be) and it is still very slow and failing:

Job ID: 6268445e-d1a9-4ebb-b536-3ed7c5e3070d

.err file says energy plus completed successfully with some errors

when I run these exact same models locally I’m able to read results from .sqls

tyler · September 1, 2021, 11:31pm

Thanks for letting us know. This turned out to be a different issue from the missing updates. It seems the output SQLite database files for these runs are very large (~2GB) and are causing issues in the compression step before they are moved to cloud storage.

We’re looking into some options and will let you know when we have a solution.

antoinedao · September 9, 2021, 7:09am

Hi @patryk_wozniczka

I wanted to give you an update on the fix we are rolling out to resolve the issue you encountered with your large simulations. As @tyler mentioned the run is failing because the process executing the run zips output files before staging them to another process which then unzips them and pushes them to our persistent storage layer. I know… It’s pretty convoluted

We have found a way around this that removes the need to zip and unzip output files and are testing it in our staging environment this week. I expect we will have a fix available for you by mid next week at the latest. This should result in the following outcomes:

Runs should no longer fail because of the size of your output files
Runs should be slightly faster as a whole zip + unzip step is removed from the process

Thanks again for pushing the platform to where it needs to be and helping us find bugs and optimise on top of them

antoinedao · September 14, 2021, 3:56pm

Hi @patryk_wozniczka ,

I’m happy to let you know that we have fixed, tested and pushed an update to the app which allows your larger energy simulations to run to completion as expected. I would be super grateful if you could try to re-run the parametric experiment you had issues with 2 weeks ago and confirm that it all runs as expected.

Thanks again for being patient with us during our early access/beta period and for helping us make Pollination a more reliable product!