From this message it shows that the container is running out of memory for what you are trying to achieve. I’ll have a close look to see what is the best approach but you can probably resolve this issue by breaking down the grids into smaller ones so every container needs less memory. It is unlikely that we want to increase the memory that’s available for containers for all the jobs on Pollination. I will have a closer look once I get a chance.
It might also have something to do with sensor count. I have merged all sensors to one sensorgrid and matched the “min_sen_count” by dividing the number of sensors on number of CPUs. This might not be the best solution for large models:
Will try to break down the grids into smaller ones and also tweak the min_sensor_count.
This should be improved with the changes that @mikkel is making to our postprocessing routine but regardless I don’t think a single sensor grid with 1M sensors is a good idea.
Thank you @mostapha for testing and feedback! Smaller grids did the job. I also did some testing with different min_sensor_count. min_sensor_count = 4425 worked, but 1000 and 200 did throw an error (with 50 CPUs). I guess both grid size and min_sensor_count should be considered when working with large models/sensor points. Thanks for helping out!
Hi @tk1, Thank you for the update. That’s correct.
The 2 inputs have 2 different reasons. The CPU count input in the recipe is used to distribute the input sensor grids. The minimum_sensor_count input is to ensure that the grids don’t get really small.
Say you have 100 sensors and then set the CPU count to 100. With a minimum sensor count set to 1, the recipe will generate 100 grids and assigns one sensor to each to distribute them to run in parallel. This will add a lot of overhead to the calculation. To avoid scenarios like this you should set the minimum sensor count to a number that is big enough. For the same scenario if the minimum sensor count is set to 50 then it will overwrite the logic and only generates 2 sensor grids.
Then there is another step in which we read all the distributed results and merge them back together to match the structure of the input sensor grids. That’s where your case was facing an issue and was running out of memory because it had to read 1000,000 data points. That needs 33.2 GB of memory. We assign ~4 GB to each container. As you noticed, you can overcome this issue by breaking the initial sensor grids input into smaller sensor grids.
Hi @tk1! I have one more update here after talking to @chriswmackey! If you are looking for cumulative radiation, you might want to use the Cumulative Radiation recipe. It is faster and it doesn’t break down the calculation for direct vs indirect which doesn’t seem to be a concern in your case.
Thank you @mostapha for taking your time. You are right, I don’t need to break down the results into direct and indirect. I need however hourly annual data. Is this also possible with one of the other recipes?
We have a recipe (Pollination Cloud App) but we never updated it to accept a radiance folder as an input. That shouldn’t be hard to do. My understanding is that we need to:
update the version of pollination-honeybee-radiance?
edit the list of acceptable extensions in the recipe inputs.
@chriswmackey, is this assumption correct? If that is the case, @mikkel should be able to help with the fix.
There might be a few more updates needed for that recipe since I believe that it pre-dates the even subdivision of sensors for a better parallelized calculation. But you’ll need to do those two things at a minimum.
Before we invest time in this we should make sure that @tk1 is ok with the underlying assumption of that recipe, which is that THERE ARE NO ABMIENT BOUNCES. It will model the direct and diffuse sun very accurately but, if you need to account for bounces and you need hourly data, then Annual Irradiance is the only recipe that fits the bill.
Thanks for elaborating and feedback @mostapha and @chriswmackey . The recipe overview on pollination with data and graph is helping a lot, great job.
Abmient bounces will be needed for more specific use cases such as PV calculations down the line. Probably need to keep using Annual Irradiance recipe, am I right? Is there however any post process capabilities with Pollination. Working with large datasets in Grasshopper using Arithmetic Operations is not preferable due to memory issues.
