How CPU utilization is distributed?

[srividyacb]

Is there any documentation on how CPU and memory allocation is done for a tile-reduce script?

A tile-reduce script which is running in a 32 CPU core machines uses all the cores to process 400 tiles uses all the cores initially for 4-5 tiles and then only one CPU is allocated till the end.

What might be the issue here? Is this the issue with cached memory or because of too many operation happening inside map.js?

[morganherlocker]

Hey @srividyacb, my guess is that you have a set of tiles that are a bit unbalanced, with a few tiles being significantly larger than the others. One option is to try tiling your data at a higher zoom to spread out the work more evenly across cores. Another thing to consider is whether your map script is in need of optimization, particularly in cases with larger tiles. Tiles that take significantly longer than others despite being only a bit larger are a good sign of either:

1. a memory leak causing the garbage collector to thrash, or...
2. an algorithm that is O(n^2). Sometimes an algorithm can only be formulated this way, but usually there is some sort of indexing work that can be done once up front to speed things up (like an R-Tree)

To rule out 1, I would check the task manager while the job is running and see how high the memory utilization climes on the busiest processes.

[srividyacb]

Thank you so much for responding @morganherlocker .

So just to understand again, one tile is assigned to one CPU core at once and execution time is dependent on the density of the tile. Is that right?