portindex: multi-threading appears bottlenecked for 10.13, 10.12, and perhaps earlier

[mascguy (Christopher Nielsen)]

Starting with release 2.9.0, portindex is now multi-threaded. And it works beautifully, providing near-linear speedup based on the number of CPU cores available.

Testing on my 24-core MacPro2012, things work perfectly for 10.15 and 10.14, with shockingly-fast indexing. And the portindex process consumes nearly all available CPU cores, as expected. (For my system, at least 2100%. Note quite the 2400% of all cores, but close enough.)

However, for 10.13 and 10.12, multi-threading appears to be significantly bottlenecked: On my system, the portindex process barely consumes two CPU cores. (And the performance matches the CPU usage, as it's nearly an order-of-magnitude slower than it is on 10.15 and 10.14.)

Inspecting the portindex process itself, the number of threads appears to be correct in all cases. (It's always 27, even for 10.13 and 10.12.) So it doesn't appear to be an issue with CPU core count detection. But rather, something is severely slowing things down.

Of note, testing was done on two different MacPorts releases - 2.9.1 and 2.9.3, both sourced from the official MacPorts installers - but the throttling appears consistent across both.

[mascguy (Christopher Nielsen)]

Josh, let me know if there are any minor changes that I can make to portindex, in terms tracing. (It feels like an eternity since we last worked on anything similar, so a bit rusty here.)

[jmroot (Joshua Root)]

If you want to just time how long different sections of code are taking to execute, you can do something like this:

# init:
set section somename
set firsttime [clock microseconds]
set prevtime [clock microseconds]
# between each section:
puts "section $section took [expr {[clock microseconds] - $prevtime}] µs"
set section anothername
set prevtime [clock microseconds]
# end:
puts "total took [expr {[clock microseconds] - $firsttime}] µs"

[cooljeanius (Eric Gallager)]

I've noticed this when trying to adapt the MacPorts CI script that runs on GitHub Actions with my own local Portfile repository; it results in runs taking upwards of 15 minutes to complete...