Opened 19 years ago

Closed 19 years ago

#4754 closed defect (wontfix)

Python statup time much slower then standard install (hardlinks?)

Reported by: nirs@… Owned by: macports-tickets@…
Priority: Normal Milestone:
Component: ports Version: 1.0
Keywords: Cc: mww@…, markd@…
Port:

Description

Python 2.4.1 installed by darwinports startup almost 3 times slower then Python built from source and installed at /usr/local.

$ time /usr/local/bin/python -c "print 'done'" done

real 0m0.046s user 0m0.050s sys 0m0.010s

$ time /opt/local/bin/python -c "print 'done'" done

real 0m0.114s user 0m0.020s sys 0m0.080s

This can have bad effect on CGI scripts. for example, this minimal CGI:

#!/path/to/python print """content-type: text/plain

hi """

Can run at 45 request per second with standard Python, but only 14 request per second when running with darwinports Python.

Here is a more "real" example. Most cgi need to import cgi and use a FieldStorage:

$ time /usr/local/bin/python -c "import cgi; args = cgi.FieldStorage(); print repr(args)" FieldStorage(None, None, [])

real 0m0.156s user 0m0.120s sys 0m0.050s

$ time /opt/local/bin/python -c "import cgi; args = cgi.FieldStorage(); print repr(args)" FieldStorage(None, None, [])

real 0m0.320s user 0m0.070s sys 0m0.260s

I suspect that this slowness is caused by using hardlinks.

Change History (17)

comment:1 Changed 19 years ago by nirs@…

More details: this python was installed by 'port install mod_python', which build python with some dynlib target.

comment:2 Changed 19 years ago by jberry@…

Is this a Panther system? Hardlinks are truly broken on Panther, and can eventually lead you to a system that is completely unusable. I just spent the day yesterday replacing a server that had turned into a block of tar, due to a daemon that used hardlinks as a regular part of it's operation.

This system bug has been fixed, apparently, on Tiger.

comment:3 Changed 19 years ago by nirs@…

Yea it is a Panther. The problem is Tiger has its own brokeness.

comment:4 Changed 19 years ago by nirs@…

I installed python again after cleaning:

# port clean --all python24 # port install -u python24 ---> Fetching python24 ---> Attempting to fetch Python-2.4.1.tar.bz2 from http://www.python.org//ftp/python/2.4.1/ ---> Verifying checksum(s) for python24 ---> Extracting python24 ---> Applying patches to python24 ---> Configuring python24 ---> Building python24 with target all libpython.2.4.1.dylib ---> Staging python24 into destroot ---> Packaging tgz archive for python24 2.4.1_5 ---> Installing python24 2.4.1_5 ---> Activating python24 2.4.1_5 ---> Cleaning python24

And I get the same results.

comment:5 Changed 19 years ago by blb@…

Nir, if this is truly a problem with hardlinks, simply reinstalling python alone won't help. You'll need to switch your DP installation from image to direct mode (by changing 'portinstalltype' in /opt/local/etc/ ports/ports.conf from 'image' to 'direct'), then redo all your installed ports. This switches DP to direct mode (you can no longer have active and inactive ports) but will get away from the hardlink issue.

comment:6 Changed 19 years ago by nirs@…

Thanks! I did not know there is such option. It little work to remove and install back all, I wish there was recursive uninstall.

comment:7 Changed 19 years ago by mww@…

I dont know if this is the problem - I'm using direct mode and have approximately the same startup-time as Nir. I've already tried to make a "real" copy of python2.4 and execute that, but that wont make a difference either. The only thing I can think of is that dp's python has a lot of auto-load modules available, while the self- installed has not (e. g. readline) and that loading those modules takes a short while.

comment:8 Changed 19 years ago by mww@…

ok, I've just took a look at "python -v -v" to see what it is loading: # trying readline.so # trying readlinemodule.so # trying readline.py # trying readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python24.zip/readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-darwin/ readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/ readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/plat-mac/lib- scriptpackages/readline.pyc # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readline.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readlinemodule.so # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readline.py # trying /opt/local/Library/Frameworks/Python.framework/Versions/2.4/lib/python2.4/lib-tk/ readline.pyc # trying /opt/local/lib/python2.4/lib-dynload/readline.so import readline # dynamically loaded from /opt/local/lib/python2.4/lib-dynload/readline.so --- I assume that the framework build looks into a lot more locations where to find its modules and takes a while to find readline. I suppose you didnt do a framework install but a plain unix one?

comment:9 Changed 19 years ago by mww@…

Cc: mww@… added

comment:10 Changed 19 years ago by nirs@…

I did a plain ./cofigure; make install.

comment:11 Changed 19 years ago by mww@…

btw. on 10.4: time /usr/bin/python -c "print 'done'" done

real 0m0.332s user 0m0.184s sys 0m0.084s

comment:12 Changed 19 years ago by nirs@…

That time suck :) But it depends on the machine, mine timed on G5 Dual 2G.

comment:13 Changed 19 years ago by nirs@…

It must be the hardlinks.

I uninstalled all ports, changed the install type to 'direct', and installed all back. Now there is only a tiny difference between standard python and dp python.

For example:

$ time /usr/local/bin/python2.4 -c 'import cgi; args = cgi.FieldStorage(); print repr(args)' FieldStorage(None, None, [])

real 0m0.154s user 0m0.080s sys 0m0.070s

$ time /opt/local/bin/python2.4 -c 'import cgi; args = cgi.FieldStorage(); print repr(args)' FieldStorage(None, None, [])

real 0m0.164s user 0m0.100s sys 0m0.070s

$ time /usr/local/bin/python2.4 -c 'print "done"' done

real 0m0.047s user 0m0.020s sys 0m0.020s

$ time /opt/local/bin/python2.4 -c 'print "done"' done

real 0m0.050s user 0m0.020s sys 0m0.040s

And in the minimal cgi test, 47 request per second with standard python vs 42 with dp python.

For most users this difference does not matter, maybe if you use it for to serve cgis, the little difference is worth the more trouble in installing python, apache 2, mod_python, mod_fastcgi etc manually. I never managed to install mod_python myself :)

comment:14 Changed 19 years ago by mww@…

(In reply to comment #12)

It must be the hardlinks.

no, its the framework build - please compare your /usr/bin/python to dp's python and you'll see that dp's python is equally fast/slow (about factor 2 slower on my albook);

comment:15 Changed 19 years ago by nirs@…

Look again at the results of same dp python using framework install, once when using 'image' install type, and then 'direct':

test image direct


minimal script 0.114s 0.050s script using cgi.FieldStorage 0.320s 0.164s minimal cgi (request per seconds) 14 4

Using direct give at least 2x improvement, practically same as non framework build of python.

comment:16 Changed 19 years ago by markd@…

Cc: markd@… added

Any objections to marking this WONTFIX?

comment:17 Changed 19 years ago by markd@…

Resolution: wontfix
Status: newclosed
Note: See TracTickets for help on using tickets.