I'm cc'ing this to the developers list so others are aware of the MKL 10.1
issue - perhaps we can all discuss it further after Christmas....
Hi Dave,
> Please examine the following patch and see if other things
> need to be fixed
> as well....thx...dac
That looks correct. Thanks for spotting this that is what I get for trying
to do three things at once. I am away for the next few days but between
Christmas and New Year I plan to fully test the QM (and other Amber code) on
as many systems as I can find.
Also we have an issue with MKL 10.1 - it uses openMP threading by default if
you don't set OMP_NUM_THREADS. It detects the number of cpus automatically.
So if you are running on a 4x4core machine and run mpirun -np 8 - then
everytime you get to an MKL call (say vdinvsqrt) it will spawn 8 openMP
threads per mpi thread and you end up wth 64 threads doing the vdinvsqrt
which is a disaster.
We can turn it off with MKL_SERIAL or OMP_NUM_THREADS=1 but I would like the
openMP threading for the QM diagonalization calls, since only the master
thread does diagonalization. So we need to work out a way to deal with this.
One option is to put explicit calls in the code that set the number of OMP
threads manually. I.e. we would have one early in multisander that set it to
1 and then around the diag call I would set it to NCPUS and then back to 1.
I believe though that you can only make the calls to set the number of
threads if you compiled with openmp support so we may have to think up other
options - or update the documentation to reflect this. I think other math
libraries like ACML and ESSL will do the same thing as MKL in their next
release. We should probably talk about it after Christmas.
All the best
Ross
/\
\/
|\oss Walker
| HPC Consultant and Staff Scientist |
| San Diego Supercomputer Center |
| Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
|
http://www.rosswalker.co.uk | PGP Key available on request |
Note: Electronic Mail is not secure, has no guarantee of delivery, may not
be read every day, and should not be used for urgent or sensitive issues.
Received on Sun Dec 23 2007 - 06:07:33 PST
