Hi,
I think this MKL threads issue existed with 10.0 too.
MKL has an API for setting its threads independent of OpenMP.
One problem with their API is a lack of inquiry functions.
Probably we can set the MKL threads to AMBER_MKL_THREADS
for everything but your stuff via
mkl_set_num_threads ( AMBER_MKL_THREADS ).
And then build on that adding ROSS_AMBER_MKL_THREADS...
See MKL user guide section
Using Intel® MKL Parallelism
for their API.
Scott
On Fri, 21 Dec 2007, Ross Walker wrote:
> I'm cc'ing this to the developers list so others are aware of the MKL 10.1
> issue - perhaps we can all discuss it further after Christmas....
>
> Hi Dave,
>
> > Please examine the following patch and see if other things
> > need to be fixed
> > as well....thx...dac
>
> That looks correct. Thanks for spotting this that is what I get for trying
> to do three things at once. I am away for the next few days but between
> Christmas and New Year I plan to fully test the QM (and other Amber code) on
> as many systems as I can find.
>
> Also we have an issue with MKL 10.1 - it uses openMP threading by default if
> you don't set OMP_NUM_THREADS. It detects the number of cpus automatically.
> So if you are running on a 4x4core machine and run mpirun -np 8 - then
> everytime you get to an MKL call (say vdinvsqrt) it will spawn 8 openMP
> threads per mpi thread and you end up wth 64 threads doing the vdinvsqrt
> which is a disaster.
>
> We can turn it off with MKL_SERIAL or OMP_NUM_THREADS=1 but I would like the
> openMP threading for the QM diagonalization calls, since only the master
> thread does diagonalization. So we need to work out a way to deal with this.
> One option is to put explicit calls in the code that set the number of OMP
> threads manually. I.e. we would have one early in multisander that set it to
> 1 and then around the diag call I would set it to NCPUS and then back to 1.
> I believe though that you can only make the calls to set the number of
> threads if you compiled with openmp support so we may have to think up other
> options - or update the documentation to reflect this. I think other math
> libraries like ACML and ESSL will do the same thing as MKL in their next
> release. We should probably talk about it after Christmas.
>
> All the best
> Ross
>
> /\
> \/
> |\oss Walker
>
> | HPC Consultant and Staff Scientist |
> | San Diego Supercomputer Center |
> | Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
> | http://www.rosswalker.co.uk | PGP Key available on request |
>
> Note: Electronic Mail is not secure, has no guarantee of delivery, may not
> be read every day, and should not be used for urgent or sensitive issues.
>
>
>
--
Received on Sun Dec 23 2007 - 06:07:37 PST
