Hi All,
I have removed the -gpu option from the command line for pmemd.cuda and
pmemd.cuda.MPI in the amber 12 git tree. The reason for this is that it does
not work properly when running in parallel and so leads to inconsistencies
about how to select gpus when running on nodes with more than one GPU. The
more consistent approach, and the one recommended for parallel anyway is to
use the environment variable CUDA_VISIBLE_DEVICES. This should now be used
to select which GPU to use in serial as well.
For more details see here: http://ambermd.org/gpus/#Running (Obviously I
have not updated this for NOT having the -gpu option since that is for AMBER
11 so just ignore that part of the discussion).
If you want the job to run on physical GPU 0 you just do:
export CUDA_VISIBLE_DEVICES="0"
If you want it to run on physical GPU 2 you would do:
export CUDA_VISIBLE_DEVICES="2"
Note in both cases this makes the GPU with this physical ID actually have
logical GPU ID 0 for the shell you are in. It physically hides all other
GPUs from that shell.
If you want to run in parallel on say physical GPUs 1 and 5 you would do:
export CUDA_VISIBLE_DEVICES="1,5"
This would give you 2 logical GPUs in the shell you are in. Physical GPU 1
with logical ID 0 and Physical GPU 5 with logical id 1. Then when you run
with mpirun -np 2 the code will automatically use GPUs 1 and 5 since these
are the only two gpus it can physically see. If you accidentally ran with
-np 4 here you would get 2 threads on physcial GPU 1 and 2 on physical GPU
5. (You can test how this works by setting the environment variable and then
running deviceQuery from the cudaSDK.)
Hope that makes sense. Removing the -gpu command line option makes things (I
hope) consistent for running in serial or parallel.
All the best
Ross
/\
\/
|\oss Walker
---------------------------------------------------------
| Assistant Research Professor |
| San Diego Supercomputer Center |
| Adjunct Assistant Professor |
| Dept. of Chemistry and Biochemistry |
| University of California San Diego |
| NVIDIA Fellow |
|
http://www.rosswalker.co.uk |
http://www.wmd-lab.org/ |
| Tel: +1 858 822 0854 | EMail:- ross.rosswalker.co.uk |
---------------------------------------------------------
Note: Electronic Mail is not secure, has no guarantee of delivery, may not
be read every day, and should not be used for urgent or sensitive issues.
_______________________________________________
AMBER-Developers mailing list
AMBER-Developers.ambermd.org
http://lists.ambermd.org/mailman/listinfo/amber-developers
Received on Fri Apr 08 2011 - 10:30:10 PDT
