magma_init() takes up to 2 seconds to init

Open discussion for MAGMA library (Matrix Algebra on GPU and Multicore Architectures)

magma_init() takes up to 2 seconds to init

Postby thanasis_giannis » Thu Aug 24, 2017 7:40 am

Is it normal for magma_init() to take up to 2 seconds to init? And if no, what could cause this?

Thank you in advance!
thanasis_giannis
 
Posts: 9
Joined: Thu Aug 24, 2017 7:35 am

Re: magma_init() takes up to 2 seconds to init

Postby mgates3 » Thu Aug 24, 2017 9:35 am

No, I wouldn't expect it to take that long. It queries the CUDA devices for their architecture (Fermi, Kepler, ...) and possibly makes a queue (stream) on each GPU. Assuming that you include magma_v2.h, instead of magma.h, you can try adding -DMAGMA_NO_V1 to CFLAGS in make.inc and recompiling MAGMA. That would disable creating any streams in magma_init.

What GPUs do you have? E.g., what is the output of a MAGMA tester (as below)?

Code: Select all
bunsen magma/testing> ./testing_dgetrf -n 100
% MAGMA 2.1.0 svn compiled for CUDA capability >= 3.5, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 16. MKL 11.3.0, MKL threads 16.
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% device 1: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 09:31:53 2017
% Usage: ./testing_dgetrf [options] [-h|--help]

% ngpu 1, version 1
%   M     N   CPU Gflop/s (sec)   GPU Gflop/s (sec)   |PA-LU|/(N*|A|)
%========================================================================
  100   100     ---   (  ---  )      0.05 (   0.01)     ---   


-mark
mgates3
 
Posts: 750
Joined: Fri Jan 06, 2012 2:13 pm

Re: magma_init() takes up to 2 seconds to init

Postby thanasis_giannis » Thu Aug 24, 2017 5:56 pm

Code: Select all
thanasis@cuda:/usr/local/magma-2.2.0/testing$ ./testing_dgetrf -n 100
% MAGMA 2.2.0  compiled for CUDA capability >= 3.0, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 1. MKL 11.3.0, MKL threads 1.
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 19:05:22 2017
% Usage: ./testing_dgetrf [options] [-h|--help]

% ngpu 1, version 1
%   M     N   CPU Gflop/s (sec)   GPU Gflop/s (sec)   |PA-LU|/(N*|A|)
%========================================================================
  100   100     ---   (  ---  )      2.36 (   0.00)     ---   
thanasis@cuda:/usr/local/magma-2.2.0/testing$



The output is this. I did put the -DMAGMA_NO_V1 yet the time is still the same.
thanasis_giannis
 
Posts: 9
Joined: Thu Aug 24, 2017 7:35 am

Re: magma_init() takes up to 2 seconds to init

Postby mgates3 » Fri Aug 25, 2017 3:34 pm

On my laptop, magma_init is reasonably quick, about 0.1 sec. On some of our servers, I found mixed results.

Code: Select all
k20:    1.35 sec
p100:   0.88 sec
k40:    6.21 sec
c2050:  1.15 sec


I was surprised these were that large. On any particular machine, the results varied a bit from run to run, perhaps 20%. Looking deeper, nearly all the time in magma_init is taken by cudaGetDeviceCount. If I put a different CUDA call, like cudaGetDeviceProperties, prior to cudaGetDeviceCount, then cudaGetDeviceProperties takes all the time and cudaGetDeviceCount takes negligible time. I surmise it is overhead in loading and initializing CUDA that happens on the first CUDA call. You can probably use CUDA's profiler to see this overhead.

Perhaps ask NVIDIA why the first call takes so long.

-mark
Last edited by mgates3 on Fri Aug 25, 2017 3:35 pm, edited 1 time in total.
Reason: typo
mgates3
 
Posts: 750
Joined: Fri Jan 06, 2012 2:13 pm

Re: magma_init() takes up to 2 seconds to init

Postby thanasis_giannis » Sun Aug 27, 2017 9:55 am

thanasis_giannis
 
Posts: 9
Joined: Thu Aug 24, 2017 7:35 am


Return to User discussion

Who is online

Users browsing this forum: No registered users and 7 guests

cron