magma_init() takes up to 2 seconds to init

Open discussion for MAGMA library (Matrix Algebra on GPU and Multicore Architectures)
Post Reply
thanasis_giannis
Posts: 9
Joined: Thu Aug 24, 2017 7:35 am

magma_init() takes up to 2 seconds to init

Post by thanasis_giannis » Thu Aug 24, 2017 7:40 am

Is it normal for magma_init() to take up to 2 seconds to init? And if no, what could cause this?

Thank you in advance!

mgates3
Posts: 842
Joined: Fri Jan 06, 2012 2:13 pm

Re: magma_init() takes up to 2 seconds to init

Post by mgates3 » Thu Aug 24, 2017 9:35 am

No, I wouldn't expect it to take that long. It queries the CUDA devices for their architecture (Fermi, Kepler, ...) and possibly makes a queue (stream) on each GPU. Assuming that you include magma_v2.h, instead of magma.h, you can try adding -DMAGMA_NO_V1 to CFLAGS in make.inc and recompiling MAGMA. That would disable creating any streams in magma_init.

What GPUs do you have? E.g., what is the output of a MAGMA tester (as below)?

Code: Select all

bunsen magma/testing> ./testing_dgetrf -n 100
% MAGMA 2.1.0 svn compiled for CUDA capability >= 3.5, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 16. MKL 11.3.0, MKL threads 16. 
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% device 1: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 09:31:53 2017
% Usage: ./testing_dgetrf [options] [-h|--help]

% ngpu 1, version 1
%   M     N   CPU Gflop/s (sec)   GPU Gflop/s (sec)   |PA-LU|/(N*|A|)
%========================================================================
  100   100     ---   (  ---  )      0.05 (   0.01)     ---   
-mark

thanasis_giannis
Posts: 9
Joined: Thu Aug 24, 2017 7:35 am

Re: magma_init() takes up to 2 seconds to init

Post by thanasis_giannis » Thu Aug 24, 2017 5:56 pm

Code: Select all

thanasis@cuda:/usr/local/magma-2.2.0/testing$ ./testing_dgetrf -n 100
% MAGMA 2.2.0  compiled for CUDA capability >= 3.0, 32-bit magma_int_t, 64-bit pointer.
% CUDA runtime 7050, driver 7050. OpenMP threads 1. MKL 11.3.0, MKL threads 1. 
% device 0: Tesla K40c, 745.0 MHz clock, 11519.6 MiB memory, capability 3.5
% Thu Aug 24 19:05:22 2017
% Usage: ./testing_dgetrf [options] [-h|--help]

% ngpu 1, version 1
%   M     N   CPU Gflop/s (sec)   GPU Gflop/s (sec)   |PA-LU|/(N*|A|)
%========================================================================
  100   100     ---   (  ---  )      2.36 (   0.00)     ---   
thanasis@cuda:/usr/local/magma-2.2.0/testing$ 

The output is this. I did put the -DMAGMA_NO_V1 yet the time is still the same.

mgates3
Posts: 842
Joined: Fri Jan 06, 2012 2:13 pm

Re: magma_init() takes up to 2 seconds to init

Post by mgates3 » Fri Aug 25, 2017 3:34 pm

On my laptop, magma_init is reasonably quick, about 0.1 sec. On some of our servers, I found mixed results.

Code: Select all

k20:    1.35 sec
p100:   0.88 sec
k40:    6.21 sec
c2050:  1.15 sec
I was surprised these were that large. On any particular machine, the results varied a bit from run to run, perhaps 20%. Looking deeper, nearly all the time in magma_init is taken by cudaGetDeviceCount. If I put a different CUDA call, like cudaGetDeviceProperties, prior to cudaGetDeviceCount, then cudaGetDeviceProperties takes all the time and cudaGetDeviceCount takes negligible time. I surmise it is overhead in loading and initializing CUDA that happens on the first CUDA call. You can probably use CUDA's profiler to see this overhead.

Perhaps ask NVIDIA why the first call takes so long.

-mark
Last edited by mgates3 on Fri Aug 25, 2017 3:35 pm, edited 1 time in total.
Reason: typo

thanasis_giannis
Posts: 9
Joined: Thu Aug 24, 2017 7:35 am

Re: magma_init() takes up to 2 seconds to init

Post by thanasis_giannis » Sun Aug 27, 2017 9:55 am


Post Reply