64-bit integer in MAGMA

[xinwu]

Hi, MAGMA developers!

MAGMA uses 32-bit integer by default (#typedef int magma_int_t; in include/magmablas.h). Do you have a plan to to support 64-bit integer in MAGMA, because it may be necessary for large scale scientific computations.

BTW, a simple "#typedef int64_t magma_int_t" will not work, because some function prototype uses magma_int_t, but the actual parameters are passed in as int.

Thanks!

[mgates3]

Yes, we cleaned up the code and it is compiling with 64-bit integers now. That will be in the next release, which will come out shortly.

Note that cublas still uses 32-bit integers, so there are still some limitations, but you can link with 64-bit integer (ilp64) BLAS and LAPACK on the host CPU.

-mark

[xinwu]

Thanks for the work! Excellent!

The 64-bit integer on GPU might not need to be considered, as the global memory available is merely 6GB at present. But it will be necessary in the future.

[evanlezar]

Just a related note. There is a bigger issue than 64bit integers. I am trying to solve a pretty large problem (87612x87612) using the zgetrf on multiple GPUs. At the moment, MAGMA crashes with a segmentation violation when I = 24576 in zgetrf3_ooc. This is about the point where I*lda (lda=87616) overflows the bounds of int and this results in the A(i,j) macro not computing correctly.

I am currently testing a fix, but I would recommend that offsets and such be computed in size_t instead of int (magma_int_t). As far as I am aware, this is also the type used by the CUDA functions.

[Stan Tomov]

The 64-bit integers are now available in the MAGMA 1.2.1. Did you try using using 64 bit integer (you also have to link with the 64-bit CPU BLAS & LAPACK).
Stan

[evanlezar]

Hi Stan, for 64bit integers, do you mean that I change the typedef of magma_int_t to a 64bit equivalent? I have not tried this yet.

[Stan Tomov]

Hi Evan,
Yes, change the definition of magma_int_t to be a 64 integer (e.g. int64_t) and link with CPU BLAS and LAPACK with 64-bit integers. You can see the example in make.inc.int64 - basically, if you are using MKL you have to substitute mkl_intel_ilp64 in place of mkl_intel_lp64.
Stan

[evanlezar]

After changing the typedef, I also get a number of warnings when calling CUBLAS functions. Obviously one is then passing 64bit integers to the int parameters in CUBLAS (such as ZGEMM).

Although it is theoretically possible for me to link with the 8-byte MKL interface, we also support a number of other MATH libraries, and would prefer to stick to the 32bit interfaces -- at least for the time being.

I suppose that in this case my only course of action is to go through the code and check the indexing operations to minimise the chance of overflow. Would it by possible to at least obtain the generation scripts for the other precisions, so I only have to perform this exercise in complex double precision?

[mgates3]

Hi Evan,

Compiling MAGMA using magma_int_t as long and linking with MKL ilp64, I was able to factor matrices up to 100,000 using a single GPU. Though I did have problems at n=60,000, where CUDA was giving a kernel launch failure. I'm not sure what the problem is there. I had to modify the testing program to have a single copy of A, rather than saving a second copy, hence no residual check.

Now, I know you are trying to use magma_int_t as int, and just fix wherever it computes the offsets. Also check wherever it allocates memory. It seems that should work, but we haven't done any tests like that. Problems should exhibit themselves at about n=50,000, where n^2 overflows a signed 32-bit int. If you can send your modified code, I can test it out and see if there are any problems I see with it.

These results are on remus, a 48 core AMD Opteron 6180 at 2.5 GHz (4 sockets x 12 cores) with 256 GB memory. It has two GPUs installed, but I was using just one. Except for the modified testing program and magma_int_t, this is with the stock MAGMA 1.2.1 distribution.

setenv MKL_NUM_THREADS 12
numactl --interleave=all --physcpubind=0-11 ./testing_zgetrf -N 1000 [...other sizes...] -N 100000
device 0: Tesla T20 Processor, 1147.0 MHz clock, 2687.4 MB memory, capability 2.0
device 1: Tesla T20 Processor, 1147.0 MHz clock, 2687.4 MB memory, capability 2.0

M N CPU GFlop/s (sec) GPU GFlop/s (sec) ||PA-LU||/(||A||*N)
========================================================
1000 1000 --- ( --- ) 52.92 ( 0.05) ---
5000 5000 --- ( --- ) 197.25 ( 1.69) ---
10000 10000 --- ( --- ) 250.48 ( 10.65) ---
15000 15000 --- ( --- ) 258.73 ( 34.79) ---
20000 20000 --- ( --- ) 262.55 ( 81.25) ---
40000 40000 --- ( --- ) 264.29 ( 645.76) ---
70000 70000 --- ( --- ) 255.69 (3577.23) ---
80000 80000 --- ( --- ) 254.50 (5364.72) ---
80000 80000 --- ( --- ) 251.66 (5425.38) ---
90000 90000 --- ( --- ) 250.28 (7767.36) ---
100000 100000 --- ( --- ) 238.58 (11177.16) ---

-mark

[evanlezar]

Thanks Mark,

As you say, problems should start occurring at around 50,000x50,000 matrix elements. With my modified code -- which I have sent to you by email -- I no longer get these crashes for a single precision run that exceeds this.

Unfortunately, although the primary indexing problems seem to be solved, I still get a crash for a large system (99,399x99,399 elements) in complex double precision. It may be that this new crash is problem-size specific (in the same way that your run fails for 60,000).

Thanks again for all the assistance and have a great weekend.