Not necessarily. Routines that take the matrix in the GPU memory, e.g., magma_dgetrf_gpu, are of course limited to GPU memory. Some routines that take the matrix in the CPU memory will automatically run in "out-of-GPU-memory" mode if needed, so the matrix can exceed the GPU memory. In particular, these should run out-of-GPU-memory:
magma_xgetrf and magma_xgesv
magma_xpotrf and magma_xgesv
where x is precision: s (single, float), d (double), c (single-complex), z (double-complex).
There are also multi-GPU routines that utilize the combined memory of multiple GPUs.