How to partial offload model to GPU correctly?

[Raboo]

I'm trying offload some layers to a NVIDIA GeForce RTX 2080 Ti.

Args:
--usecuda --contextsize 100000 --usemmap --model llm/MiniMax-M2.1.i1-Q4_K_M.gguf
(I actually tried different quants of MiniMax-M2.1

If I use --autofit it tries to load 17 layers.. But I can't even offload 1 layer (--gpulayers 1), it has 11 GB RAM. I get these logs..

...
Loading Text Model: /workspace/llm/MiniMax-M2.1.i1-Q4_K_M.gguf
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 2080 Ti) (0000:65:00.0) - 10666 MiB free
llama_model_loader: loaded meta data with 48 key-value pairs and 809 tensors from /workspace/llm/MiniMax-M2.1.i1-Q4_K_M.gguf (version GGUF V3 (latest))
print_info: file format = GGUF V3 (latest)
print_info: file size   = 128.83 GiB (4.84 BPW)
init_tokenizer: initializing tokenizer for type 2
load: 0 unused tokens
load: control token: 200048 '<filepath>' is not marked as EOG
load: control token: 200047 '<review_comment>' is not marked as EOG
load: control token: 200046 '<pr_start>' is not marked as EOG
load: control token: 200045 '<source_files>' is not marked as EOG
load: control token: 200043 '<code_context>' is not marked as EOG
load: control token: 200042 '<repo_struct>' is not marked as EOG
load: control token: 200039 '<edit_file>' is not marked as EOG
load: control token: 200038 '<rename_file>' is not marked as EOG
load: control token: 200036 '<add_file>' is not marked as EOG
load: control token: 200035 '<jupyter_error>' is not marked as EOG
load: control token: 200034 ']~!b[' is not marked as EOG
load: control token: 200033 ']<]vision pad[>[' is not marked as EOG
load: control token: 200031 ']<]start of video[>[' is not marked as EOG
load: control token: 200029 ']<]start of image[>[' is not marked as EOG
load: control token: 200026 ']<]video[>[' is not marked as EOG
load: control token: 200025 ']<]image[>[' is not marked as EOG
load: control token: 200020 '[e~[' is not marked as EOG
load: control token: 200018 '<commit_after>' is not marked as EOG
load: control token: 200017 '<commit_msg>' is not marked as EOG
load: control token: 200014 '<jupyter_output>' is not marked as EOG
load: control token: 200013 '<jupyter_code>' is not marked as EOG
load: control token: 200011 '<jupyter_start>' is not marked as EOG
load: control token: 200006 '<filename>' is not marked as EOG
load: control token: 200003 '<fim_suffix>' is not marked as EOG
load: control token: 200002 '<fim_middle>' is not marked as EOG
load: control token: 200001 '<fim_prefix>' is not marked as EOG
load: control token: 200000 ']!p~[' is not marked as EOG
load: control token: 200037 '<delete_file>' is not marked as EOG
load: control token: 200049 '<file_sep>' is not marked as EOG
load: control token: 200021 ']!d~[' is not marked as EOG
load: control token: 200010 '<issue_closed>' is not marked as EOG
load: control token: 200024 ']<]speech[>[' is not marked as EOG
load: control token: 200030 ']<]end of image[>[' is not marked as EOG
load: control token: 200028 ']<]end of speech[>[' is not marked as EOG
load: control token: 200032 ']<]end of video[>[' is not marked as EOG
load: control token: 200040 '<commit_message>' is not marked as EOG
load: control token: 200019 ']~b]' is not marked as EOG
load: control token: 200007 '<gh_stars>' is not marked as EOG
load: control token: 200015 '<empty_output>' is not marked as EOG
load: control token: 200008 '<issue_start>' is not marked as EOG
load: control token: 200041 '<empty_source_file>' is not marked as EOG
load: control token: 200023 '<code_interpreter>' is not marked as EOG
load: control token: 200022 '<function_call>' is not marked as EOG
load: control token: 200027 ']<]start of speech[>[' is not marked as EOG
load: control token: 200044 '<file_content>' is not marked as EOG
load: control token: 200012 '<jupyter_text>' is not marked as EOG
load: control token: 200016 '<commit_before>' is not marked as EOG
load: control token: 200009 '<issue_comment>' is not marked as EOG
load: special_eos_id is not in special_eog_ids - the tokenizer config may be incorrect
load: printing all EOG tokens:
load:   - 200004 ('<fim_pad>')
load:   - 200005 ('<reponame>')
load:   - 200020 ('[e~[')
load: special tokens cache size = 54
load: token to piece cache size = 1.3355 MB
print_info: arch                  = minimax-m2
print_info: vocab_only            = 0
print_info: no_alloc              = 0
print_info: n_ctx_train           = 196608
print_info: n_embd                = 3072
print_info: n_embd_inp            = 3072
print_info: n_layer               = 62
print_info: n_head                = 48
print_info: n_head_kv             = 8
print_info: n_rot                 = 64
print_info: n_swa                 = 0
print_info: is_swa_any            = 0
print_info: n_embd_head_k         = 128
print_info: n_embd_head_v         = 128
print_info: n_gqa                 = 6
print_info: n_embd_k_gqa          = 1024
print_info: n_embd_v_gqa          = 1024
print_info: f_norm_eps            = 0.0e+00
print_info: f_norm_rms_eps        = 1.0e-06
print_info: f_clamp_kqv           = 0.0e+00
print_info: f_max_alibi_bias      = 0.0e+00
print_info: f_logit_scale         = 0.0e+00
print_info: f_attn_scale          = 0.0e+00
print_info: n_ff                  = 1536
print_info: n_expert              = 256
print_info: n_expert_used         = 8
print_info: n_expert_groups       = 0
print_info: n_group_used          = 0
print_info: causal attn           = 1
print_info: pooling type          = 0
print_info: rope type             = 2
print_info: rope scaling          = linear
print_info: freq_base_train       = 5000000.0
print_info: freq_scale_train      = 1
print_info: n_ctx_orig_yarn       = 196608
print_info: rope_yarn_log_mul     = 0.0000
print_info: rope_finetuned        = unknown
print_info: model type            = 230B.A10B
print_info: model params          = 228.69 B
print_info: general.name          = MiniMax M2.1
print_info: vocab type            = BPE
print_info: n_vocab               = 200064
print_info: n_merges              = 199744
print_info: BOS token             = 200034 ']~!b['
print_info: EOS token             = 200020 '[e~['
print_info: UNK token             = 200021 ']!d~['
print_info: LF token              = 10 'Ċ'
print_info: FIM PRE token         = 200001 '<fim_prefix>'
print_info: FIM SUF token         = 200003 '<fim_suffix>'
print_info: FIM MID token         = 200002 '<fim_middle>'
print_info: FIM PAD token         = 200004 '<fim_pad>'
print_info: FIM REP token         = 200005 '<reponame>'
print_info: EOG token             = 200004 '<fim_pad>'
print_info: EOG token             = 200005 '<reponame>'
print_info: EOG token             = 200020 '[e~['
print_info: max token length      = 256
load_tensors: loading model tensors, this can take a while... (mmap = true, direct_io = false)
load_tensors: relocated tensors: 807 of 809
load_tensors: offloading output layer to GPU
load_tensors: offloading 0 repeating layers to GPU
load_tensors: offloaded 1/63 layers to GPU
load_tensors:   CPU_Mapped model buffer size = 131444.86 MiB
load_tensors:        CUDA0 model buffer size =   480.82 MiB
....................................................................................................
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 100352
llama_context: n_ctx_seq     = 100352
llama_context: n_batch       = 512
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = disabled
llama_context: kv_unified    = true
llama_context: freq_base     = 5000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_seq (100352) < n_ctx_train (196608) -- the full capacity of the model will not be utilized
set_abort_callback: call
llama_context:  CUDA_Host  output buffer size =     0.76 MiB
llama_kv_cache:        CPU KV buffer size = 24304.00 MiB
llama_kv_cache: size = 24304.00 MiB (100352 cells,  62 layers,  1/1 seqs), K (f16): 12152.00 MiB, V (f16): 12152.00 MiB
llama_context: enumerating backends
llama_context: backend_ptrs.size() = 2
sched_reserve: reserving ...
sched_reserve: max_nodes = 6472
sched_reserve: reserving full memory module
sched_reserve: worst-case: n_tokens = 512, n_seqs = 1, n_outputs = 1
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 10238.12 MiB on device 0: cudaMalloc failed: out of memory
ggml_gallocr_reserve_n_impl: failed to allocate CUDA0 buffer of size 10735452160
graph_reserve: failed to allocate compute buffers
llama_init_from_model: failed to initialize the context: failed to allocate compute pp buffers
gpttype_load_model: error: failed to load model '/workspace/llm/MiniMax-M2.1.i1-Q4_K_M.gguf'
Load Text Model OK: False

Could not load text model: /workspace/llm/MiniMax-M2.1.i1-Q4_K_M.gguf
Something possibly went wrong, stalling for 3 minutes before exiting so you can check for errors. (No error? You may have run out of memory. Try deleting the image generation model if you don't need it or use a larger GPU.)

Let me know if you need a complete log.

Why can't I use partial GPU offloading?

[LostRuins]

Please try lower the context size first, also enable --flashattention

[Raboo]

But supposedly MiniMax-M2.1 is a full attention model.
https://www.reddit.com/r/LocalLLaMA/comments/1ou8b89/why_is_minimax_m2_a_full_attention_model/

Or is this something else and I should still enable flash attention?

[LostRuins]

Yes, please enable flash attention and try

[Raboo]

Seems to have worked. GPU is being utilized.