TensorRT_LLM Recipes-2 (PEFT, LoRA techniques)
In this article we will look at fine-tuning of a LLM, techniques like LoRA, PEFT with LoRA(Hugging face Transformers).
TensorRT_LLM Recipes-2 (PEFT, LoRA techniques)
In this article we will look at fine-tuning of a LLM, techniques like LoRA, PEFT with LoRA(Hugging face Transformers).
If you need Recepies for Quantization(Pytorch & tensorrt backend) and KV cache refer to my article here….
First we will look at LoRA technique on a LLM. For this we will need tensorRT_LLM, Hugging Face login for model access. If you need steps to install a tensorRT_LLM you can refer to my earlier article on medium the link to which is here…. (pip install: https://rangapv.medium.com/74f7bd24c100?source=friends_link&sk=65bb6134e3eed1638e4d5fc53389f888
Source install: https://rangapv.medium.com/8fc755e44a30?source=friends_link&sk=844438763757fcab4c438371945e0461).
you could instead download my Built wheel from here…( download the Built Wheel from here…) ; just make sure you have the pre-requisties installed as mentioned from either of the above two articles.
If you are new and do not have a Nvidia ecosystem setup then you can refer to my article (https://rangapv.medium.com/tensorrt-install-for-deep-neural-network-dnn-inference-0f0ffb8a4258 ).
Now that we have the architecture in place lets look at LoRA technique.
- LLM with LoRA:
We will employ the meta-llama as the base model and do a translation for a Japanese text using LoRA technique.
git-lfs clone [https://huggingface.co/meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-13b-hf)
git-lfs clone [https://huggingface.co/kunishou/Japanese-Alpaca-LoRA-7b-v0](https://huggingface.co/kunishou/Japanese-Alpaca-LoRA-7b-v0)
Lets convert the check point…
$ python3 ./TensorRT-LLM/examples/models/core/llama/convert_checkpoint.py
--model_dir ./Llama-2-7b-hf
--output_dir ./tllm_checkpoint_1gpu_lora_rank
--dtype float16
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
[2026-01-14 12:15:39] INFO config.py:54: PyTorch version 2.7.1 available.
[2026-01-14 12:15:39] INFO config.py:112: TensorFlow version 2.20.0 available.
2026-01-14 12:15:40.337710: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import huggingface plugin due to: ValueError('Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.'). You may ignore this warning if you do not need this plugin.
warnings.warn(
/home/ubuntu/.local/lib/python3.10/site-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import transformers trainer plugin due to: ValueError('Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.'). You may ignore this warning if you do not need this plugin.
warnings.warn(
2026-01-14 12:15:43,113 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT LLM version: 1.0.0
1.0.0
[01/14/2026-12:15:43] [TRT-LLM] [W] Implicitly setting LLaMAConfig.has_partial_lora_mask = False
[01/14/2026-12:15:43] [TRT-LLM] [W] Implicitly setting LLaMAConfig.tie_word_embeddings = False
230it [00:26, 8.66it/s]
Total time of reading and converting: 26.609 s
Total time of saving checkpoint: 209.280 s
Total time of converting checkpoints: 00:03:56
$Now lets Build the llm engine…
$ trtllm-build --checkpoint_dir ./tllm_checkpoint_1gpu_lora_rank
--output_dir /tmp/llama_7b_with_lora_qkv/trt_engines/fp16/1-gpu/
--gpt_attention_plugin float16
--gemm_plugin float16
--lora_plugin float16
--max_batch_size 1
--max_input_len 512
--lora_target_modules "attn_q" "attn_k" "attn_v"
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
[2026-01-14 12:26:13] INFO config.py:54: PyTorch version 2.7.1 available.
[2026-01-14 12:26:13] INFO config.py:112: TensorFlow version 2.20.0 available.
2026-01-14 12:26:14.745322: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import huggingface plugin due to: ValueError('Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.'). You may ignore this warning if you do not need this plugin.
warnings.warn(
/home/ubuntu/.local/lib/python3.10/site-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import transformers trainer plugin due to: ValueError('Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.'). You may ignore this warning if you do not need this plugin.
warnings.warn(
2026-01-14 12:26:17,733 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT LLM version: 1.0.0
[01/14/2026-12:26:17] [TRT-LLM] [I] Set bert_attention_plugin to auto.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set gpt_attention_plugin to float16.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set gemm_plugin to float16.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set gemm_swiglu_plugin to None.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set fp8_rowwise_gemm_plugin to None.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set nccl_plugin to auto.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set lora_plugin to float16.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set dora_plugin to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set moe_plugin to auto.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set mamba_conv1d_plugin to auto.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set low_latency_gemm_plugin to None.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set low_latency_gemm_swiglu_plugin to None.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set gemm_allreduce_plugin to None.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set context_fmha to True.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set bert_context_fmha_fp32_acc to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set remove_input_padding to True.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set norm_quant_fusion to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set reduce_fusion to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set user_buffer to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set tokens_per_block to 32.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set use_paged_context_fmha to True.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set use_fp8_context_fmha to True.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set fuse_fp4_quant to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set multiple_profiles to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set paged_state to True.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set streamingllm to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set use_fused_mlp to True.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set pp_reduce_scatter to False.
[01/14/2026-12:26:17] [TRT-LLM] [W] Implicitly setting LLaMAConfig.fc_after_embed = False
[01/14/2026-12:26:17] [TRT-LLM] [W] Implicitly setting LLaMAConfig.use_input_layernorm_in_first_layer = True
[01/14/2026-12:26:17] [TRT-LLM] [W] Implicitly setting LLaMAConfig.use_last_layernorm = True
[01/14/2026-12:26:17] [TRT-LLM] [W] Implicitly setting LLaMAConfig.layer_idx_offset = 0
[01/14/2026-12:26:17] [TRT-LLM] [W] Implicitly setting LLaMAConfig.has_partial_lora_mask = False
[01/14/2026-12:26:17] [TRT-LLM] [W] Implicitly setting LLaMAConfig.tie_word_embeddings = False
[01/14/2026-12:26:17] [TRT-LLM] [W] Provided but not required tensors: {'embed_positions_for_gpt_attention', 'rotary_inv_freq', 'embed_positions'}
[01/14/2026-12:26:17] [TRT-LLM] [I] Set dtype to float16.
[01/14/2026-12:26:17] [TRT-LLM] [I] Set paged_kv_cache to True.
[01/14/2026-12:26:17] [TRT-LLM] [W] Overriding paged_state to False
[01/14/2026-12:26:17] [TRT-LLM] [I] Set paged_state to False.
[01/14/2026-12:26:17] [TRT-LLM] [I] max_seq_len is not specified, using deduced value 4096
[01/14/2026-12:26:17] [TRT-LLM] [W] remove_input_padding is enabled, while opt_num_tokens is not set, setting to max_batch_size*max_beam_width.
[01/14/2026-12:26:17] [TRT-LLM] [W] max_num_tokens (8192) shouldn't be greater than max_seq_len * max_batch_size (4096), specifying to max_seq_len * max_batch_size (4096).
[01/14/2026-12:26:17] [TRT-LLM] [W] padding removal and fMHA are both enabled, max_input_len is not required and will be ignored
[01/14/2026-12:26:17] [TRT-LLM] [I] Set use_fp8_context_fmha to False.
[01/14/2026-12:26:17] [TRT-LLM] [W] FP8 Context FMHA is disabled because it must be used together with the fp8 quantization workflow.
[01/14/2026-12:28:04] [TRT] [I] [MemUsageChange] Init CUDA: CPU +0, GPU +0, now: CPU 5791, GPU 190 (MiB)
[01/14/2026-12:28:20] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +1588, GPU +8, now: CPU 7580, GPU 198 (MiB)
[01/14/2026-12:28:20] [TRT-LLM] [I] Set nccl_plugin to None.
[01/14/2026-12:28:21] [TRT-LLM] [I] Total time of constructing network from module object 123.3897454738617 seconds
[01/14/2026-12:28:21] [TRT-LLM] [I] Total optimization profiles added: 1
[01/14/2026-12:28:21] [TRT-LLM] [I] Total time to initialize the weights in network Unnamed Network 0: 00:00:00
[01/14/2026-12:28:21] [TRT-LLM] [I] Build TensorRT engine Unnamed Network 0
[01/14/2026-12:28:21] [TRT] [W] Unused Input: position_ids
[01/14/2026-12:28:23] [TRT] [W] [RemoveDeadLayers] Input Tensor position_ids is unused or used only at compile-time, but is not being removed.
[01/14/2026-12:28:23] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored.
[01/14/2026-12:28:23] [TRT] [I] Compiler backend is used during engine build.
[01/14/2026-12:28:27] [TRT] [I] [GraphReduction] The approximate region cut reduction algorithm is called.
[01/14/2026-12:28:27] [TRT] [I] Detected 209 inputs and 1 output network tensors.
[01/14/2026-12:31:54] [TRT] [I] Total Host Persistent Memory: 139792 bytes
[01/14/2026-12:31:54] [TRT] [I] Total Device Persistent Memory: 0 bytes
[01/14/2026-12:31:54] [TRT] [I] Max Scratch Memory: 67142400 bytes
[01/14/2026-12:31:54] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 691 steps to complete.
[01/14/2026-12:31:54] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 38.9757ms to assign 21 blocks to 691 nodes requiring 515906560 bytes.
[01/14/2026-12:31:54] [TRT] [I] Total Activation Memory: 515906560 bytes
[01/14/2026-12:31:54] [TRT] [I] Total Weights Memory: 13478929024 bytes
[01/14/2026-12:31:54] [TRT] [I] Compiler backend is used during engine execution.
[01/14/2026-12:31:54] [TRT] [I] Engine generation completed in 211.577 seconds.
[01/14/2026-12:31:54] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 12855 MiB
[01/14/2026-12:32:01] [TRT-LLM] [I] Total time of building Unnamed Network 0: 00:03:40
[01/14/2026-12:32:01] [TRT] [I] Serialized 27 bytes of code generator cache.
[01/14/2026-12:32:01] [TRT] [I] Serialized 148802 bytes of compilation cache.
[01/14/2026-12:32:01] [TRT] [I] Serialized 17 timing cache entries
[01/14/2026-12:32:01] [TRT-LLM] [I] Timing cache serialized to model.cache
[01/14/2026-12:32:01] [TRT-LLM] [I] Build phase peak memory: 28241.12 MB, children: 2782.29 MB
[01/14/2026-12:32:02] [TRT-LLM] [I] Serializing engine to /tmp/llama_7b_with_lora_qkv/trt_engines/fp16/1-gpu/rank0.engine...
[01/14/2026-12:33:35] [TRT-LLM] [I] Engine serialized. Total time: 00:01:33
[01/14/2026-12:33:36] [TRT-LLM] [I] Total time of building all engines: 00:07:18
$Let us input a text in Japanese “アメリカ合衆国の首都はどこですか? \n答え:”
(The translation of the input text is “What is the Capital of United States”)
$ python3 ./TensorRT-LLM/examples/run.py
--engine_dir "/tmp/llama_7b_with_lora_qkv/trt_engines/fp16/1-gpu/"
--max_output_len 10
--tokenizer_dir ./Llama-2-7b-hf
--input_text "アメリカ合衆国の首都はどこですか? \n答え:"
--lora_dir ./Japanese-Alpaca-LoRA-7b-v0
--use_py_session
--top_p 0.5
--top_k 0
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cuda module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.driver module instead.
<frozen importlib._bootstrap_external>:1184: FutureWarning: The cuda.cudart module is deprecated and will be removed in a future release, please switch to use the cuda.bindings.runtime module instead.
[2026-01-14 13:01:39] INFO config.py:54: PyTorch version 2.7.1 available.
[2026-01-14 13:01:39] INFO config.py:112: TensorFlow version 2.20.0 available.
2026-01-14 13:01:40.061107: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
/home/ubuntu/.local/lib/python3.10/site-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import huggingface plugin due to: ValueError('Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.'). You may ignore this warning if you do not need this plugin.
warnings.warn(
/home/ubuntu/.local/lib/python3.10/site-packages/modelopt/torch/utils/import_utils.py:32: UserWarning: Failed to import transformers trainer plugin due to: ValueError('Your currently installed version of Keras is Keras 3, but this is not yet supported in Transformers. Please install the backwards-compatible tf-keras package with `pip install tf-keras`.'). You may ignore this warning if you do not need this plugin.
warnings.warn(
2026-01-14 13:01:42,897 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT LLM version: 1.0.0
[01/14/2026-13:01:43] [TRT-LLM] [I] Using Python session
[01/14/2026-13:01:50] [TRT-LLM] [W] Implicitly setting LLaMAConfig.fc_after_embed = False
[01/14/2026-13:01:50] [TRT-LLM] [W] Implicitly setting LLaMAConfig.use_input_layernorm_in_first_layer = True
[01/14/2026-13:01:50] [TRT-LLM] [W] Implicitly setting LLaMAConfig.use_last_layernorm = True
[01/14/2026-13:01:50] [TRT-LLM] [W] Implicitly setting LLaMAConfig.layer_idx_offset = 0
[01/14/2026-13:01:50] [TRT-LLM] [W] Implicitly setting LLaMAConfig.has_partial_lora_mask = False
[01/14/2026-13:01:50] [TRT-LLM] [W] Implicitly setting LLaMAConfig.tie_word_embeddings = False
[01/14/2026-13:01:50] [TRT-LLM] [I] Set dtype to float16.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set bert_attention_plugin to auto.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set gpt_attention_plugin to float16.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set gemm_plugin to float16.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set explicitly_disable_gemm_plugin to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set gemm_swiglu_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set fp8_rowwise_gemm_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set qserve_gemm_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set identity_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set nccl_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set lora_plugin to float16.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set dora_plugin to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set weight_only_groupwise_quant_matmul_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set weight_only_quant_matmul_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set smooth_quant_plugins to True.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set smooth_quant_gemm_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set layernorm_quantization_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set rmsnorm_quantization_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set quantize_per_token_plugin to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set quantize_tensor_plugin to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set moe_plugin to auto.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set mamba_conv1d_plugin to auto.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set low_latency_gemm_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set low_latency_gemm_swiglu_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set gemm_allreduce_plugin to None.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set context_fmha to True.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set bert_context_fmha_fp32_acc to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set paged_kv_cache to True.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set remove_input_padding to True.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set norm_quant_fusion to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set reduce_fusion to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set user_buffer to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set tokens_per_block to 32.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set use_paged_context_fmha to True.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set use_fp8_context_fmha to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set fuse_fp4_quant to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set multiple_profiles to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set paged_state to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set streamingllm to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set manage_weights to False.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set use_fused_mlp to True.
[01/14/2026-13:01:50] [TRT-LLM] [I] Set pp_reduce_scatter to False.
[01/14/2026-13:01:50] [TRT] [I] Loaded engine size: 12859 MiB
[01/14/2026-13:01:51] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 12854 (MiB)
[01/14/2026-13:01:51] [TRT] [I] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 12854 (MiB)
[01/14/2026-13:01:51] [TRT-LLM] [W] The paged KV cache in Python runtime is experimental. For performance and correctness, please, use C++ runtime.
[01/14/2026-13:01:52] [TRT-LLM] [I] Load engine takes: 9.36447811126709 sec
/home/ubuntu/.local/lib/python3.10/site-packages/torch/nested/__init__.py:250: UserWarning: The PyTorch API of nested tensors is in prototype stage and will change in the near future. We recommend specifying layout=torch.jagged when constructing a nested tensor, as this layout receives active development, has better operator coverage, and works with torch.compile. (Triggered internally at /pytorch/aten/src/ATen/NestedTensorImpl.cpp:178.)
return _nested.nested_tensor(
Input [Text 0]: "<s> アメリカ合衆国の首都はどこですか? \n答え:"
Output [Text 0 Beam 0]: "ワシントンD.C.
"
$The translation of the output (ワシントンD.C.) is “Washington DC” in Japanese.
2. PEFT with LoRA Technique (Hugging face Transformers):
The requirements for this code base is..
tf_keras
transformers==4.50.3
accelerate==1.12.0
torchvision==0.24.1
peft==0.18.1We will use GPT2 as base model to fine tune IMDb dataset!
Here we will use the PEFT with LoRA technique for smart fine tuning
#!/usr/bin/env python3
#author:rangapv@yahoo.com
#04-01-26
from datasets import load_dataset
from transformers import Trainer, TrainingArguments
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import get_peft_model, LoraConfig, TaskType
from peft import prepare_model_for_kbit_training
from peft import PeftModelForSequenceClassification, get_peft_config
from peft import PeftModel, PeftConfig
base_model = "gpt2"
#base_model="meta-llama/Llama-2-7b-hf"
tokenizer = AutoTokenizer.from_pretrained(base_model)
#tokenizer = AutoTokenizer.from_pretrained(base_model,num_labels=128)
model = AutoModelForCausalLM.from_pretrained(base_model)
tokenizer.pad_token = tokenizer.eos_token
model.resize_token_embeddings(len(tokenizer))
lora_config = LoraConfig(
r=8, # Low-rank dimension
lora_alpha=32,
target_modules=["c_attn"], # Target GPT2's attention layers
lora_dropout=0.1,
bias="none",
task_type=TaskType.CAUSAL_LM # Causal Language Modeling task
)
model = prepare_model_for_kbit_training(model)
model = get_peft_model(model, lora_config)
model.print_trainable_parameters()
# Load small dataset
dataset = load_dataset("imdb", split="train[:1%]")
#dataset = dataset1.batch(batch_size=8)
print("dataset 1")
print(dataset[1])
print(dataset[2])
print("dataset 1")
# Preprocess the data
def tokenize(example):
encoding = tokenizer(str([example["text"],]), padding="max_length", truncation=True, max_length=250)
return encoding
tokenized_dataset = dataset.map(tokenize, batched=True)
#tokenized_dataset.set_format(type="torch", columns=["input_ids", "label"])
tokenized_dataset.set_format(type="torch")
#tokenized_dataset.set_format(type="torch", columns=["text","label","input_ids", "attention_mask"])
#tokenized_dataset.set_format(type="torch", columns=["label", "input_ids", "attention_mask"])
training_args = TrainingArguments(
output_dir="./lora_imdb",
per_device_train_batch_size=8,
per_device_eval_batch_size=8,
num_train_epochs=1,
logging_steps=10,
save_steps=100,
save_total_limit=2,
fp16=True,
label_names=["label"],
# remove_unused_columns=False,
report_to="none"
)
trainer = Trainer(
model=model,
args=training_args,
train_dataset=tokenized_dataset
)
trainer.train()
# Save the LoRA adapter (not full model)
model.save_pretrained("./lora_adapter_only")
tokenizer.save_pretrained("./lora_adapter_only")
base_model1 = AutoModelForCausalLM.from_pretrained(base_model)
tokenizer = AutoTokenizer.from_pretrained(base_model)
peft_model = PeftModel.from_pretrained(base_model1, "./lora_adapter_only")
peft_model.eval()
# Inference
prompt = "No Country for Old Men"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = peft_model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))Bingo!
If you have any issues, you can open an issue in the github repo page or alternatively contact me at rangapv@yahoo.com and you can also find me on X.com(twitter) @rangapv . My linkedin ***https://linkedin.com/in/rangapv***
Some of my other MLOps/Devops articles on medium…
메타데이터
- post_id
- 2282f34344d6
- slug
- tensorrt-llm-recipes-2-peft-lora-techniques-2282f34344d6
- url
- https://medium.com/@rangapv/tensorrt-llm-recipes-2-peft-lora-techniques-2282f34344d6
- canonical_url
- https://medium.com/@rangapv/tensorrt-llm-recipes-2-peft-lora-techniques-2282f34344d6
- author_url
- https://medium.com/@rangapv
- status
- ok
- fetched_at
- 2026-06-15 20:49:13
