Audio version — Estimated duration: 6 min 55 sec

Implementation Guide

This document details the practical steps, scripts, and code used to build, train, and deploy the Gemma 3-1B fine-tuned model.

1. Data Preparation: Subset Creation

Efficiently process large datasets by creating manageable subsets for rapid prototyping and debugging.

Script: create_subset.py

• Reads the massive Alpaca 120k dataset.
• Randomly samples a specified number of items (default: 1,000).
• Outputs a new, valid JSON file for use in training.

Example Code

import json
import random
 
def create_subset(input_file, output_file, subset_size=1000):
    with open(input_file, 'r', encoding='utf-8') as f:
        full_dataset = json.load(f)
    subset = random.sample(full_dataset, min(len(full_dataset), subset_size))
    with open(output_file, 'w', encoding='utf-8') as f:
        json.dump(subset, f, ensure_ascii=False, indent=2)
    print(f"Subset of {len(subset)} items -> {output_file}")
 
if __name__ == "__main__":
    create_subset("dataset/code_instructions_120k_alpaca.json",
                  "dataset/code_instructions_1k_alpaca_subset.json", 1000)

2. Fine-Tuning Engine

The core training process, leveraging Unsloth and LoRA for efficient parameter tuning.

Script: main.py

• Loads the base model in 4-bit quantization.
• Attaches LoRA adapters for parameter-efficient training.
• Uses SFTTrainer for supervised fine-tuning.
• Generates training reports and visualizations.

Example Code

from unsloth import FastLanguageModel
from datasets import Dataset
from trl import SFTTrainer
from transformers import TrainingArguments
 
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/gemma-3-1b-it",
    max_seq_length=2048, load_in_4bit=True)
 
model = FastLanguageModel.get_peft_model(model, r=64,
    target_modules=["q_proj","k_proj","v_proj","o_proj",
                    "gate_proj","up_proj","down_proj"],
    lora_alpha=128, lora_dropout=0, bias="none")
 
trainer = SFTTrainer(model=model, tokenizer=tokenizer,
    train_dataset=dataset, dataset_text_field="text",
    max_seq_length=2048,
    args=TrainingArguments(per_device_train_batch_size=2,
        gradient_accumulation_steps=4, num_train_epochs=1,
        learning_rate=2e-4, output_dir="outputs",
        report_to="tensorboard"))
trainer.train()

3. Model Merging

After training, merge LoRA adapters into the base model to create a standalone, deployable model.

Script: merge_lora.py

• Loads the checkpoint with LoRA adapters.
• Merges adapters into the base model (16-bit precision).
• Saves the unified model to the merged_model directory.

Example Code

from unsloth import FastLanguageModel
 
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="outputs/checkpoint-45735",
    max_seq_length=8192, load_in_4bit=True)
 
model.save_pretrained_merged("merged_model", tokenizer,
                             save_method="merged_16bit")
print(f"Merged model saved to merged_model/")

4. Model Export: GGUF Conversion

Convert the merged model to GGUF format for compatibility with local inference tools.

Script: export_gguf.py

• Loads the merged model.
• Converts and quantizes to GGUF (q8_0).
• Outputs .gguf file for use with llama.cpp, LM Studio, etc.

Example Code

from unsloth import FastLanguageModel
 
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="outputs/checkpoint-625",
    max_seq_length=8192, load_in_4bit=True)
 
model.save_pretrained_merged("merged_model", tokenizer,
                             save_method="merged_16bit")
model.save_pretrained_gguf("merged_model", tokenizer,
                           quantization_method="q8_0")

5. Inference Interface

Interactive CLI chat interface for model evaluation and demonstration.

Script: inference.py

• Loads the final model from merged_model.
• Formats prompts in Alpaca style.
• Maintains conversation history for multi-turn interactions.

Example Code

import torch
from unsloth import FastLanguageModel
 
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="merged_model", max_seq_length=8192, load_in_4bit=True)
model.eval()
 
while True:
    instruction = input("You: ")
    if instruction.lower() in ('exit', 'quit'):
        break
    prompt = f"### Instruction:\n{instruction}\n\n### Response:\n"
    inputs = tokenizer([prompt], return_tensors="pt").to("cuda")
    outputs = model.generate(**inputs, max_new_tokens=256)
    response = tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:],
                                skip_special_tokens=True)
    print(f"Model: {response.strip()}")

🔗 Related Documents

• SPEC
• QUICK_REFERENCE
• index

---

Last Updated: 2026-04-29
Version: 1.0
Status: Complete