Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment (2024)

2.1 Conversational Ability of LLM

In the pre-training phase, LLMs learn from a vast corpus of text data collected from various sources, including the internet, books, articles, and conversationsRaffel etal. (2019); Zhu etal. (2015); Gao etal. (2020); Penedo etal. (2023). Through this process, they acquire extensive knowledge on a wide range of topics, which forms the foundation that enables LLMs to flexibly respond to diverse conversational subjectsZhang etal. (2022); Touvron etal. (2023a, b); Brown etal. (2020). Subsequently, LLMs develop the capability to follow instructions through instruction fine-tuning and learn to align with human preferences via RLHFTaori etal. (2023); Longpre etal. (2023); Chung etal. (2022); Mukherjee etal. (2023); Ouyang etal. (2022). Through such processes, LLM-based chatbots like GPT-4OpenAI (2023) and VicunaChiang etal. (2023) have acquired the conversational ability to engage with humans on various topics over multiple turns, distinguishing them from conventional language models.

To evaluate LLM-based chatbots, it is essential to assess their conversational ability, which is their key capability. However, existing task-dependent benchmarks such as MMLUHendrycks etal. (2020) and HELMLiang etal. (2023) do not adequately capture human preferences, rendering them insufficient for evaluating LLM-based chatbots. In response, proposals for new benchmarks such as MT-BenchZheng etal. (2023) and FLASKYe etal. (2023) are emerging, focusing on multi-turn questions or alignment with human preferences to effectively evaluate conversational abilities.

2.2 LLM Quantization

LLMs demand high serving costs due to their extensive number of parametersBrown etal. (2020).Weight quantization techniquesLin etal. (2023); Frantar etal. (2023); Lee etal. (2023a); Kim etal. (2023); Lee etal. (2023b) address this issue by representing the model’s weights in lower bit-precision, thereby reducing memory size, lowering memory load time, and speeding up inference. Post-training quantization (PTQ) changes the model’s weights directly to lower precision without additional training, offering cost benefits. However, due to concerns about accuracy loss, PTQ utilizes a portion of the training samples to calibrate and minimize the layer-wise quantization error through methods such as AWQLin etal. (2023). Quantization-aware training (QAT), on the other hand, maintains the performance of a quantized model by applying quantization during the forward pass and training the model accordingly. When applying QAT to LLMs, due to the insufficient information from the ground truth, techniques often use Knowledge Distillation (KD) by reducing the distance between the logits of the quantized model and the full-precision modelKim etal. (2023); Liu etal. (2023b).

However, previous quantization studies have evaluated their methods on task-dependent benchmarks, which show a limited scope for comprehensive evaluation of conversational abilities. For example, AWQLin etal. (2023) emphasizes that the quantized model achieves accuracy comparable to the baseline on CSQA. However, they do not analyze why only 35% of the sentences generated by the quantized model are considered as good as those from the baseline, according to GPT-4OpenAI (2023)’s evaluation in assessing conversational abilities. In this research, we analyze how the model’s quantization error impacts the conversational abilities of large language models and propose methods to enhance these capabilities.

2.3 Alignment with Human Preferences

The RLHF is an advanced method to improve the performance of LLMs by aligning with human preferences. It comprises three stages:

Supervised Fine-Tuning (SFT). SFT utilizes a dataset of human instructions to refine pre-trained LLMs.

$r^{*}$ is defined as the optimal reward function. $y_{1}$ and $y_{2}$, assumed to be sampled from the optimal preference distribution $p^{*}$ with prompt $x$, the parameterized reward model estimates the parameter using maximum likelihood.

SROLiu etal. (2023a) criticizes the preference sampling method of DPO. Sampling data $y_{w}$ and $y_{l}$ from the $\pi^{*}$ is the optimal way for estimating $\pi_{\theta}$. However, all experiments in DPO use preference pairs not from the $\pi^{*}$ but from $\pi_{\text{ref}}$, and there is a lack of research into the implications of this approach. SRO proposes a solution by constructing an additional reward-ranking model to directly form preference pairs from an approximated optimal policy and statistical rejection sampling.

3.1 Observations

Recent advancements in PTQ have demonstrated that 4-bit quantized LLMs are effective for a variety of tasks, as evidenced by references such as AWQ and OPTQLin etal. (2023); Frantar etal. (2023). However, our observations reveal that these quantized LLMs struggle to sustain engaging conversations, particularly in multi-turn chatbot interactions. For instance, Fig.1 illustrates the contrast between the 16-bit baseline and 4-bit quantized LLMs in sentence generation. The baseline model begins its responses with “The circle of fifths is a musical diagram,” providing relevant answers. On the other hand, the 4-bit quantized model starts to deviate at the seventh token, switching its focus from “musical” to “visual,” and often generates limited and repetitive phrases. Although both models display similar task performance metrics, such as accuracy in multiple-choice benchmarks, there’s a noticeable difference in the logit probability for the seventh token in the 4-bit model, causing a change in the token from “musical” to “visual.” This issue of altered text generation, observed across multiple examples (see A.9 for additional examples), prompts an investigationinto its underlying causes.

3.2 Breakdown Analysis

Setup.To evaluate the impact of each identified factor, we analyze eight possible scenarios shown in Fig.2(b). Case 1, where all three factors are present, mirrors the standard text generation of a quantized LLM, whereas Case 8, devoid of these factors, corresponds to the baseline model’s inference. For this analysis, we generate text using both 4-bit and 16-bit models with 1,000 instruction samples randomly chosen from the Alpaca datasetTaori etal. (2023). We record the first token where discrepancies in text generation between the two models occurred, along with the key-value cache status up to that point for each scenario. To quantify the deviation from the baseline text, we utilize the ROUGE-LLin (2004) as a metric (the higher the better). More details of the implementation for the breakdown analysis are provided in A.2.

Results. To assess the contribution of each factor to deviations in sentence generation from the baseline, we contrast each scenario with Case 8. As depicted in Fig.2(b), sentences become more divergent with the inclusion of additional factors. Specifically, from Case 4, it is evident that the Flipped Token ($\mathrm{T_{F}}$) significantly affects sentence variation, as indicated by the largest decrease in ROUGE scores. Conversely, the effects of perturbed KV cache ($\mathrm{KV_{Perturb}}$) and quantization error in generation ($\mathrm{Q_{Error}}$) are comparatively minor. This pattern is further highlighted in Fig.2(c), where ROUGE scores, sorted by samples, show that Cases 1-4 cluster on the right, signifying greater deviation. This suggests that even a single token difference, resulting from quantization-affected probability shifts, can substantially alter the overall sentence structure in quantized inference.

Ablation: Advanced Quantization. Advanced quantization techniques designed to minimize errors may not fully address the issue of deviated text generation caused by flipped tokens. Recent PTQ methods that employ calibration using a small sample by scaling weight channels or adjusting quantization step sizesFrantar etal. (2023); Lin etal. (2023); Lee etal. (2023b) aim to lessen layer-specific quantization errors. However, our observations indicate that while these calibrated PTQ models reduce quantization error effects, they do not mitigate the issues stemming from flipped tokens. The case study for a 4-bit quantized model calibrated with AWQLin etal. (2023), shown in Fig.2(d), reveals that although calibration decreases the impacts of $\mathrm{KV_{Perturb}}$ and $\mathrm{Q_{Error}}$, sentence variations are still predominantly influenced by $\mathrm{T_{F}}$. A similar trend can be observed by KD-based QAT (Fig.8), highlighting the need for strategies that specifically address flipped tokens.

3.3 Why Token-Flipping Happens?

We hypothesize that token-flipping occurs due to inherently ambiguous token distributions in sentence generation, which become prone to flipping when quantization errors introduce alterations. To empirically validate this, Fig.3(a) demonstrates token-flipping during text generation by a quantized model. It shows the probabilities for the top-1 and top-2 tokens throughout the auto-regressive generation. Notably, the 16-bit baseline and 4-bit quantized models produce nearly identical probabilities for most tokens. However, at certain points (e.g., $t=0,7,11$), the probability margin between the top-1 and top-2 tokens is minimal. Token-flipping occurs when quantization-induced deviations in the probability distribution surpass this narrow margin, altering subsequent sentence generation and leading to unnatural phrasing.

Fig.3(b) shows the average probability margin between the top-1 and top-2 tokens across each text sample. By feeding identical inputs to each model, we note that the 4-bit quantized model has a narrower average probability margin between the top-1 and top-2 tokens than the 16-bit baseline. This indicates a higher likelihood of the 4-bit model experiencing token-flipping due to quantization error-induced deviations exceeding this margin. Additionally, our examination of beam searchGraves (2012) in Section5.5 reveals its limited effectiveness in mitigating this issue. This underscores the need for strategies that ensure the quantized model retains clear decision-making capabilities.

4.1 Method

Drawing inspiration from the success of DPO in aligning LLMs with human preferences, we have developed a novel approach that extends its application to overcome the challenges introduced by quantization.

The challenge in preference dataset generation arises from human labeling.To mitiagte this, we introduce for efficiently creating dataset $\mathcal{D}_{\text{QDPO}}$, which is composed of triplets $\{y_{w},y_{l},x\}$. Here, $y_{w}$ denotes the response from the full-precision model $\pi_{\text{fp}}$, which is also referred to as the optimal policy. $y_{l}$ represents the corresponding response from the quantized model $\pi_{\text{q}}$. $x$ serves as the prompt. Specifically, $y_{w}$ is obtained as $\arg\max_{y}{\pi_{\text{fp}}(y|x)}$ and $y_{l}$ as $\arg\max_{y}{\pi_{\text{q}}(y|x)}$.The preference of $y_{w}$ over $y_{l}$ is ensured by Theorem4.1. The proof is in A.8. Unlike conventional DPO methods, QDPO automatically distinguishes preferences without relying on expensive human-annotated datasets.

By precisely distinguishing preferences, we can clearly eliminate errors in data labeling. This leads to improved performance by accurately estimating the policy model’s density. $\mathcal{L}_{\text{QDPO}}$ is define with high-quality preference data $\mathcal{D}_{\text{QDPO}}$ as follows:

4.2 Implementation

Given $\pi_{\theta}$ as the quantized model’s policy, integrating $\mathcal{L}_{\text{QDPO}}$ with QAT adjusts for quantization effects. The quantization technique we employ uniformly quantizes each channel across its entire min-max range, ensuring comprehensive accommodation of the full spectrum of values within each channel. To overcome the challenge posed by the non-differentiable rounding within the quantization process, we employ the Straight-Through Estimator (STE) for gradient approximation, facilitating effective gradient approximation and ensuring smooth training despite quantization. As shown in Fig.4, QDPO demonstrates an increase in the chosen reward and a decrease in the rejected reward throughout the training process, indicating effective loss convergence. The complete procedure of QDPO is described in Algorithm 1. Details of the training settings and hyperparameters for QDPO can be found in A.1.

5.1 Experimental Settings

Models. We evaluate QDPO on two representative conversational LLMs. Vicuna-v1.5Zheng etal. (2023), instruction-finetuned from LLaMA2 for improved conversational ability, and a bilingual (English-Korean) LLM, Mi:dmKT-AI (2023), to confirm QDPO’s effectiveness to support multiple languages. All these models have 7B parameters.

Benchmarks. For a comprehensive evaluation of conversational abilities, we employ three distinct benchmarks: MT-BenchZheng etal. (2023), Vicuna-EvalChiang etal. (2023), and FLASKYe etal. (2023). MT-Bench utilizes GPT-4 to evaluate the quality of two responses obtained from an initial question and an additional follow-up question, offering an evaluation of multi-turn responses. For assessing Korean capability, we also translate the MT-Bench dataset into Korean using GPT-4. Vicuna-Eval consists of 80 questions for evaluation by GPT-4 to determine which model generates better sentences. FLASK includes 1.7K samples designed to assess LLM’s fine-grained language skills, such as robustness and harmlessness.

5.2 Experimental Results: MT-Bench

We evaluate quantized LLMs on MT-Bench to understand the impact of different quantization methods on conversational abilities. Following conventionZheng etal. (2023), we report both pairwise comparison and single-answer grading results (A.4 for detailed evaluation metrics).

Pairwise Comparison. Table1 shows the results of pairwise comparison on MT-Bench for various quantized LLMs. Each quantized LLM is compared with the Baseline (16-bit weight) by GPT-4 for their multi-turn responses to the questions in various categories of MT-Bench. We focus on the lose-rate since our alignment objective is to improve the answer quality of the quantized LLM superior to (win) or comparable with (tie) the 16-bit weight baseline. In all the cases, RTN suffers from the highest lose-rate compared to AWQ due to its simplest quantization mechanism. However, lose-rate of the same RTN can be significantly improved by QDPO; QDPO achieves the lowest lose-rate in all the cases. We can further compare QDPO with KD as they finetune the model weights to be quantization-friendly. Interestingly, QDPO outperforms KD with a noticeable increase in winning cases. These results showcase that QDPO can effectively align the answer quality to the 16-bit weight baseline.

Single-Answer Grading. Table 2 presents the single-answer grading results of Mi:dm on MT-Bench across eight categories, each with 10 questions, and reports the average GPT-4 rating (higher is better). Throughout the categories, RTN suffers from the lowest grading due to the quantization errors, which can be marginally improved by AWQ. In contrast, QDPO significantly improves the average grading from RTN, achieving the average grading on par with the 16-bit weight baseline. This also highlights the effectiveness of QDPO in recovering conversational abilities. Details on category-wise analysis can be found in A.5.

5.3 Experimental Results: Vicuna-Eval

Since Vicuna-Eval is a widely used benchmark for evaluating conversational abilities, we further employ it for evaluating QDPO. We take Mi:dm as a target language model to apply different quantization methods and evaluate its performance on 80 questions by GPT-4. As shown in Fig.5, it can be seen that models with QDPO applied exhibit the highest wins and the lowest losses, demonstrating a lose-rate of 50%, which indicates a near recovery of the language capabilities of the baseline model.

5.4 Experimental Results: FLASK

We use the FLASK benchmark on Mi:dm to verify how the proposed method enhances the fine-grained skills of the language model. Fig.6 shows the relative performance of different quantized LLMs across the 12 fine-grained skills. RTN significantly diminishes certain capabilities of the model, while AWQ and KD slightly improve performance toward the 16-bit weight baseline. In contrast, QDPO shows a significant enhancement in most skills; in particular, QDPO significantly improves metacognition skills, whereas RTN and AWQ significantly fall short. (Details on skill-wise analysis can be found in A.6.) Overall, QDPO achieves the abilities closest to the 16-bit weight baseline, showcasing the effectiveness of its alignment objective in recovering conversational skills.

5.5 Ablation Studies

We further conduct ablation studies to provide insights on QDPO for improving the conversational skills of quantized LLMs.

Conversation Abilities vs. Task Accuracy. As discussed, QDPO has the particular role of aligning the quantized LLMs to the 16-bit weight baseline. What is the impact of this alignment on the task-specific performance of LLMs? To answer this question, we further evaluate the quantized LLMs on well-known benchmarks that test the task-specific capability of language models. In particular, Common Sense Question Answering (CSQA)Talmor etal. (2019) and Massive Multitask Language Understanding (MMLU)Hendrycks etal. (2020) assess the models’ reasoning and multitask-solving abilities through multiple-choice questions. Furthermore, DROPDua etal. (2019) and BBHSrivastava etal. (2023) evaluate the problem-solving abilities of instruction-tuned models in logic and math. (Details on the task-specific benchmarks are in A.7.) Table3 compares the task accuracy (CSQA, MMLU, DROP, BBH) as well as the conversational abilities (MT-Bench) on the quantized LLM with and without QDPO. RTN suffers degradation on the task accuracy as well as the conversational abilities. Interestingly, AWQ significantly improves task accuracy while its conversational abilities are marginally improved. Meanwhile, QDPO improves conversational ability while mostly preserving task accuracy, showcasing its usefulness.

Conversation Abilities vs. Perplexity.

Perplexity is a key metric for evaluating language models, as it measures the exponentiated average negative log probability of predicted word sequences. We examine whether the enhanced conversational capabilities through QDPO are also reflected in perplexity by comparing the perplexity and the lose-rate on the MT-bench. We measure perplexity using Wikitext-2Merity etal. (2016) for English and Korean textbooks¹¹1https://huggingface.co/datasets/maywell/korean_textbooks dataset for Korean. As shown in Table4, RTN significantly increases perplexity across all models. While AWQ decreases perplexity in all models, it does not guarantee an improvement in conversational ability. For example, in Mi:dm’s Korean benchmark, AWQ significantly reduces perplexity by 0.55 compared to RTN, yet the lose-rate increases by 2%. On the other hand, QDPO significantly enhances conversational ability, even though it does not achieve as low a perplexity as the baseline. We believe that the discrepancy between perplexity and conversational ability stems from the difficulty of using next-word prediction perplexity on reference text to capture the impact of flipped tokens in an auto-regressive generation. From our observation, these tokens significantly contribute to sentence variation, as discussed in Sec.3.2.

QDPO vs. Beam Search.

Beam search Graves (2012) generates higher probability outcomes by considering multiple generation possibilities simultaneously. Therefore, even if the quantized model makes different judgments from the baseline, which significantly influences sentence generation, there is still a possibility of generating outcomes without issues in overall probability. We aim to observe how decoding strategies affect quantized generation across three beam sizes (1, 3, 5). Table5 shows the results of the MT-Bench pairwise comparison according to decoding strategies. In the case of a beam size of 3, generating a more diverse range of sentences slightly reduces the lose-rate, yet it still exhibits many losses, and increasing the beam size further does not fundamentally solve the problem, as it also increases defeats. In contrast, QDPO demonstrates a lower lose-rate by creating models that are robust to quantization.

A.1 Experimental Details

PTQ Calibration Settings. For PTQ calibration, we use the widely utilized method AWQLin etal. (2023), with the calibration set consisting of 64 samples randomly extracted from the C4Raffel etal. (2019) dataset. We apply channel-wise quantization and do not consider fine-grained quantization (e.g. group quantization) to better observe the impact of quantization on the LLM’s conversational abilities.

Knowledge Distillation Settings. For KD setting, we follow KD method introduced in LLM-QATLiu etal. (2023b), excluding the data curation process. To facilitate a fair comparison with QDPO, we extracted 50,000 prompts from the Anthropic Helpful and Harmless dialogue datasetBai etal. (2022), and set the learning rate to 3e-6.

Training Settings. In our QDPO experiments, similar to the KD, we sample 50,000 prompts in English from the Anthropic Helpful and Harmless dialogue dataset and 21,155 prompts in Korean from the KoAlpaca²²2https://huggingface.co/datasets/beomi/KoAlpaca-v1.1a dataset. We collect responses using both the full-precision policy ($\pi_{\text{fp}}$) and the quantized policy ($\pi_{\text{q}}$) to construct a preference pair dataset. The learning rate is set to 3e-6.

A.2 More Details on Breakdown Analysis

A.3 QDPO’s Compatibility with Existing Techniques

QDPO on RLHF-tuned Models. We conduct additional experiments to investigate whether QDPO can serve as a complementary approach to recover conversational in quantized RLHF-tuned models, such as LLaMA2-ChatTouvron etal. (2023b) in Table6, In LLaMA2-Chat, which has improved conversational abilities through reflecting human preferences via RLHF, W4A16 RTN exhibits a 50% lose-rate compared to the baseline model. However, QDPO demonstrates further recovery of conversational ability. With more aggressive quantization at 3-bit, a clearer trend is observed. RTN experiences a rapid decline in conversational ability. In contrast, QDPO significantly reduces the lose-rate by restoring the conversational ability of the baseline model. This demonstrates that QDPO effectively enhances the conversational ability of a quantized RLHF-tuned model, indicating that it is a method compatible with existing RLHF.

QDPO with Memory-Efficient Fine-Tuning Method. We extend our experiments with QDPO training using LoRAHu etal. (2022). Following the approach in QLoRADettmers etal. (2023), we keep the quantized base weights frozen and train only the high-precision adapter. To ensure a fair comparison with other methods, we utilize INT4 for the base weights instead of NF4Dettmers etal. (2023). The adapter rank and $\alpha$ used in this experiment is 64.As shown in Table7, QDPO with LoRA significantly enhances the conversational ability of quantized LLMs, achieving levels nearly identical to those of QDPO. Moreover, QDPO with LoRA reduces the required memory by keeping the quantized base weights fixed and significantly decreases the number of training parameters by only training the LoRA adapter. These results suggest that QDPO serves as a complementary method that can be utilized alongside other techniques.

A.4 MT-Bench Evaluation Metrics

Pairwise Comparison. In pairwise comparison within MT-Bench for 80 samples, GPT-4 evaluates which model provides better responses between the two models. However, due to most LLMs’ tendency to prefer the first positionZheng etal. (2023), the evaluation occurs twice in reversed order, counting victories only if one model wins in both cases. If judgments reverse or both evaluations result in ties, it counts as an actual tie. We observe that GPT-4 frequently evaluates "tie" more often than usual in comparisons between different models in MT-Bench. This increased frequency of ties is because our study focuses on comparing similar models (the baseline model and the quantized model). We find that cases evaluated as a tie in both positions present many obstacles to the evaluation we desire for judging alignment. For example, as shown in Fig.12 and Fig.13, when the baseline model provides an incorrect answer and the quantized model also offers a wrong answer (but a different response), GPT-4 provides a "tie" because they are both incorrect. However, this "tie" does not reflect our goal of assessing "how well two models are aligned." Therefore, we evaluate a tie only in cases where win/lose changes due to swapping positions, causing GPT-4 confusion. We find this evaluation method to be the most consistent with the results of other benchmarks, like Vicuna-EvalChiang etal. (2023). Results obtained using the original evaluation criteria of MT-Bench is in Table8.

Single-Answer Grading. In single-answer grading, we directly request GPT-4 to assign scores of up to 10 points. While this approach may not be as nuanced as pairwise comparison in model comparisons, it enables observation of how quantization induces changes in specific categories where the model has strengths and weaknesses by measuring absolute scores by category.

A.5 Detailed Analysis by Catagory in MT-Bench

As shown in Table2, QDPO improves overall capability compared to other methods. However, we observe that in some categories, QDPO scores are lower than the AWQ model. We conducted a more detailed observation of GPT-4’s evaluations in areas where QDPO exhibits lower performance. Interestingly, as depicted in Fig.14, we can see that QDPO fails in cases where RTN already provides a good response and receives a high score, almost the same as the baseline generation. We believe this might be the case because QDPO is trained to reject sentences generated by the quantized model, which can lead to optimization challenges in such situations. Additional examples are present in Fig.15.

A.6 Skill-wise Analysis in FLASK

We aim to investigate how QDPO recovers skills that, according to FLASK’s fine-grained categorization, significantly underperform in RTN and AWQ compared to the baseline. As shown in Fig.17, RTN opts for "<[!newline]>" instead of ":", leading to subsequent generations consisting solely of simple listings, and it can be observed that sentences become repetitive as they lengthen. In contrast, models applying QDPO follow the baseline by providing explanations for each item.

A.7 Details of Task-Specific Benchmarks

To assess the reasoning capabilities of Large Language Models (LLMs), benchmarks such as Common Sense Question Answering (CSQA)Talmor etal. (2019) and MMLUHendrycks etal. (2020) have been widely utilized. CSQA assesses models’ reasoning abilities through multiple-choice questions, while MMLU verifies models’ multitask-solving capabilities across 57 different tasks with multiple-choice questions. Recently, benchmarks like DROPDua etal. (2019) and BBHSrivastava etal. (2023) have been used to evaluate the problem-solving abilities of instruction-tuned models, testing skills in logic and math. Additionally, the Helpful, Honest, and Harmless (HHH)Askell etal. (2021) benchmark is widely used to assess the extent to which these models are safe or beneficial to humans. In our experiments, we measure zero-shot CSQA benchmark and average across five tasks (WinoGrandeSakaguchi etal. (2019), COPARoemmele etal. (2011), PIQABisk etal. (2019), BoolQClark etal. (2019), HellaSwagZellers etal. (2019)).

A.8 Proof of Theorem.4.1

A.9 Generation Examples

Fig.10 demonstrates a decline in language model performance due to the generation of different tokens compared to the baseline. The baseline model selects “Wear" following “1.", whereas the PTQ model, experiencing a change in probability ranking, chooses “Always." The PTQ model then repeats this word, leading to expressions that feel awkward to humans. On the other hand, QDPO recovers the probabilities similar to the baseline model, thereby continuing with the natural generation.