Tiered Precision on a 4090: Why Per-Layer Quantization Beats Per-Model

When you squeeze a large language model onto a single consumer GPU like the RTX 4090, the first instinct is to quantize everything to the same bit width. It's simple, it's uniform, and it's almost always wrong. The 4090 has 24 GB of VRAM, which sounds like a lot until you try to fit a 70B model at FP16. So you reach for INT8, or maybe NF4, and you apply it across the board. The result is a model that fits, but the quality loss is uneven: some layers degrade gracefully, others turn into gibberish. The fix is to think about precision as a per-layer decision, not a per-model one.

This article walks through why per-layer quantization is the right approach for edge inference, how to identify which layers can tolerate lower precision, and what practical steps you can take to implement a tiered precision strategy. No benchmarks here—just engineering reasoning and concrete patterns that teams commonly use in production.

The Problem with Uniform Quantization

Uniform quantization treats every layer as equally sensitive to precision loss. That's rarely true. In a transformer, the embedding layer and the final output layer often behave differently from the attention projections or the feed-forward networks. Some layers have weight distributions that are tightly clustered, making them robust to aggressive quantization. Others have outliers that dominate the dynamic range, and clipping those outliers wrecks the model's ability to represent fine distinctions.

A common pattern is that the first few layers and the last few layers are more sensitive than the middle ones. The first layers process raw tokens and need to preserve high-fidelity representations; the last layers produce the final logits and any error there directly affects the output distribution. The middle layers, especially the feed-forward networks, often have redundant capacity and can tolerate lower precision.

If you quantize everything to INT8, you might see acceptable overall quality, but you're wasting bits on layers that could go lower, and you're losing accuracy on layers that need more. The result is a model that's either too large or too inaccurate. Per-layer quantization lets you trade off memory and accuracy where it matters.

Why Per-Layer Quantization Works

Per-layer quantization assigns a different bit width to each layer based on its sensitivity to quantization error. The idea is to use higher precision (e.g., FP16 or INT8) for sensitive layers and lower precision (e.g., INT4 or NF4) for robust ones. This way, you get the memory savings of aggressive quantization without the quality cliff.

The key insight is that the memory footprint of a model is dominated by the number of parameters, not the precision of any single layer. If you have a 7B model, the embeddings and LM head might account for only a few hundred million parameters, while the bulk of the parameters live in the transformer blocks. So you can afford to keep the embeddings at FP16 and still save a lot of memory by dropping the transformer blocks to INT4.

In practice, teams often find that a small number of layers are responsible for most of the quality degradation. By keeping those layers at higher precision, you can maintain near-FP16 quality while using a fraction of the memory. This is especially important for consumer GPUs like the 4090, where every gigabyte counts.

How to Identify Sensitive Layers

There's no universal rule for which layers are sensitive—it depends on the model architecture and the task. But there are systematic ways to find out.

Sensitivity Analysis

A common approach is to perform a sensitivity analysis: quantize one layer at a time (or a group of layers) to a low precision, keep everything else at full precision, and measure the impact on a validation set. This gives you a per-layer error profile. Layers that cause a large drop in accuracy are sensitive; layers that barely change the output are robust.

For example, you might take a model, quantize the attention query projection to INT4, and measure perplexity on a held-out corpus. Then reset it, quantize the value projection, and measure again. After iterating through all layer types, you'll have a ranking. This is computationally expensive, but it's a one-time cost per model architecture.

Heuristics Based on Weight Distributions

A cheaper heuristic is to look at the weight distributions. Layers with high kurtosis (heavy tails) or large outlier values are typically more sensitive to quantization. You can compute the ratio of the maximum absolute value to the standard deviation for each layer's weights. If that ratio is high, the layer has outliers that will be clipped during quantization, leading to larger errors.

For example, in many transformer models, the embedding layer has a high dynamic range because it learns token representations that are spread out. The feed-forward networks, on the other hand, often have weights that are more uniformly distributed, making them good candidates for lower precision.

Attention vs. Feed-Forward

Attention layers tend to be more sensitive than feed-forward layers. The attention mechanism involves softmax, which is sensitive to small changes in logits. A slight perturbation in the query or key projections can shift the attention distribution, causing the model to focus on the wrong tokens. Feed-forward layers, by contrast, apply pointwise nonlinearities that can absorb some noise.

So a typical tiered strategy might look like:

• Embeddings and LM head: FP16 (or even FP32 if you have the memory)
• Attention projections (Q, K, V, O): INT8
• Feed-forward layers: INT4

This is just a starting point. The actual mix depends on your model and your quality requirements.

Implementation Strategies on a 4090

Once you've identified which layers are sensitive, you need to implement the mixed-precision model. There are a few ways to do this, depending on your stack.

Using vLLM with Per-Layer Quantization

If you're using vLLM for inference, you can take advantage of its support for mixed quantization. vLLM allows you to specify quantization per module, which lets you assign different bit widths to different parts of the model. For example, you can set the attention layers to INT8 and the MLP layers to INT4.

Here's a conceptual example of how you might configure it (pseudo-code, not a direct API reference):

from vllm import LLM

# Hypothetical config for per-layer quantization
quant_config = {
    'embedding': 'fp16',
    'lm_head': 'fp16',
    'attention': 'int8',
    'mlp': 'int4'
}

llm = LLM(model="your-model", quantization=quant_config)

In practice, you'd need to check the current vLLM documentation for the exact syntax, but the idea is that you can control precision at a granular level.

Custom Inference with PyTorch

If you're rolling your own inference loop, you can manually quantize individual layers. For example, you could use PyTorch's quantization tools to convert specific nn.Linear layers to INT8, while leaving others as FP16.

import torch
import torch.nn as nn

# Assume model is a transformer
for name, module in model.named_modules():
    if 'mlp' in name and isinstance(module, nn.Linear):
        # Quantize to INT8
        quantized = torch.quantization.quantize_dynamic(
            module, {nn.Linear}, dtype=torch.qint8
        )
        # Replace the module
        setattr(model, name, quantized)

This approach gives you full control but requires more manual work.

Calibration and Data-Free Quantization

For per-layer quantization, you need to calibrate the quantization ranges for each layer. This typically involves passing a small calibration dataset through the model and recording the activation ranges. Some methods use data-free quantization, which estimates ranges from the weights themselves, but this is less accurate.

A common pattern is to use a few hundred samples from your training data as calibration. For each layer, you collect the min and max values of the weights (for weight quantization) and the activations (for activation quantization). Then you compute the scale and zero-point for each layer.

Trade-offs and Pitfalls

Per-layer quantization is not a free lunch. There are several trade-offs to consider.

Kernel Efficiency

Mixed precision can hurt kernel efficiency. GPUs are optimized for uniform data types. If a layer is INT4 and another is INT8, the GPU may need to switch kernels or perform type conversions, which adds overhead. In some cases, the memory savings are offset by slower inference.

To mitigate this, you can group layers with the same precision together and ensure that operations are batched appropriately. Some inference engines, like vLLM, handle this internally, but custom implementations may need careful tuning.

Complexity

Per-layer quantization adds complexity to the model loading and inference pipeline. You need to maintain a mapping of layers to precisions, and ensure that the quantization parameters are correctly applied. This can be error-prone, especially when you're iterating on the model.

Overfitting to Calibration Data

The calibration process can overfit to the calibration dataset. If you choose a calibration set that isn't representative of your actual input distribution, the quantization ranges may be off, leading to poor performance on real data. It's important to use a diverse calibration set that covers the range of inputs you expect.

A Practical Workflow

Here's a step-by-step workflow that teams commonly use to implement per-layer quantization:

1. Start with a baseline: Run the model in FP16 and measure quality on your validation set. This gives you a reference point.
2. Perform sensitivity analysis: Quantize each layer type to a low precision (e.g., INT4) one at a time and measure the impact. Rank layers by sensitivity.
3. Define precision tiers: Based on the sensitivity ranking, assign precisions. For example, top 20% sensitive layers get FP16, middle 50% get INT8, bottom 30% get INT4.
4. Calibrate: Run a calibration dataset through the model to compute quantization parameters for each layer.
5. Implement: Modify your inference engine to apply the per-layer quantization.
6. Validate: Measure quality and memory usage. If quality is too low, upgrade some layers to higher precision. If memory is too high, downgrade some layers.
7. Iterate: Repeat steps 4-6 until you find the sweet spot.

Real-World Considerations

On a 4090, you have 24 GB of VRAM. A 7B model at FP16 takes about 14 GB, leaving room for activations and KV cache. If you quantize to INT8, you can fit a 13B model. With per-layer quantization, you might fit a 13B model with better quality than a uniform INT8, or even fit a 30B model with acceptable quality.

But remember: the 4090 is a consumer card. It doesn't have the memory bandwidth of a data-center GPU like an A100. So you need to be mindful of memory bandwidth as well as capacity. Lower precision not only reduces memory footprint but also reduces the amount of data that needs to be read from VRAM, which can speed up inference.

Another consideration is the software stack. If you're using vLLM, you benefit from its optimized kernels for various precisions. But not all precisions are supported for all operations. For example, INT4 might only be supported for linear layers, not for embeddings. You need to check the capabilities of your inference engine.

Why Not Just Use NF4?

NF4 (4-bit NormalFloat) is a popular quantization format that works well for weights. It's designed to be more accurate than uniform INT4 because it allocates more quantization levels to values near zero, where most weights are concentrated. However, NF4 is still a uniform precision across the model. It doesn't solve the problem of sensitivity variation across layers.

You can combine NF4 with per-layer quantization: use NF4 for robust layers and INT8 or FP16 for sensitive ones. This gives you the best of both worlds.

Conclusion

Per-layer quantization is a powerful technique for getting the most out of limited GPU memory. By treating precision as a per-layer decision, you can achieve better quality than uniform quantization at the same memory footprint, or fit a larger model without sacrificing too much accuracy.

The key is to understand your model's sensitivity to quantization. Use sensitivity analysis to identify which layers need high precision, and don't be afraid to experiment with different precision mixes. The 4090 is a capable card, but it demands careful engineering to run large models effectively.

In the end, the goal is not to quantize for the sake of quantizing, but to make the best use of the hardware you have. Per-layer quantization is a tool that helps you do exactly that.