AI for Everyone, Part 3 — Loss of 0.0017, Accuracy of 7.5%: The Culprit Was One Padding Slot

• When perfect metrics are the most dangerous
• Experiment design
• Architecture — two input types in one sequence
  • Different mask rules for different spans
• First result — loss 0.0017, accuracy 7.5%
• Finding the culprit — look at the output
  • Why it could not stop
• The fix — one line
• Result after the fix
• What we learned
• 🧠 Comprehension quiz
• References

When perfect metrics are the most dangerous

This part targets VQA (Visual Question Answering): feed the model an image together with a question, and it produces an answer.

[handwritten digit image] + "is it even or odd?"  ->  "odd"
[handwritten digit image] + "how many holes?"      ->  "zero"

The headline result first: the initial model reached a loss of 0.0017 with 7.5% accuracy. By loss alone it had converged almost perfectly, yet its accuracy was below random guessing.

Half of this article is about building the model. The other half is about hunting down what that 7.5% actually was. To spoil the ending: the model had done nothing wrong.

Experiment design

Each MNIST image was paired with one of four randomly chosen questions.

The last one is interesting. The digit 8 has two holes; 0, 4, 6, and 9 have one; the rest have none. Because the same image must produce different answers depending on what is asked, we can verify that the model is genuinely reading the question.

Architecture — two input types in one sequence

The image is compressed by a CNN into 16 tokens, the question is embedded character by character, and the two are concatenated. The whole thing becomes a single sequence a transformer can process.

class VQA(nn.Module):
    def __init__(self):
        super().__init__()
        self.cnn = nn.Sequential(
            nn.Conv2d(1, 32, 3, 2, 1), nn.GELU(),    # 28 -> 14
            nn.Conv2d(32, 64, 3, 2, 1), nn.GELU(),   # 14 -> 7
            nn.Conv2d(64, 128, 3, 2, 1), nn.GELU(),  # 7 -> 4
        )
        self.img_proj = nn.Linear(128, 192)
        self.emb = nn.Embedding(V, 192)
        self.pos = nn.Embedding(64, 192)
        layer = nn.TransformerEncoderLayer(192, 6, 768, batch_first=True, norm_first=True)
        self.tr = nn.TransformerEncoder(layer, 3)
        self.head = nn.Linear(192, V)

The CNN reduces a 28×28 image to 4×4×128, which flattens into 16 vectors. Each vector covers one region of the image — these are visual tokens. Today's large multimodal models work the same way at heart.

Different mask rules for different spans

This is the heart of the model. The sequence has three spans.

[16 image tokens][24 question tokens][6 answer tokens]

The first two may look at each other freely: understanding the question requires seeing the whole image, and interpreting the image requires knowing the question. The answer span, however, may only look backwards, so that it can be generated one character at a time.

T = h.shape[1]
A0 = 16 + q.shape[1]            # where the answer starts
m = torch.zeros(T, T, device=h.device)
m[:, A0:] = float("-inf")       # first, hide the answer span from everyone
idx = torch.arange(T, device=h.device)
m[idx.unsqueeze(0) <= idx.unsqueeze(1)] = 0   # allow self and past
m[:A0, A0:] = float("-inf")     # re-block only the prefix from peeking ahead

Image and question spans are bidirectional; the answer span is causal. Two rules coexist inside one attention operation.

First result — loss 0.0017, accuracy 7.5%

The record after ten minutes of training.

[  603.7s] SUMMARY {"name": "03-image-text-to-text", "steps": 82120,
                    "final_loss": 0.0017, "vqa_acc": 0.075, "params": 1475870}

82,120 steps, loss 0.0017. That normally means training went perfectly. Yet accuracy is 7.5%.

Count the answer space and random guessing should beat that: even/odd is one of two, yes/no is one of two. When loss and accuracy disagree this badly, one of them is lying.

Finding the culprit — look at the output

When metrics contradict each other, the fastest route is to look at what the model actually emits.

Q: how many holes?         (expected zero) -> A: zeroe
Q: is it even or odd?      (expected even) -> A: evene
Q: what digit is this?     (expected one)  -> A: oneeo
Q: is it bigger than four? (expected no)   -> A: nonrn
Q: is it even or odd?      (expected odd)  -> A: oddee

There it is. The model got every answer right. zero, even, one, no, odd are all present. It simply could not stop after finishing, appended junk characters, and exact-match scoring marked every one of them wrong.

So 7.5% was not the model's ability but an artifact of scoring. The model had perfectly learned what to answer and had never learned when to stop.

Why it could not stop

The code that built the target sequence looked like this.

def tok(s, L):
    ids = [enc[c] for c in s][:L]
    return ids + [0] * (L - len(ids))     # 0 = PAD

a_full = torch.tensor([tok(s, AL - 1) + [1] for s in ans])   # 1 = EOS

With AL equal to 6, what happens to "even"?

tok("even", 5)  ->  [e, v, e, n, PAD]
+ [EOS]         ->  [e, v, e, n, PAD, EOS]

A padding slot slipped in between the answer and the EOS. And padding is excluded from the loss.

loss = F.cross_entropy(..., ignore_index=0)   # PAD is ignored

As a result, the target at position 4 — the slot right after the answer ends — is PAD, and PAD is dropped from the loss, so that position never received a single training signal. At inference the model has never learned what belongs there, so it emits whatever.

This also explains the loss of 0.0017. Only four or five of the six slots are actually scored, and those are short answers drawn from a small set, so they are easy to memorise. Loss is computed only over scored positions, so problems at unscored positions never show up in it.

The fix — one line

Attach EOS immediately after the answer, then pad.

def tok_ans(s, L):
    """For answers: put EOS right after the answer, then pad."""
    ids = [enc[c] for c in s][:L - 1] + [1]
    return ids + [0] * (L - len(ids))

tok_ans("even", 6)  ->  [e, v, e, n, EOS, PAD]

Now the target after "n" is EOS, and EOS is not padding, so it counts toward the loss. The model finally has a chance to learn that the answer ends here.

Result after the fix

[  603.1s] SUMMARY {"name": "03-image-text-to-text", "steps": 82030,
                    "final_loss": 0.00075, "vqa_acc": 0.995, "params": 1475870}

Same model, same hyperparameters, same training time. One line changed in how targets are built, and 7.5% became 99.5%.

The outputs after the fix.

Q: is it even or odd?      (expected odd)  -> A: odd
Q: how many holes?         (expected zero) -> A: zero
Q: is it bigger than four? (expected no)   -> A: no
Q: what digit is this?     (expected nine) -> A: nine

What we learned

Loss only sees what it is shown. Positions excluded by ignore_index cannot influence it, so they can be broken while the loss stays quiet. Whenever you use masking or an ignore index, check what is being excluded.

When metrics contradict, look at the output. A loss of 0.0017 and an accuracy of 7.5% cannot both be true. Tuning hyperparameters at that point is wasted time; a few lines of real output answer the question instantly. The moment you see zeroe, the problem is obvious.

Exact-match scoring is harsh, and that is why it is useful. Had we scored loosely — "correct if the answer appears in the output" — this model would have shown 100% from the start and the bug would have survived forever. Strict scoring exposed it.

The next part covers captioning: attaching sentences to images. That experiment's code did not have this bug, and we will look at why not.

🧠 Comprehension quiz

References

• MNIST dataset
• VQA — Visual Question Answering
• PyTorch cross_entropy and ignore_index