Programming

H.264 CAVLC: run_before > zerosLeft Table 9-10 Fix

Fix H.264 CAVLC decoding errors where run_before exceeds zerosLeft in Table 9-10. Learn validation rules, prevent negative zerosLeft, and implement compliant residual block decoding for H.264/AVC decoders.

1 answer• 1 view

H.264 CAVLC: Correctly applying Table 9-10 ‘run_before’ when decoded value exceeds zerosLeft

Implementing an H.264/AVC decoder (ISO/IEC 14496-10:2022) for residual data in macroblocks using CAVLC. The run_before decoding from Table 9-10 produces invalid values greater than zerosLeft, leading to negative zerosLeft.

Problem Details

  • runVal starts with three values: 0,0,0. zerosLeft = 12 (from If(totalCoeff < endIdx - startIdx + 1, total_zeros, 0)).
  • In the for-loop, DecodeRunBefore returns: 3, 13, 0.
  • Subtracting sequentially: zerosLeft becomes -4.
  • Finally, runVal(totalCoeff - 1) = zerosLeft (negative value written).

In DecodeRunBefore:

  • Reads bits incrementally, matches first VLC in Table 9-10, column 7.
  • Failing bitstream: 0000000001 → matches run_before = 13 (column 7), but 13 > zerosLeft = 12 → negative zerosLeft.

Relevant Code

RunBefore Table (VB.NET)

vb
Private Shared Function Get_RunBefore_Table() As Dictionary(Of Integer, Dictionary(Of String, Integer))
 Dim d As New Dictionary(Of Integer, Dictionary(Of String, Integer))()

 d.Add(1, New Dictionary(Of String, Integer) From { {"1", 0}, {"0", 1} })
 d.Add(2, New Dictionary(Of String, Integer) From { {"1", 0}, {"01", 1}, {"00", 2} })
 d.Add(3, New Dictionary(Of String, Integer) From { {"11", 0}, {"10", 1}, {"01", 2}, {"00", 3} })
 d.Add(4, New Dictionary(Of String, Integer) From { {"11", 0}, {"10", 1}, {"01", 2}, {"001", 3}, {"000", 4} })
 d.Add(5, New Dictionary(Of String, Integer) From { {"11", 0}, {"10", 1}, {"011", 2}, {"010", 3}, {"001", 4}, {"000", 5} })
 d.Add(6, New Dictionary(Of String, Integer) From { {"11", 0}, {"10", 1}, {"01", 2}, {"001", 3}, {"0001", 4}, {"0000", 5} })
 d.Add(7, New Dictionary(Of String, Integer) From {
 {"111", 0}, {"110", 1}, {"101", 2}, {"100", 3}, {"011", 4}, {"010", 5}, {"001", 6}, {"0001", 7},
 {"00001", 8}, {"000001", 9}, {"0000001", 10}, {"00000001", 11}, {"000000001", 12}, {"0000000001", 13}, {"00000000001", 14}
 })
 Return d
End Function

DecodeRunBefore (VB.NET)

vb
Private Function DecodeRunBefore(zerosLeft As Integer, ByRef avc_Bitstream As BitStream, nalUnit As Byte(), ByRef absoluteBitPos As UInteger) As (runBefore As Integer, success As Boolean)
 Dim col As Integer
 If zerosLeft <= 0 Then Return (0, True)
 col = If(zerosLeft >= 7, 7, zerosLeft)

 Dim runBeforeTable As Dictionary(Of Integer, Dictionary(Of String, Integer)) = Get_RunBefore_Table()
 If Not runBeforeTable.ContainsKey(col) Then Return (0, False)

 Dim table As Dictionary(Of String, Integer) = runBeforeTable(col)
 Dim bitBuffer As String = String.Empty

 Do
 Dim bit As Integer = ByteTools.GetBit(nalUnit(avc_Bitstream.BytePos), avc_Bitstream.BitPos, avc_Bitstream, absoluteBitPos)
 bitBuffer &= If(bit = 1, "1", "0")

 If table.ContainsKey(bitBuffer) Then Return (table(bitBuffer), True)
 If bitBuffer.Length > 16 Then Exit Do
 Loop
 Return (0, False)
End Function

Usage in DecodeResidualBlock_CAVLC (key loop)

vb
Dim zerosLeft As Integer = If(totalCoeff < endIdx - startIdx + 1, total_zeros, 0)
Dim runVal() As Integer = New Integer(totalCoeff - 1) {}

For i As Integer = 0 To totalCoeff - 2
 If zerosLeft > 0 Then
 Dim res As (runBefore As Integer, success As Boolean) = DecodeRunBefore(zerosLeft, AVC_Bitstream, nalUnit, absoluteBitPos)
 If Not res.success Then Throw New Exception("Invalid run_before VLC.")
 runVal(i) = res.runBefore
 Else
 runVal(i) = 0
 End If
 zerosLeft -= runVal(i)
Next
runVal(totalCoeff - 1) = zerosLeft

Question

Is there a special rule in H.264 CAVLC (e.g., partial unary coding, truncation, prefix-less values) that prevents run_before > zerosLeft? How should Table 9-10 column 7 be applied correctly to avoid negative zerosLeft?

In H.264 CAVLC decoding for residual blocks, Table 9-10’s run_before values must never exceed zerosLeft—doing so signals an invalid bitstream, as your example with 13 > 12 demonstrates. The spec requires using column 7 only when zerosLeft >=7, but always validate the decoded run_before <= zerosLeft afterward to prevent negative zerosLeft. No special truncation or partial coding applies; simply reject oversized values to match compliant decoder behavior.

Contents

Sources

  1. Vcodex: H.264/AVC Context Adaptive Variable Length Coding
  2. Alibaba Cloud: CAVLC and CABAC Introduction
  3. Springer: CAVLCU GPU Implementation
  4. CuPDF: CAVLC Example
  5. Wobbly Cucumber: CAVLC Encoding Tutorial
  6. Google Patents: US20090316792 Decoding Method
  7. GitHub: h264-fer residual.cpp

Conclusion

H.264 CAVLC decoding demands strict validation after Table 9-10 lookups: run_before can’t exceed zerosLeft, or it’s a bitstream error. Fix your decoder by adding that check in DecodeRunBefore, update zerosLeft safely, and you’ll avoid negatives while staying spec-compliant. Test with conformance streams—suddenly, those edge cases like “0000000001” won’t crash your residual block parsing.

Authors
NeuroAnswers
Author
Verified by moderation
NeuroAnswers
Moderation
H.264 CAVLC: run_before > zerosLeft Table 9-10 Fix