topology: harden binary topology decoder against malformed input

[HarshRajSinghania]

As discussed over email with Jaroslav, this hardens the binary topology decoder
against malformed / untrusted .tplg input. Three related issues, each confirmed
with AddressSanitizer:

1. snd_tplg_decode() (src/topology/decoder.c): hdr->size + hdr->payload_size
   is evaluated in 32-bit and can wrap, bypassing the payload bounds check; the
   block decoder is then called with the raw (huge) payload_size, causing an
   out-of-bounds read. Fixed by doing the arithmetic in 64-bit (and the advance
   in size_t).

2. tplg_decode_dapm_widget() (src/topology/dapm.c): the kcontrol loop reads the
   control header (chdr->type, then mc/ec/bc->size and ->priv.size) before
   the size2 > size check, giving an out-of-bounds read on a truncated payload.
   Fixed by adding size < sizeof(*chdr) / sizeof(*mc|ec|bc) guards and
   computing size2 in size_t.

3. tplg_decode_dapm_graph() (src/topology/dapm.c): alloca() is called with a
   size derived from the input payload, allowing stack exhaustion on large input.
   Fixed by switching to calloc() / free(), which fails gracefully.

Scope is the debug-oriented decode path only, as you noted. Standalone ASan PoC
harnesses are available if useful.

Signed-off-by: Harsh Raj Singhania raj.harshraut@gmail.com

[tiwai]

All issues look independent, so could you rather split the changes to individual commits instead of squashing into one?

[HarshRajSinghania]

Thanks @tiwai — done. The changes are now split into individual commits instead of one squashed commit:

1. topology: fix integer overflow in snd_tplg_decode()
2. topology: harden dapm widget and graph decoders against malformed input

Please take another look when you have a moment.