msgpack buffer with a huge number of nested arrays could lead to a stack overflow

msgpack buffer with a huge number of nested arrays could lead a stack  overflow.

### How to reproduce

Setup cmsgpack and luzer [^1]:

```sh
luarocks install --tree modules --lua-version 5.1 lua-cmsgpack 0.4.0-0 CC="clang" CFLAGS="-ggdb -fPIC -fsanitize=fuzzer-no-link"
eval $(luarocks path)                                                             
export LUA_CPATH="$LUA_CPATH;modules/lib/lua/5.1/?.so;../../?.so"              
export LUA_PATH="$LUA_PATH;modules/lib/lua/5.1/?.lua"
```

Create a file `msgpack.lua` with test harness:

```lua
  local luzer = require("luzer")                                                   
  local msgpack = require("cmsgpack")                                              
  local msgpack_safe = require("cmsgpack.safe")                                    
  local unpack = unpack or table.unpack                                            
                                                                                   
  local function TestOneInput(buf)                                                 
      if #buf == 0 then return end                                                 
      local data, err = msgpack_safe.unpack(buf)                                   
      if not err and data ~= 0 then                                                
          local res = { msgpack_safe.unpack(data) }                                
          if #res ~= 0 then                                                        
              msgpack.pack(unpack(res))                                            
          end                                                                      
      end
  end                                                                              
                                                                                   
  luzer.Fuzz(TestOneInput)
```

Execute the test harness (feel free to use PUC Rio Lua or LuaJIT instead of Tarantool):

```sh
LD_PRELOAD=build/luzer/libfuzzer_with_asan.so tarantool msgpack.lua

INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed: 1177135163
INFO: Loaded 2 modules   (243 inline 8-bit counters): 77 [0x7ffff5eca963, 0x7ffff5eca9b0), 166 [0x7ffff668d310, 0x7ffff668d3b6),
INFO: Loaded 2 PC tables (243 PCs): 77 [0x7ffff5eca9b0,0x7ffff5ecae80), 166 [0x7ffff668d3b8,0x7ffff668de18),
[New Thread 0x7bffeecb96c0 (LWP 38743)]
INFO: -max_len is not provided; libFuzzer will not generate inputs larger than 4096 bytes
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 30 ft: 31 corp: 1/1b exec/s: 0 rss: 188Mb
#3      NEW    cov: 30 ft: 46 corp: 2/2b lim: 4 exec/s: 0 rss: 188Mb L: 1/1 MS: 1 ChangeByte-
#4      NEW    cov: 30 ft: 60 corp: 3/4b lim: 4 exec/s: 0 rss: 188Mb L: 2/2 MS: 1 InsertByte-
#20     NEW    cov: 30 ft: 70 corp: 4/5b lim: 4 exec/s: 0 rss: 188Mb L: 1/2 MS: 1 EraseBytes-
#22     NEW    cov: 33 ft: 75 corp: 5/8b lim: 4 exec/s: 0 rss: 188Mb L: 3/3 MS: 2 CrossOver-InsertByte-
#25     NEW    cov: 35 ft: 77 corp: 6/9b lim: 4 exec/s: 0 rss: 188Mb L: 1/3 MS: 3 InsertByte-EraseBytes-ChangeByte-
#32     NEW    cov: 36 ft: 79 corp: 7/11b lim: 4 exec/s: 0 rss: 188Mb L: 2/3 MS: 2 CrossOver-InsertByte-
        NEW_FUNC[1/1]: [Detaching after fork from child process 38744]
0x7ffff6689450 in mp_decode_to_lua_hash /tmp/luarocks_lua-cmsgpack-0.4.0-0-5478545/lua-cmsgpack/lua_cmsgpack.c:561

<snipped>

#254022 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 841 exec/s: 127011 rss: 188Mb L: 89/809 MS: 3 InsertByte-ChangeBinInt-EraseBytes-
#255673 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 850 exec/s: 127836 rss: 188Mb L: 15/809 MS: 1 EraseBytes-
#259576 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 886 exec/s: 129788 rss: 188Mb L: 94/809 MS: 3 ShuffleBytes-InsertByte-EraseBytes-
#259928 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 886 exec/s: 129964 rss: 188Mb L: 318/809 MS: 2 PersAutoDict-EraseBytes- DE: "\031\00
0\000\000"-
#260186 REDUCE cov: 137 ft: 689 corp: 382/20Kb lim: 886 exec/s: 130093 rss: 188Mb L: 36/809 MS: 3 ShuffleBytes-CopyPart-EraseBytes-
#262144 pulse  cov: 137 ft: 689 corp: 382/20Kb lim: 904 exec/s: 131072 rss: 188Mb
#263653 NEW    cov: 137 ft: 690 corp: 383/21Kb lim: 913 exec/s: 131826 rss: 188Mb L: 887/887 MS: 2 CopyPart-CopyPart-
#265109 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 922 exec/s: 132554 rss: 188Mb L: 25/887 MS: 1 EraseBytes-
#265187 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 922 exec/s: 132593 rss: 188Mb L: 24/887 MS: 3 ChangeBit-ShuffleBytes-EraseBytes-
#267424 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 940 exec/s: 133712 rss: 188Mb L: 38/887 MS: 2 CopyPart-EraseBytes-
#267515 REDUCE cov: 137 ft: 690 corp: 383/21Kb lim: 940 exec/s: 133757 rss: 188Mb L: 65/887 MS: 1 EraseBytes-
#267753 NEW    cov: 137 ft: 691 corp: 384/22Kb lim: 940 exec/s: 133876 rss: 188Mb L: 937/937 MS: 3 ChangeByte-ChangeByte-CopyPart-
#268845 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 949 exec/s: 134422 rss: 188Mb L: 37/937 MS: 2 CopyPart-EraseBytes-
#270625 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 958 exec/s: 135312 rss: 188Mb L: 145/937 MS: 5 CopyPart-CopyPart-PersAutoDict-EraseB
ytes-InsertByte- DE: "\377\377"-
#274896 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 994 exec/s: 137448 rss: 188Mb L: 10/937 MS: 1 EraseBytes-
#276032 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1003 exec/s: 138016 rss: 188Mb L: 36/937 MS: 1 EraseBytes-
#276065 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1003 exec/s: 138032 rss: 188Mb L: 34/937 MS: 3 ChangeBit-ChangeBinInt-EraseBytes-
#277030 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1012 exec/s: 138515 rss: 188Mb L: 936/936 MS: 5 EraseBytes-ChangeBinInt-InsertByte-I
nsertRepeatedBytes-CopyPart-
#277081 REDUCE cov: 137 ft: 691 corp: 384/22Kb lim: 1012 exec/s: 138540 rss: 188Mb L: 17/936 MS: 1 EraseBytes-
#278252 REDUCE cov: 137 ft: 693 corp: 385/22Kb lim: 1021 exec/s: 139126 rss: 188Mb L: 35/936 MS: 1 CrossOver-
#279465 NEW    cov: 137 ft: 699 corp: 386/22Kb lim: 1030 exec/s: 139732 rss: 188Mb L: 35/936 MS: 3 CrossOver-ChangeBinInt-ChangeBinInt-
#280317 REDUCE cov: 137 ft: 699 corp: 386/22Kb lim: 1030 exec/s: 140158 rss: 188Mb L: 173/936 MS: 2 ChangeBit-EraseBytes-
#282109 REDUCE cov: 137 ft: 699 corp: 386/22Kb lim: 1040 exec/s: 141054 rss: 188Mb L: 213/936 MS: 2 PersAutoDict-EraseBytes- DE: "\000\0
00\000\000\000\000\000\000"-
#282306 NEW    cov: 137 ft: 700 corp: 387/23Kb lim: 1040 exec/s: 141153 rss: 188Mb L: 1021/1021 MS: 2 InsertByte-CopyPart-
#283698 REDUCE cov: 137 ft: 700 corp: 387/23Kb lim: 1050 exec/s: 141849 rss: 188Mb L: 13/1021 MS: 2 InsertByte-EraseBytes-
#285356 REDUCE cov: 137 ft: 700 corp: 387/23Kb lim: 1060 exec/s: 142678 rss: 188Mb L: 391/1021 MS: 3 ChangeBinInt-ChangeByte-EraseBytes-

Thread 1 "tarantool" received signal SIGSEGV, Segmentation fault.
0x00007ffff6688098 in mp_decode_to_lua_array (L=0x7bfff07021b0, c=0x0, len=93824994521398) at lua_cmsgpack.c:548
warning: 548    lua_cmsgpack.c: No such file or directory
```

<details>

<summary>Backtrace</summary>

```c++
(gdb) bt
#0  0x00007ffff6688098 in mp_decode_to_lua_array (L=0x7bfff07021b0, c=0x0, len=93824994521398) at lua_cmsgpack.c:548
#1  0x00007ffff6689391 in mp_decode_to_lua_type (L=0x40000378, c=0x7bfff0780530) at lua_cmsgpack.c:778
#2  0x00007ffff66881ae in mp_decode_to_lua_array (L=0x40000378, c=0x7bfff0780530, len=9) at lua_cmsgpack.c:555
#3  0x00007ffff6689391 in mp_decode_to_lua_type (L=0x40000378, c=0x7bfff0780530) at lua_cmsgpack.c:778
#4  0x00007ffff66881ae in mp_decode_to_lua_array (L=0x40000378, c=0x7bfff0780530, len=9) at lua_cmsgpack.c:555
#5  0x00007ffff6689391 in mp_decode_to_lua_type (L=0x40000378, c=0x7bfff0780530) at lua_cmsgpack.c:778
#6  0x00007ffff66881ae in mp_decode_to_lua_array (L=0x40000378, c=0x7bfff0780530, len=9) at lua_cmsgpack.c:555
#7  0x00007ffff6689391 in mp_decode_to_lua_type (L=0x40000378, c=0x7bfff0780530) at lua_cmsgpack.c:778
#8  0x00007ffff66881ae in mp_decode_to_lua_array (L=0x40000378, c=0x7bfff0780530, len=9) at lua_cmsgpack.c:555
#9  0x00007ffff6689391 in mp_decode_to_lua_type (L=0x40000378, c=0x7bfff0780530) at lua_cmsgpack.c:778
#10 0x00007ffff66881ae in mp_decode_to_lua_array (L=0x40000378, c=0x7bfff0780530, len=9) at lua_cmsgpack.c:555

<snipped>

#2085 0x00007ffff6689391 in mp_decode_to_lua_type (L=0x40000378, c=0x7bfff0780530) at lua_cmsgpack.c:778
#2086 0x00007ffff6689860 in mp_unpack_full (L=0x40000378, limit=2147483647, offset=0) at lua_cmsgpack.c:813
#2087 0x00007ffff6689a45 in mp_unpack (L=0x40000378) at lua_cmsgpack.c:843
#2088 0x00005555557805a3 in lj_BC_FUNCC () at buildvm_x86.dasc:811
#2089 0x0000555555787355 in lua_pcall (L=<optimized out>, nargs=<optimized out>, nresults=<optimized out>, errfunc=<optimized out>)
    at ./third_party/luajit/src/lj_api.c:1173
#2090 0x00007ffff6689c23 in mp_safe (L=0x40000378) at lua_cmsgpack.c:872
#2091 0x00005555557805a3 in lj_BC_FUNCC () at buildvm_x86.dasc:811
#2092 0x00007ffff5e82a1a in luaL_test_one_input (L=0x40000378) at /home/sergeyb/sources/luzer/luzer/luzer.c:287
#2093 0x00007ffff5e82962 in TestOneInput (data=0x7d8ff39a2680 "\232\232~\232\232\232\232\232\031", size=1092)
    at /home/sergeyb/sources/luzer/luzer/luzer.c:342
#2094 0x00007ffff745c78c in fuzzer::Fuzzer::ExecuteCallback(unsigned char const*, unsigned long) ()
   from build/luzer/libfuzzer_with_asan.so
#2095 0x00007ffff745be26 in fuzzer::Fuzzer::RunOne(unsigned char const*, unsigned long, bool, fuzzer::InputInfo*, bool, bool*) ()
   from build/luzer/libfuzzer_with_asan.so
#2096 0x00007ffff745d7d6 in fuzzer::Fuzzer::MutateAndTestOne() () from build/luzer/libfuzzer_with_asan.so
#2097 0x00007ffff745e2a6 in fuzzer::Fuzzer::Loop(std::vector<fuzzer::SizedFile, std::allocator<fuzzer::SizedFile> >&) ()
   from build/luzer/libfuzzer_with_asan.so
#2098 0x00007ffff744b6b6 in fuzzer::FuzzerDriver(int*, char***, int (*)(unsigned char const*, unsigned long)) ()
   from build/luzer/libfuzzer_with_asan.so
#2099 0x00007ffff5e8307b in luaL_fuzz (L=0x40000378) at /home/sergeyb/sources/luzer/luzer/luzer.c:537
#2100 0x00005555557805a3 in lj_BC_FUNCC () at buildvm_x86.dasc:811
#2101 0x0000555555787355 in lua_pcall (L=L@entry=0x40000378, nargs=<optimized out>, nresults=<optimized out>, errfunc=errfunc@entry=0)
    at ./third_party/luajit/src/lj_api.c:1173
#2102 0x0000555555725b8f in luaT_call (L=0x40000378, nargs=<optimized out>, nreturns=<optimized out>) at ./src/lua/utils.c:625
#2103 0x000055555571cc11 in lua_main (L=L@entry=0x40000378, is_debugging=is_debugging@entry=false, argc=argc@entry=1,
    argv=argv@entry=0x7e3ff33e2108) at ./src/lua/init.c:1032
#2104 0x000055555571d430 in run_script_f (ap=<error reading variable: value has been optimized out>) at ./src/lua/init.c:1155
#2105 0x00005555555a1431 in fiber_cxx_invoke(fiber_func, typedef __va_list_tag __va_list_tag *) (f=<optimized out>, ap=<optimized out>)
    at ./src/lib/core/fiber.h:1248
#2106 0x000055555574a0c6 in fiber_loop (data=<optimized out>) at ./src/lib/core/fiber.c:1153
#2107 0x00005555559ab6dc in coro_init () at ./third_party/coro/coro.c:108
(gdb) quit
```

</details>


[^1]: https://github.com/ligurio/luzer

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

msgpack buffer with a huge number of nested arrays could lead to a stack overflow #70

How to reproduce

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

msgpack buffer with a huge number of nested arrays could lead to a stack overflow #70

Description

How to reproduce

Footnotes

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions