lame: two UBSAN crashes

["Agostino Sarubbo" <ago () gentoo org>]

Description:
lame is a high quality MPEG Audio Layer III (MP3) encoder licensed under the LGPL.

Few notes before the details of this bug. Time ago a fuzz was done by Brian Carpenter and Jakub Wilk which posted the 
results on the debian 
bugtracker. In cases like this, when upstream is not active and people do not post on the upstream bugzilla is easy 
discover duplicates, so I 
downloaded all available testcases, and noone of the bug you will see on my blog is a duplicate of an existing issue. 
Upstream seems a bit 
dead, latest release was into 2011, so this blog post will probably forwarded on the upstream bugtracker just for the 
record.

The complete ASan output of the issue:

# lame -f -V 9 $FILE out.wav
/var/tmp/portage/media-sound/lame-3.99.5-r1/work/lame-3.99.5/frontend/brhist.c:204:60: runtime error: signed integer 
overflow: 953447384 + 
1908859798 cannot be represented in type 'int'
Reproducer:
https://github.com/asarubbo/poc/blob/master/00298-lame-signintoverflow-brhist.c
CVE:
N/A

#######################

# lame -f -V 9 $FILE out.wav
/var/tmp/portage/media-sound/lame-3.99.5-r1/work/lame-3.99.5/frontend/get_audio.c:1234:21: runtime error: value -nan is 
outside the range of 
representable values of type 'int'
Reproducer:
https://github.com/asarubbo/poc/blob/master/00299-lame-outside-int-get_audio.c
CVE:
N/A

#######################

Affected version:
3.99.5

Fixed version:
N/A

Commit fix:
N/A

Credit:
These bugs were discovered by Agostino Sarubbo of Gentoo.

Timeline:
2017-06-01: bug discovered
2017-06-17: blog post about the issue

Note:
These bugs were found with American Fuzzy Lop.

Permalink:
https://blogs.gentoo.org/ago/2017/06/17/lame-two-ubsan-crashes/

--
Agostino Sarubbo
Gentoo Linux Developer