You should be able to build minimad with just "make minimad". This will link with the library in the libmad subdirectory, which is usually more appropriate than the installed /usr/local/lib/libmad.a especially when cross-compiling.
I had a go at doing 'make minimad' like you suggested. For some reason it seems to be looking for files in tmp/ of my home directory.
/home/gar/tm/ccPMTjTj.o: In function 'decode': /home/gar/tm/ccPMTjTj.o: undefined reference to 'mad_decoder_init' /home/gar/tm/ccPMTjTj.o: relocation truncated to fit: R_ARM_PC24 mad_decoder_run
I get about 5 of these errors and then it exits with
collect2: ld returned 1 exit status
I had a look in tmp/ for the .o files it is refering to but they arn't there.