How to cross-compile Linux tools/perf for embedded system?

I am working on an embedded Linux system (kernel-5.19.20) on MIPS, and I want to build tools/perf for my system. I want to have libelf feature enabled when cross compile perf, so I firstly cross-compile and installed the libelf to $(proj)/sysroot/usr/lib/. Then I tried following command to cross compile perf.

CC=mips-none-gnu-gcc make ARCH=mips CROSS_COMPILE=mips-none-gnu- EXTRA_CFLAGS="-I/home/t/proj/target/usr/include" LDFLAGS="-L/home/t/proj/target/usr/lib -Wl,-rpath-link=/home/t/proj/target/usr/lib"

And I got following feature list.

Auto-detecting system features:
...                         dwarf: [ OFF ]
...            dwarf_getlocations: [ OFF ]
...                         glibc: [ on  ]
...                        libbfd: [ OFF ]
...                libbfd-buildid: [ OFF ]
...                        libcap: [ OFF ]
...                        libelf: [ OFF ]
...                       libnuma: [ OFF ]
...        numa_num_possible_cpus: [ OFF ]
...                       libperl: [ OFF ]
...                     libpython: [ OFF ]
...                     libcrypto: [ OFF ]
...                     libunwind: [ OFF ]
...            libdw-dwarf-unwind: [ OFF ]
...                          zlib: [ OFF ]
...                          lzma: [ OFF ]
...                     get_cpuid: [ OFF ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ OFF ]
...        disassembler-four-args: [ OFF ]

I checked the ../build/feature/test-libelf.make.output, and I got,

/home/t/proj/mips-none-gnu/bin/ld: warning: libz.so.1, needed by /home/t/proj/target/usr/lib/libelf.so, not found (try using -rpath or -rpath-link)
/home/t/proj/target/usr/lib/libelf.so: undefined reference to `inflate'
/home/t/proj/target/usr/lib/libelf.so: undefined reference to `deflate'
/home/t/proj/target/usr/lib/libelf.so: undefined reference to deflateInit_'    /home/t/proj/target/usr/lib/libelf.so: undefined reference to inflateEnd'
/home/t/proj/target/usr/lib/libelf.so: undefined reference to `deflateEnd'
/home/t/proj/target/usr/lib/libelf.so: undefined reference to inflateInit_'    /home/t/proj/target/usr/lib/libelf.so: undefined reference to inflateReset'    collect2: error: ld returned 1 exit status

The libz.so is built within buildroot and already installed into /home/t/proj/target/usr/lib. So, how to make perf with libelf succeed in this case?