Log every time users access a certain table in Postgres

Recently we changed our production database log_statement from 'all' to 'mod', because the resulting log file was too large for our available storage. Unfortunately, we still need to log every SELECT made by users to a specific table N for audit purposes. Is there any solution for that? I have tried using pgaudit and pg_stat_statements without any success. We have set pgaudit.log to read, but it logs every SELECT query instead of just selects to table N.