Can't enable wal2json in postgres 12 docker image

I'm trying to enable wal2json in an image extended from postgres:12 image. But i'm getting an error:

postgres=# create extension wal2json;
ERROR:  could not open extension control file "/usr/share/postgresql/12/extension/wal2json.control": No such file or directory

This might be because wal2json is also missing from pg_available_extensions:

postgres=# select * from pg_available_extensions where name ilike '%%json%%';
 name | default_version | installed_version | comment 
------+-----------------+-------------------+---------
(0 rows)

Dockerfile:

FROM postgres:12
RUN apt update && apt install -y postgresql-12-wal2json postgresql-contrib

postgres server logs:

db_1  | 2020-05-10 13:53:04.611 GMT  LOG:  database system is ready to accept connections
db_1  | 2020-05-10 13:54:03.587 GMT  ERROR:  could not open extension control file "/usr/share/postgresql/12/extension/wal2json.control": No such file or directory
db_1  | 2020-05-10 13:54:03.587 GMT  STATEMENT:  create extension wal2json;

the postgresql.conf file has been tweaked to

wal_level = logical		
shared_preload_libraries = 'wal2json'

Any help with getting this working will be appreciated.