In Memory OLTP causing OOM Issue in SQL Server even though no data in In Memory OLTP Table

We have only 1 memory optimized table in each database and there are total 10 databases as such. This table is completely empty. But still we can see in DBCC MEMORYSTATUS Output that In memory OLTP is consuming 7.68 GB out of 13 GB that has been assigned to SQL Server. MEMORYCLERK_XTP (node 0) KB ---------------------------------------- ---------- VM Reserved 0 VM Committed 0 Locked Pages Allocated 0 SM Reserved 0 SM Committed 0 Pages Allocated 8055696 -- 7.68 GB Sometimes CHECKPOINT is also not running because of OOM situation. Other than restarting SQL Service, is there any other solution to resolve this issue? SQL Server Version: Microsoft SQL Server 2016 (SP2-CU12) (KB4536648) - 13.0.5698.0 (X64) Feb 15 2020 01:47:30 Copyright (c) Microsoft Corporation Enterprise Edition: Core-based Licensing (64-bit) on Windows Server 2012 R2 Standard 6.3 (Build 9600: ) (Hypervisor)