How can I secure unencrypted credential files, for programs that assume them (like gmi/lieer)?

### Brief Q: How can I cryptographically secure a credentials file that is stored on disk as plaintext? Or, rather: how can I avoid storing credentials like those for Gmail and other API keys on disk? For existing programs that assume such an unencrypted file containing secrets. I ask this question motivated by wanting to access Gmail using gmi/lieer and notmuch - which AFAICT use an unencrypted credentials file on disk. But there are lots of other programs that require similar credentials files. Surely there must already be a generic solution to this problem? Something like ssh-agent, that asks the user for a passphrase and then decrypts the secrets into memory for some time. But not necessarily as fancy as ssh-agent... the agent doesn't need to do all of the crypto operations, which might differ by application or API or protocol. IMHO just decrypting the credentials file into memory would be of value. TL;DR - You might be able to stop here without reading the rest --- Some people will understand what I'm asking for from the above BRIEF section. Others, probably not. ### Surely there must be a generic solution to this problem? Surely there is something like ssh-agent that reads such secrets from an encrypted file, asks the user for password (or better), decrypts the secrets, and keeps them only in memory for some time, so that you don't constantly have to reenter the password/etc? Doesn't have to go quite as far as ssh-agent, where the agent does all or most of the cryptographic operations - and hence the protocol between ssh client and ssh-agent is not just "give me the credential", but must also describe the operations to be performed. Since there are lots of different protocols that have lots of different credentials with lots of slightly different operations, there may be an obstacle to creating a custom agent for each right away. But simply having a persistent agent ask the user and then decrypt credentials from disk into memory would be an improvement over nothing at all. Surely this has already been done, in a manner that can work with lots of different apps XYZ? But I certainly don't know of anything like this. Nor, for that matter, do any AI assistants that I have tried - although it might be a question of me not phrasing the LLM prompt or Google search correctly. For that matter, ChatGPT suggested that I do the following: * encrypt the credentials file on disk * when I want to use it * temporarily create an unencrypted credentials file - on disk * let the client program like gmi/lieer access the unencrypted credential file while it is running * and when I no longer am running the client, delete the unencrypted temporary credentials file I hope I don't need to explain how unsatisfactory this is. ### Could this be done using UNIX domain sockets or FUSE? Has it been done already? If I knew that the client application was always reading or replacing the entire credential file, I could imagine having an XYZ-agent write the unencrypted secrets to the socket all at once. or if I do not know the access pattern, e.g. if the secret is large enough that seeks a random-access are performed, I could imagine that a user domain filesystem like FUSE could be used. Q: has anyone created such a generic "decrypt secrets into memory, so it looks like an unencrypted credentials file to software that cannot handle an encrypted credentials file?". * Using UNIX domain sockets * or FUSE * or whatever Even better if such as change to the namespace were limited to a partent process and its children, such as you might be able to do in OSes like Plan9 or Brazil, although AFAIK existing UNIXes like Linux do not make this easy to do. ### Details As is my wont, I provide way too much background detail for my question. For many people reasonably knowledgeable about security this much detail should not be necessary. But sometimes it may not be clear exactly what I am talking about. Sometimes I may be using incorrect terms. And so on. Hence, I provide all this extra detail hoping to short-circuit misunderstandings. If you truly know of an answer to my question, you can probably stop without reading all the rest. Heck, I might as well admit it: I'm trying to short-circuit stupid nonanswers to my question. But previous attempts to do this I'm not always been successful. ### Motivating Example: gmi/lieer access to gmail uses an unencrypted credential file E.g. lieer, a program to synchronize gmail with local storage, stores an unencrypted credentials file for Gmail in the filesystem. This file, .credentials.gmailieer.json, is completely unencrypted ordinary plaintext. > Excerpting: > gmi init will now open your browser and request limited access to your e-mail. > … > The access token is stored in .credentials.gmailieer.json in the local mail repository. If you wish, you can specify your own api key that should be used. Of course file system permissions should make it accessible only by my UNIX login id. It is used by the gmi/lieer program to access my gmail account. But unless I am totally missing something, any program running as me can access this file. E.g. one of the umpteen sandbox escapes in web browsers might allow it to access this file. Or I might have filesystem permissions set incorrectly. Or I might have misconfigured filesystem/disk encryption, and other user IDs on my machine may be able to access it. Etc. I thought that it was standard/best practice for security that plaintext secrets should never be stored on disk. I have long been somewhat surprised by how many software systems require credentials like API keys to be stored on disk. I have usually avoided using such systems, although it gets in the way of doing things like Google API development that require such API keys. Or I might use such systems or work, but resist using them for stuff that is personal. However, I really do want to use such systems for personal stuff. Not just personal software development, but for gmi/lieer access my Gmail account, which is about as personal as I can get, much more sensitive to me than a GitHub project. This is not just an issue with gmi/lieer. Many programs, many software systems, require you to store credentials like API keys on disk. I don't think I've encountered any of them that keep them encrypted on disk. Except, of course, for ssh/ssh-agent and gpg/gpg-agent, where the credential files are protected, not only protected by file system permissions, but also by a passphrase, and are decrypted only within the ssh-agent's process memory. #### ssh-agent => no plaintext credentials on disk Except, of course, for ssh/ssh-agent and gpg/gpg-agent. + Where the key files are protected + not only protected by file system permissions, but are also encrypted by a passphrase. * When you load an ID into ssh-agent * it asks you for the passphrase, * reads the encrypted key file(s), * and decrypts them into it's process memory. * ssh-agent is persistent, so you only have to do this once in a while * ssh, if configured appropriately, * won't be able to run without asking ssh-agent to "do stuff". * communicates yo ssh-agent via a UNIX domain socket * ssh-agent actually does all or most of the public key computations * => ssg itself does not have the private keys I thought that it was standard/best practice for security that 1. Plaintext secrets should never be stored on disk * with the possible exception of swap files, * but that should be a solved problem * so even if someboday can access the raw data on disk you should be safe * e.g. if you don't have full disk or filesystem encryption * and the disk drive is accessed outside of its "home" OS 2. Plaintext private keys may be stored in ssh-agent's process memory * not in the ssh client program * and should not be accessible by any other programs, * even running as the same user in the same machine * possibly also not by more privileged users like root or admin * with the possible exception of debuggers * but that also should be a solved problem (although not so much in my experience) #### Going beyond ssh-agent… Skip this section, It isn't really necessary for my question, except it helps me organize the issues in my mind. Also, if somebody can tell me that these items (3) and (4) below are in much wider use and I currently know, I'd love to hear about it. Items (1) and (2) are, AFAIK, the state of the art, or at least practice. But they leave some hardware/logic analyzer security holes vulnerable, which have been addressed by certain academic and industry projects, but which as far as I know are much less common: 3. In most present-day systems plaintext secrets may be stored in DRAM + unless the programmer has been very careful to keep them only in registers + and has control of context switches that might save the registers to memory + but various hardware memory encryption proposals and products prevent even this from happening + e.g. data may be stored unencrypted in cache, but may be encrypted between cache and DRAM. 4. and similarly various proposals and products ensure that all of the traffic on buses and connections etc. where you could attach a logic analyzer are encrypted. * 2.5: I'm actually a little bit uncomfortable that the ssh client/agent communication is done via a UNIX domain socket + AFAIK any process running with the appropriate user ID can access that socket, and can get the ssh-agent to do stuff + AFAIK the UNIX domain socket is protected only by filesystem permissions + AFAIK the ssh-agent and ssh program do not talk via an encrypted channel + Although the fact that the UNIX domain socket can be somewhat random reduces exposure. And I know that some operating systems - not standard LINUX, AFAIK - allow permissions to be restricted not just by user ID but also by executable ID, or position in the process tree. + You can of course use JNIX user IDs to accomplish this, but as far as I know this is not commonly done. #### Example: plaintext file containing gmail credentials for gmi/lieer credential gmi/lieer, a program to synchronize gmail with local storage, stores an unencrypted credentials file for Gmail in the filesystem > Excerpting: > gmi init will now open your browser and request limited access to your e-mail. > … > The access token is stored in .credentials.gmailieer.json in the local mail repository. If you wish, you can specify your own api key that should be used. The credential lieer stores looks like the below. I hope that I have edited out anything that is sensitive. "Gibberish" is of course what looks like random letters and numbers with occasional punctuation, the sort of stuff one associates with a credential. {"access_token": "xyzzy ~200 bytes of gibberish", "client_id": "~40 bytes of gibberish.apps.googleusercontent.com", "client_secret": "~20 bytes of gibberish", "refresh_token": "~100 bytes of gibberish", "token_expiry": "2024-09-15T07:16:24Z", "token_uri": "https://accounts.google.com/o/oauth2/token ", "user_agent": "Lieer", "revoke_uri": "https://oauth2.googleapis.com/revoke ", "id_token": null, "id_token_jwt": null, "token_response": {"access_token": "~200 bytes of gibberish", "expires_in": 3599, "scope": "https://www.googleapis.com/auth/gmail.readonly https://www.googleapis.com/auth/gmail.modify https://www.googleapis.com/auth/gmail.labels ", "token_type": "Bearer"}, "scopes": ["https://www.googleapis.com/auth/gmail.labels ", "https://www.googleapis.com/auth/gmail.readonly ", "https://www.googleapis.com/auth/gmail.modify "], "token_info_uri": "https://oauth2.googleapis.com/tokeninfo ", "invalid": false, "_class": "OAuth2Credentials", "_module": "oauth2client.client" } This is completely plaintext, although of course it is accessible only by my Linux user id. It is used by the gmi program to authenticate to gmail. If not present, I cannot acces my gmail. I don't get asked for my password, etc. Unless I am missing something, this credential could allow almost any program that can read this file to access my Gmail. this concerns me. It's not just gmi/lieer -- many programs. I'm not going to bother listing more examples. But just googling API KEY should yield a lot of them. ### Is it just obsolete legacy software? Possibly, but IMHO not completely. E.g. the gmi/lieer source code and/or documentation indicates that it using an old Gmail API, and should be upgraded. Possibly more recent APIs solve this problem - but not as far as I can tell. Possibly there is already a generic OpenAuth-agent - but not that I can find. AFAICT Google really prefers to keep the OpenAuth stuff in its own libraries, used by Google Chrome and other web browsers, and has not really done much to support command line or other non-browser utilities. They would really prefer that you did not use such utlities, unless Google wrote them. They only grudgingly support such utlities, allowing you to obtain API KEYs, etc. If there are security holes caused by storing such credentials unencrypted on disk, they will just use that as more evidence to justify locking things down, and locking other software out. Anyway: If there is a generic OpenAuth-agent (probably not called that) - I would love to hear about it. But anyway, furthermore: even if there is a generic OpenAuth-agent, there are a lot of existing programs that assume an unencrypted creditials file on disk. There would be value in having a generic solution fir these, until they can be upgraded. Assuming they can be.