I'm looking for a simple, generic way to input arbitrary unicode characters in a text document on the terminal(e.g. in a terminal editor). A basic method I can imagine is having a simple text(utf-8) file containing two columns, the character's name or description, and the character itself. Then I can have a simple script to lookup a character through this file, e.g. using dmenu or similar. Are there similar methods available that would take care of this for me? Otherwise where can I find such mappings(names, utf-8 value) for common unicode characters(e.g. smileys, greek characters, mathematical symbols)? This seems like such a basic common need yet I'm having difficulty finding simple solutions readily available. Thanks!

So, I'm using Linux Mint 21.3 with MATE 1.26.0. I've noticed that my system supports a number of nonstandard flag emoji. I'm wondering what part of the system is responsible for this, if this is documented anywhere, and where the files or source for this can be found. Since I'm guessing not everyone is familiar: In addition to its mechanism for national flags, Unicode also has a mechanism for flags of subnational entities. However, this mechanism is mostly unused, with only the flags of England, Scotland, and Wales being officially supported according to the standard (the reason being, as I understand it, that the Unicode Consortium quickly realized that this would get completely out of hand if they allowed it to go any further). I've noticed that my computer actually supports a number of nonstandard subnational flags: American states and DC; Canadian provinces and territories; Mexican states and its federal district; and Northern Ireland (which officially doesn't even have a flag, but one is displayed regardless). Maybe there's more, but other subnational entities I thought to try (such as American territories or Australian states) didn't work. As an example, if your computer is like mine, this will display the flag of Delaware: 🏴󠁵󠁳󠁤󠁥󠁿 (my apologies to the Unicode Consortium for including a nonstandard emoji in general interchange! :P ) You might be wondering, why would I even think to try this, given that certainly I've never seen such nonstandard emoji in the wild, and had to assemble them myself out of the necessary non-printable code points (I mean with a script obviously, not by hand!), after inferring what those would be? The answer is that I got curious about the possibility after seeing [this MIT Mystery Hunt puzzle](https://puzzles.mit.edu/2024/mythstoryhunt.world/puzzles/crossed-fingers-memo-puzzle) the other year, which includes some US state flags as "emoji". Of course, the emoji in that puzzle are actually images, so there's no requirement for them to be actual emoji on anyone's system, but it got me wondering about the possibility, leading me to try out creating such nonstandard emoji on my own system, and, to my surprise, it worked. (I wonder if the writers of that puzzle were aware of this? Just realizing I didn't think to try contacting them... oh well.) (Edit: Actually, maybe I have seen this in the wild on one occasion? I forget, my memory's a bit fuzzy here. It's not important to the question anyway.) Anyway yeah this has been bugging me and I'm wondering if anyone knows what is responsible for this and where I could find more info. Thanks all!

On the Fedora 23 system I upgraded from F22, the U1F32D symbol 🌭 shows up just fine in the terminal, but on the one I installed from scratch, I get the box-with-numbers placeholder. I just checked and I have about 60 font packages on the system where it doesn't work, and over 200 where it does. Without inspecting each one manually, is there a way to identify which font I need to add?

I want to use pdfgrep and it works. When I want to search for an Arabic text or string, it shows nothing. However, it works properly when I search for an English string. Does anyone have a solution or even an alternative? This is the code I used:

-sh
pdfgrep -in 'احمد' name.pdf

Do gibberish characters found in EFI variables serve any purpose? Out of curiosity, i am trying to read out EFI variables. Specifically ones related to the booting mechanism. Under /sys/firmware/efi/efivars one can find boot entries such as Boot000, Boot001, etc. However, opening one up with VIM, and decoding it with UCS-2le (i'm assuming the prescribed encoding for (U)EFI), mostly gives me gibberish characters. Not just that, mind you. It *does* give me the location of bootmgfw.efi, and other data such as the label for the boot entry, for example. But it is also full of gibberish characters (in this specific context). A few examples:

^@^@^A^@
痧駢疠

Anyone have any ideas?

In Unicode, some character combinations have more than one representation. For example, the character *ä* can be represented as * "ä", that is the codepoint U+00E4 (two bytes c3 a4 in UTF-8 encoding), or as * "ä", that is the two codepoints U+0061 U+0308 (three bytes 61 cc 88 in UTF-8). According to the Unicode standard, the two representations are equivalent but in different "normalization forms", see UAX #15: Unicode Normalization Forms . The unix toolbox has all kinds of text transformation tools, *sed*, *tr*, *iconv*, Perl come to mind. How can I do quick and easy NF conversion on the command-line?

gvim doesn't seem to support it out of the box on my Ubuntu lucid.

Is it advisable to have or not Byte Order Mark (BOM) in **UTF-8 text** files on Linux? Is it correct to say byte order (even for multi-byte characters) is already strictly defined/fixed in UTF-8 standard? Would you say Linux soft & CLI utilities are generally more compatible with UTF-8 (w/o BOM) or with UTF-8 BOM? Any examples of incompatibility? Example soft I primary use: Doublecmd, terminal, Libreoffice, bash (scripts),Kate. I'm doing encoding conversion of many non-ASCII text files to **UTF-8**, so need to consciously decide whether to choose UTF-8 or UTF-8 BOM. Text files include: HTML, Bash scripts, C/C++ code, TeX etc. Thank you!

In Perl, /a/i matches both A and a, so I don't have to write /A|a/. What is the easy way to write /４|4/ ? Yes, I'm talking about

lang-sh
$ unicode ４ 4|grep U+
U+FF14 FULLWIDTH DIGIT FOUR
U+0034 DIGIT FOUR

Must I repeat each line,

if (/大茅埔段３２(７|８|９).地/ ||
        /大茅埔段32(7|8|9).地/)     {...}

or is there a better way? It's not that many characters, https://en.wikipedia.org/wiki/Halfwidth_and_Fullwidth_Forms_(Unicode_block) By the way, how to make

$ echo Ａ|perl -pwle 's/ａ/x/i'

match? (Both A's are wide.)

I have problem converting a file to PDF. I create the file with echo ęąśćżńł > text and convert: enscript -O text -o - | ps2pdf - out.pdf However, out.pdf has an encoding problem:

I'm running a simple PKI web application in Php that uses OpenSSL shell commands and stores information in a text database. I have to deal with non-ASCII input characters (eg. German) but when a new cert is created, the fields turn into [...] /C=DE/ST=H\xC3\xA4mburg/L=H\xC3\xA4mburg/O=\xC3\x9FBCD/OU=\xC3\xA4BC/ [...] I've added [req] utf8 = yes string_mask = utf8only name_opt = multiline,-esc_msb,utf8 to the OpenSSL config files, and the certs are created using openssl req -utf8 [...] openssl ca -utf8 [...] The program writes out temporary config files used to request and create the user cert. I've checked and the fields in the user config file are fine, as well as in the CA's. The problem appears when OpenSSL creates the certificates. The database and the certs contain these codes instead of the correct characters. I also tried to convert the database file to UTF-8 using iconv but the file remains in US-ASCII format. This approach changes what file --mime-encoding outputs to utf-8, but OpenSSL continues to write new entries the same way as before. I'll also mention I'm sending the utf-8 header and setting AddDefaultCharset utf-8 in virtualhosts. The locale is set to de_DE.utf8 in the code and on the server. What am I missing? Any help is appreciated.

My input data is as follows (as generated by hexdump): 000000f0 69 61 6e e2 80 99 73 20 65 79 65 73 20 61 62 72 |ian...s eyes abr| When I open this html () file in Firefox, it displays these characters as: ian’s eyes abr According to the link https://superuser.com/questions/1237545/characters-in-email-displayed-like-e2-80-99 , "E2 80 99 is the sequence of hex values that encode a right single quotation mark (’) in UTF-8". This website concurs: https://www.utf8-chartable.de/unicode-utf8-table.pl?start=8192&number=128 When I run this iconv command on the file containing these characters: iconv -f UTF-8 -t ISO-8859-15 test_chapter.html > blah.html I get the output: iconv: illegal input sequence at position 243 and the content of "blah.html" is truncated exactly where the apostrophe would be. So, to summarise, the internet says that is a valid sequence of bytes for UTF-8, but iconv disagrees. Can anyone please help me understand what is going on. Is this a bug in iconv? As a side note, when I use this html file with kindlegen to generate an AZW file, the character is not displayed correctly. All the internet can tell me is that I need to convert the file to UTF-8, but as far as I can tell, it already is!

Everything on my system (that needs it) supports UTF-8 just fine.
That's all nice when you want output... But what if you want easy **in**put ? At the moment the only non-ASCII chars I can easily type are chars like é by using AtlGr.
But for chars like ₂ ² ≈ √ π 😀 at the moment I have to: 1. Open a browser 2. Surf to https://www.utf8icons.com or a similar site 3. Click, type and search a lot on the site to get to a page that contains the symbol i want 4. Copy it 5. Paste it in the program where I need it 6. (Optionally) close the browser What I'm looking for is a program that can do something like this: - Run in the background in a modern desktop environment (in my case Cinnamon) - Jump to the foreground to show a whole list of reasonably popular UTF-8 symbols after pressing something like F1 - Let me click a symbol after which it will be sent to the program I was last using as if it was a keypress - Give me the option to configure it to either stay visible after this "fake keypress" or jump back to background In short: Are there virtual keyboard programs with support for non-ASCII UTF-8 ? Actually... I am already happy with *any* method that improves mine. **Edit:** *For others ending up here and don't want to read all the answers themselves (or add a answer that's already given):
These are the options already mentioned + links to the answers + pro's and contra's.
Feel free to add extra solutions below (after providing them as detailed answer)*: - ibus (usually with CtrlShiftE) → Can't get it to work on Cinnamon - onboard → *pro*: Seems to do everything I need + has support for snippets, *con*: Only (by default) included non-latin layout is for math, other layouts with popular UTF-8 chars have to be created manually - gucharmap → *pro:* Lots of chars and easy to search *con:* Doesn't easily jump between foreground/background (can probably be handled with a workaround in Cinnamon itself) - kcharselect→ Same pro/con as gucharmap - Solutions from the programs themselves (e.g. Ctrl. for a couple of them) → *pro*: Ideal for that exact program *con*: Most programs, including the ones where it's needed the most, don't have one + it's not uniform - https://www.unicodeit.net/ → *pro*: Good for long math formula's. *con*: Same problem as the one I originally stated + useless for non-math symbols - Keyboard with extra symbols → *pro*: Easy *con*: Small amount of chars + extra keyboard needed for each system - Shortcuts for the most used chars with xcompose → *pro*: Easy *con*: Depending on your memory (as human, not as computer) it only works for a limited amount of chars - HTML entities to compose - *pro/con*: Too much of each, see answer - Use CtrlShiftU, Hexcode,Space : *pro/con*: Same as above

i have never been able to get my terminal to display unicode symbols. for example, before i had my present os, i mapped ctrl+a to the greek mu in vim, and it works on other computers, but not on my current xterm. here is the relevant section of my .vimrc: set encoding=utf-8 "map control-a to mu imap m* also, i need to output sympy equations in python, and this works on other computers, but not on my current xterm. instead of this: $ python Python 2.7.3 (default, Mar 14 2014, 11:57:14) [GCC 4.7.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import sympy >>> x = sympy.symbols('x') >>> sympy.init_printing() >>> (sympy.sqrt(x**3/(x+1)), 1) ⎛ _______ ⎞ ⎜ ╱ 3 ⎟ ⎜ ╱ x ⎟ ⎜ ╱ ───── , 1⎟ ⎝╲╱ x + 1 ⎠ i get this: >>> (sympy.sqrt(x**3/(x+1)), 1) n ------- n n n 3 n n n x n n n ───── , 1n nnnn x + 1 n infact it seems to just use the n character whenever it can't display a unicode character. i'm running xterm from an ~/.xinitrc file and setting some fonts and colors for the terminal in ~/.Xresources. here is all the relevant information i could think of: $ uname -a Linux mypcname 3.2.0-4-amd64 #1 SMP Debian 3.2.60-1+deb7u3 x86_64 GNU/Linux $ xterm -version XTerm(278) $ cat ~/.xinitrc #!/bin/bash #update the xterm colors, font size, etc [[ -f ~/.Xresources ]] && xrdb -merge ~/.Xresources # run the window manager in the background first metacity & # get the window manager process id wm_pid=$! # wait a little while for the window manager to load (extend this if the xterm is not being properly maximised) sleep 2 # run the xterm in fullscreen #xterm +u8 -js -fullscreen & xterm -en en_AU.UTF-8 -js -fullscreen & # do not let the window manager become a zombie wait $wm_pid # this would run xterm first, then the window manager. doesn't maximise properly the first time startx is run #xterm -fullscreen & #exec mutter $ cat ~/.Xresources ! see man xterm under the resources heading for explanations ! run xrdb -merge ~/.Xresources after altering this file ! run xrdb -query -all to see the current settings xterm.vt100.faceName: Terminus xterm.vt100.faceSize: 14 ! do not display bold fonts in bold xterm.vt100.AllowBoldFonts: false ! display bold fonts in a different color to make them stand out xterm.vt100.colorBDMode: true ! use green as the bold color (same as in ~/.bashrc) xterm.vt100.colorBD: #98E34D ! cols x lines ... update with values from $(echo $COLUMNS) and $(echo $LINES) xterm.vt100.geometry: 126x52 ! dark green foreground (same as in ~/.bashrc) *foreground: #4E9A06 ! black background *background: #000000 ! scroll quickly xterm*fastScroll: true ! enable utf-8 encoding xterm*locale: true xterm*utf8: 1 ! flash the current line instead of making the bell sound *visualBell: true *visualBellLine: true ! black *color0: #2E3436 ! darkred *color1: #CC0000 ! dark green *color2: #4E9A06 ! brown *color3: #C4A000 ! darkblue *color4: #3465A4 ! darkmagenta *color5: #75507B ! darkcyan *color6: #06989A ! lightgrey *color7: #D3D7CF ! darkgrey *color8: #555753 ! red *color9: #EF2929 ! green *colorA: #8AE234 ! yellow *colorB: #FCE94F ! blue *colorC: #729FCF ! magenta *colorD: #AD7FA8 ! cyan *colorE: #34E2E2 ! white *colorF: #EEEEEC $ tail -10 .bashrc PATH=/usr/local/bin:/usr/bin:/bin:/sbin export LC_ALL=en_AU.UTF-8 export LANG=en_AU.UTF-8 export LANGUAGE=en_AU.UTF-8 # final logon actions: # go straight to x on login. only do this for tty1 so that we can still use the other tty consoles without starting x. also only do this when there is not already a display, otherwise the xterm will try and do this after x starts aswell [[ -z $DISPLAY ]] && [[ $(tty) = /dev/tty1 ]] && startx $ locale LANG=en_AU.UTF-8 LANGUAGE=en_AU.UTF-8 LC_CTYPE="en_AU.UTF-8" LC_NUMERIC="en_AU.UTF-8" LC_TIME="en_AU.UTF-8" LC_COLLATE="en_AU.UTF-8" LC_MONETARY="en_AU.UTF-8" LC_MESSAGES="en_AU.UTF-8" LC_PAPER="en_AU.UTF-8" LC_NAME="en_AU.UTF-8" LC_ADDRESS="en_AU.UTF-8" LC_TELEPHONE="en_AU.UTF-8" LC_MEASUREMENT="en_AU.UTF-8" LC_IDENTIFICATION="en_AU.UTF-8" LC_ALL=en_AU.UTF-8 $ printenv XTERM_LOCALE en_AU.UTF-8 $ xrdb -query -all *background: #000000 *color0: #2E3436 *color1: #CC0000 *color2: #4E9A06 *color3: #C4A000 *color4: #3465A4 *color5: #75507B *color6: #06989A *color7: #D3D7CF *color8: #555753 *color9: #EF2929 *colorA: #8AE234 *colorB: #FCE94F *colorC: #729FCF *colorD: #AD7FA8 *colorE: #34E2E2 *colorF: #EEEEEC *foreground: #4E9A06 *visualBell: true *visualBellLine: true xterm*fastScroll: true xterm*locale: true xterm*utf8: 1 xterm.vt100.AllowBoldFonts: false xterm.vt100.colorBD: #98E34D xterm.vt100.colorBDMode: true xterm.vt100.faceName: Terminus xterm.vt100.faceSize: 14 xterm.vt100.geometry: 126x52 **how can i get utf-8 working to display greek symbols in vim and equations in sympy?** *** extra information requested $ echo $TERM xterm $ appres XTerm *form.Thickness: 0 *tekMenu*tekreset*Label: RESET *tekMenu*tektext2*Label: #2 Size Characters *tekMenu*tekhide*Label: Hide Tek Window *tekMenu*tekcopy*Label: COPY *tekMenu*tektext3*Label: #3 Size Characters *tekMenu*vtshow*Label: Show VT Window *tekMenu*tektextsmall*Label: Small Characters *tekMenu*vtmode*Label: Switch to VT Mode *tekMenu*tektextlarge*Label: Large Characters *tekMenu*tekpage*Label: PAGE *tekMenu.Label: Tek Options *mainMenu*redraw*Label: Redraw Window *mainMenu*sunKeyboard*Label: VT220 Keyboard *mainMenu*terminate*Label: Send TERM Signal *mainMenu*backarrow key*Label: Backarrow Key (BS/DEL) *mainMenu*logging*Label: Log to File *mainMenu*hpFunctionKeys*Label: HP Function-Keys *mainMenu*kill*Label: Send KILL Signal *mainMenu*num-lock*Label: Alt/NumLock Modifiers *mainMenu*print-immediate*Label: Print-All Immediately *mainMenu*scoFunctionKeys*Label: SCO Function-Keys *mainMenu*quit*Label: Quit *mainMenu*alt-esc*Label: Alt Sends Escape *mainMenu*print-on-error*Label: Print-All on Error *mainMenu*tcapFunctionKeys*Label: Termcap Function-Keys *mainMenu*meta-esc*Label: Meta Sends Escape *mainMenu*toolbar*Label: Toolbar *mainMenu*print*Label: Print Window *mainMenu*suspend*Label: Send STOP Signal *mainMenu*delete-is-del*Label: Delete is DEL *mainMenu*print-redir*Label: Redirect to Printer *mainMenu*fullscreen*Label: Full Screen *mainMenu*continue*Label: Send CONT Signal *mainMenu*oldFunctionKeys*Label: Old Function-Keys *mainMenu*securekbd*Label: Secure Keyboard *mainMenu*interrupt*Label: Send INT Signal *mainMenu*8-bit control*Label: 8-Bit Controls *mainMenu*allowsends*Label: Allow SendEvents *mainMenu*sunFunctionKeys*Label: Sun Function-Keys *mainMenu*hangup*Label: Send HUP Signal *mainMenu.Label: Main Options *VT100.utf8Fonts.font4: -misc-fixed-medium-r-normal--13-120-75-75-c-80-iso10646-1 *VT100.utf8Fonts.font2: -misc-fixed-medium-r-normal--8-80-75-75-c-50-iso10646-1 *VT100.utf8Fonts.font6: -misc-fixed-medium-r-normal--20-200-75-75-c-100-iso10646-1 *VT100.utf8Fonts.font5: -misc-fixed-medium-r-normal--18-120-100-100-c-90-iso10646-1 *VT100.utf8Fonts.font3: -misc-fixed-medium-r-normal--14-130-75-75-c-70-iso10646-1 *VT100.utf8Fonts.font: -misc-fixed-medium-r-semicondensed--13-120-75-75-c-60-iso10646-1 *VT100.font4: 7x13 *VT100.font2: 5x7 *VT100.font6: 10x20 *VT100.font5: 9x15 *VT100.font3: 6x10 *VT100.font1: nil2 *vtMenu*selectToClipboard*Label: Select to Clipboard *vtMenu*reversewrap*Label: Enable Reverse Wraparound *vtMenu*softreset*Label: Do Soft Reset *vtMenu*cursesemul*Label: Enable Curses Emulation *vtMenu*autolinefeed*Label: Enable Auto Linefeed *vtMenu*hardreset*Label: Do Full Reset *vtMenu*visualbell*Label: Enable Visual Bell *vtMenu*appcursor*Label: Enable Application Cursor Keys *vtMenu*clearsavedlines*Label: Reset and Clear Saved Lines *vtMenu*bellIsUrgent*Label: Enable Bell Urgency *vtMenu*appkeypad*Label: Enable Application Keypad *vtMenu*tekshow*Label: Show Tek Window *vtMenu*poponbell*Label: Enable Pop on Bell *vtMenu*scrollbar*Label: Enable Scrollbar *vtMenu*scrollkey*Label: Scroll to Bottom on Key Press *vtMenu*tekmode*Label: Switch to Tek Mode *vtMenu*scrollttyoutput*Label: Scroll to Bottom on Tty Output *vtMenu*jumpscroll*Label: Enable Jump Scroll *vtMenu*cursorblink*Label: Enable Blinking Cursor *vtMenu*vthide*Label: Hide VT Window *vtMenu*allow132*Label: Allow 80/132 Column Switching *vtMenu*reversevideo*Label: Enable Reverse Video *vtMenu*titeInhibit*Label: Enable Alternate Screen Switching *vtMenu*altscreen*Label: Show Alternate Screen *vtMenu*keepSelection*Label: Keep Selection *vtMenu*autowrap*Label: Enable Auto Wraparound *vtMenu*activeicon*Label: Enable Active Icon *vtMenu.Label: VT Options *SimpleMenu*menuLabel.font: -adobe-helvetica-bold-r-normal--*-120-*-*-*-*-iso8859-* *SimpleMenu*menuLabel.vertSpace: 100 *SimpleMenu*Sme.height: 16 *SimpleMenu*BackingStore: NotUseful *SimpleMenu*HorizontalMargins: 16 *SimpleMenu*Cursor: left_ptr *SimpleMenu*borderWidth: 2 *menubar.borderWidth: 0 *tek4014*fontLarge: 9x15 *tek4014*font2: 8x13 *tek4014*font3: 6x13 *tek4014*fontSmall: 6x10 *MenuButton*borderWidth: 0 *fontMenu*render-font*Label: TrueType Fonts *fontMenu*fontdefault*Label: Default *fontMenu*font6*Label: Huge *fontMenu*allow-window-ops*Label: Allow Window Ops *fontMenu*utf8-mode*Label: UTF-8 Encoding *fontMenu*font1*Label: Unreadable *fontMenu*fontescape*Label: Escape Sequence *fontMenu*utf8-fonts*Label: UTF-8 Fonts *fontMenu*fontsel*Label: Selection *fontMenu*allow-bold-fonts*Label: Bold Fonts *fontMenu*utf8-title*Label: UTF-8 Titles *fontMenu*font-linedrawing*Label: Line-Drawing Characters *fontMenu*font2*Label: Tiny *fontMenu*allow-color-ops*Label: Allow Color Ops *fontMenu*font-doublesize*Label: Doublesized Characters *fontMenu*font3*Label: Small *fontMenu*allow-font-ops*Label: Allow Font Ops *fontMenu*font-loadable*Label: VT220 Soft Fonts *fontMenu*font4*Label: Medium *fontMenu*allow-tcap-ops*Label: Allow Termcap Ops *fontMenu*font-packed*Label: Packed Font *fontMenu*font5*Label: Large *fontMenu*allow-title-ops*Label: Allow Title Ops *fontMenu.Label: VT Fonts *colorD: #AD7FA8 *color5: #75507B *backarrowKeyIsErase: true *colorE: #34E2E2 *color6: #06989A *ptyInitialErase: true *colorF: #EEEEEC *background: #000000 *color7: #D3D7CF *saveLines: 1024 *color8: #555753 *color0: #2E3436 *foreground: #4E9A06 *IconFont: nil2 *color9: #EF2929 *color1: #CC0000 *visualBell: true *colorA: #8AE234 *color2: #4E9A06 *visualBellLine: true *colorB: #FCE94F *color3: #C4A000 *colorC: #729FCF *color4: #3465A4 $ xterm -u8 -fa "DejaVu Sans Mono" # the following is typed in the resulting terminal: $ echo -e "\xE2\x98\xA0" n # however when i copy the result from echo -e "\xE2\x98\xA0" # into my browser, i get this: ☠ (a skull) but it does not show # up as a skull in my xterm $ lsof -p $PPID | grep fonts xterm 5990 me mem REG 254,1 4971 13501810 /usr/share/fonts/X11/misc/ter-u18b_iso-8859-1.pcf.gz xterm 5990 me mem REG 254,1 4897 13505403 /usr/share/fonts/X11/misc/ter-u18n_iso-8859-1.pcf.gz i also ran $ fc-list but the output was too large to paste into this question. so i have put it here what it shows in my browser: /usr/share/fonts/truetype/freefont/FreeSansBold.ttf: FreeSans:style=Bold,получерен,negreta,tučné,fed,Fett,Έντονα,Negrita,Lihavoitu,Gras,Félkövér,Grassetto,Vet,Halvfet,Pogrubiony,Negrito,gros,Полужирный,Fet,Kalın,huruf tebal,жирний,Krepko,treknraksts,pusjuodis,đậm,Lodia,धृष्ट what i see in my terminal: /usr/share/fonts/truetype/freefont/FreeSansBold.ttf: FreeSans:style=Bold,nnnnnnnnn,negreta,tunné,fed,Fett,nnnnnn,Negrita,Lihavoitu,Gras,Félkövér,Grassetto,Vet,Halvfet,Pogrubiony,Negrito,gros,nnnnnnnnnn,Fet,Kalın,huruf tebal,nnnnnn,Krepko,treknraksts,pusjuodis,nậm,Lodia,nnn interestingly, some "special" characters do show up in my terminal, but most are relaced by n. you can see in the previous output that none of получерен can be displayed, but the final character of tučné can be displayed (while the middle č cannot - it is replaced by n) *** as per @apaul's comments it seems that xterm isn't loading the right font. try to set a dummy class so it doesn't load the xterm resources: $ xterm -class Foo -name foo -u8 -fa "DejaVu Sans Mono:style=Book" $ # the following commands are all executed in the resulting terminal: $ echo -e "\xE2\x98\xA0" ☠ $ # the above skull actually shows up now. and so does the unicode $ # output from sympy and also vi can display greek symbols now :) all that remains is to figure out why xterm cannot set the font using ~/.Xresoureces, and to get this working. it seems like something must be overriding the font settings? actually i just thought to try above command with the terminus font, and it seems that this is the problem: $ xterm -class Foo -name foo -u8 -fa "Terminus" $ # the following commands are all executed in the resulting terminal: $ echo -e "\xE2\x98\xA0" n maybe terminus is not properly installed? or is being mapped to something else. how could i find that out?

This question is closely related to: https://unix.stackexchange.com/questions/99350/how-to-insert-text-before-the-first-line-of-a-file . I deliberately made the title similar to that question to highlight this. Except the target file is UTF-8 with BOM. So, I want to add a first line to a file that has UTF-8 BOM bytes at its heading (0xef 0xbb 0xbf characters). At least the UTF-8 with BOM files I have here begin with this. If I just go ahead and follow the solutions in the related question, sed "1i My First line is now this." file.txt I will get (in VSCode in my case) something like My First line is now this. ?The first line was this one Second line and so on Being ? in the second line the UTF-8 character to express something non-printable. Another consequence, as one would expect, is the file no longer opens as UTF-8 with BOM and we rely on the text editor features to "guess" its encoding now. We have determined a pattern in our project to have files with BOM to ensure everything is in the same encoding. How do I preserve the BOM header in the file while adding the text?

I have two computers installed slightly differently : - A: KUbuntu based on 22.04.3 LTS - B: Ubuntu 24.04.1 LTS + KDE somehow added after I noticed between the 2 that some (not all) Unicode chars where displayed differently when appearing inside JetBrains Intellij Idea (either in source files or console tool window). I suspected IJ to be the source of the problem until i noticed that differences were beyond that. Let's consider two chars that behave differently: - U+2705 "White Heavy Check Mark" ✅ (should be displayed as a white check inside a green square) - U+1F9E1 "Orange heart" 🧡 (displayed in orange...) On computer A, no issue. Both chars are displayed with green/orange colors everywhere (idea, zsh/bash console, sublime text document, location bar or file names in Dolphin). On computer B, - orange heart is - a correct orange heart in shells, sublime or IJ - an empty square inside dolphin - the check is - a green check in shells - a bad looking green check on a white background in IJ, or dolphin filenames when created from outside dolphin - an empty square inside dolphin location bar, or filenames when created from dolphin Maybe it's basic and a simple installation could fix it but i don't know a lot in this area and i'm confused as on a given computer, the same char can be displayed differently. To be noted that both IJs are configure to use the internal font JetBrains Mono. When using "Font management" and changing the displayed sample text, none of the fonts is able to display the colors (and only one in black&white...), on both computers :( So how can i find which font is used to display which character so that i can install what's missing, or change whatever is needed to solve the issue ? Thanks, P.

I cannot get Unicode glyphs to render on Monjaro i3. I have installed a bunch of noto fonts, including noto-fonts, noto-fonts-cjk, noto-fonts-emoji, noto-fonts-extra, ttf-noto-nerd. When I run fc-match with an unrecognised Unicode character, it supposedly matches the correct font. This is for U+F312:

fc-match ""                                                                                                                                                                                      
NotoSans-Regular.ttf: "Noto Sans" "Regular"

I have run fc-cache -fv and rebooted multiple times.

fc-cache -fv
Font directories:
	/home/aidanb/.local/share/fonts
	/usr/local/share/fonts
	/usr/share/fonts
	/var/lib/snapd/desktop/fonts
	/home/aidanb/.fonts
	/usr/share/fonts/OTF
	/usr/share/fonts/TTF
	/usr/share/fonts/adobe-source-code-pro
	/usr/share/fonts/cantarell
	/usr/share/fonts/droid
	/usr/share/fonts/encodings
	/usr/share/fonts/gsfonts
	/usr/share/fonts/liberation
	/usr/share/fonts/misc
	/usr/share/fonts/noto
	/usr/share/fonts/noto-cjk
	/usr/share/fonts/encodings/large
/home/aidanb/.local/share/fonts: caching, new cache contents: 1 fonts, 0 dirs
/usr/local/share/fonts: skipping, no such directory
/usr/share/fonts: caching, new cache contents: 0 fonts, 11 dirs
/usr/share/fonts/OTF: caching, new cache contents: 23 fonts, 0 dirs
/usr/share/fonts/TTF: caching, new cache contents: 554 fonts, 0 dirs
/usr/share/fonts/adobe-source-code-pro: caching, new cache contents: 30 fonts, 0 dirs
/usr/share/fonts/cantarell: caching, new cache contents: 6 fonts, 0 dirs
/usr/share/fonts/droid: caching, new cache contents: 27 fonts, 0 dirs
/usr/share/fonts/encodings: caching, new cache contents: 0 fonts, 1 dirs
/usr/share/fonts/encodings/large: caching, new cache contents: 0 fonts, 0 dirs
/usr/share/fonts/gsfonts: caching, new cache contents: 35 fonts, 0 dirs
/usr/share/fonts/liberation: caching, new cache contents: 12 fonts, 0 dirs
/usr/share/fonts/misc: caching, new cache contents: 125 fonts, 0 dirs
/usr/share/fonts/noto: caching, new cache contents: 2137 fonts, 0 dirs
/usr/share/fonts/noto-cjk: caching, new cache contents: 80 fonts, 0 dirs
/var/lib/snapd/desktop/fonts: skipping, no such directory
/home/aidanb/.fonts: skipping, no such directory
/usr/share/fonts/OTF: skipping, looped directory detected
/usr/share/fonts/TTF: skipping, looped directory detected
/usr/share/fonts/adobe-source-code-pro: skipping, looped directory detected
/usr/share/fonts/cantarell: skipping, looped directory detected
/usr/share/fonts/droid: skipping, looped directory detected
/usr/share/fonts/encodings: skipping, looped directory detected
/usr/share/fonts/gsfonts: skipping, looped directory detected
/usr/share/fonts/liberation: skipping, looped directory detected
/usr/share/fonts/misc: skipping, looped directory detected
/usr/share/fonts/noto: skipping, looped directory detected
/usr/share/fonts/noto-cjk: skipping, looped directory detected
/usr/share/fonts/encodings/large: skipping, looped directory detected
/var/cache/fontconfig: not cleaning unwritable cache directory
/home/aidanb/.cache/fontconfig: cleaning cache directory
/home/aidanb/.fontconfig: cleaning cache directory
fc-cache: succeeded

Here's my current version:

cat /etc/lsb-release
DISTRIB_ID="ManjaroLinux"
DISTRIB_RELEASE="24.1.2"
DISTRIB_CODENAME="Xahea"
DISTRIB_DESCRIPTION="Manjaro Linux"

Please help! Thank you.

### The context Today, I wanted to keep the unique lines in a file containing Chinese characters. I decided to use the sort utility because I'm familiar with that tool and deleting repeated lines in a file is as easy as using the -u flag. I learnt that I needed to change the locale to make sort work correctly with Chinese characters. I noticed that using different locales made sort have different behaviors. In this post, I showed what I found. I'm aware that the task of deleting repeated lines in a file can be done using multiple tools/programming languages. While I thank anyone that suggests a tool for doing that task, I'm more interested in knowing more about locales and how they affect Unix utilities. ### Issues found #### Issue no. 1 The following is the locale of my system.

locale

LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

Consider the following file named main.txt

䔍
䏝

If I try to sort it using en_US.UTF-8 as my $LANG. I lose line no. 2

sort -u main.txt

䔍

I have solved this issue by setting LANG to zh_CN.UTF-8.

export LANG=zh_CN.UTF-8
sort -u main.txt

䔍
䏝

I created [a question](https://stackoverflow.com/questions/74295249) about this issue in Stack Overflow. Someone suggested me changing the locale and it seemed to work, I thought I had solved the problem, but then I found Issue no. 2 and Issue no. 3 (described below). #### Issue no. 2 Now, let's suppose that we add two lines to our file main.txt

䔍
䏝
𠵇
𠳐

If I execute sort -u on it while having LANG equal to zh_CN.UTF-8, I lose line no. 4.

export LANG=zh_CN.UTF-8
sort -u main.txt

䔍
𠵇
䏝

I tried different locales that started with zh.CN since the characters are Chinese characters. The uncommented locales are the ones that I tried using.

grep '^#\?zh' /etc/locale.gen

zh_CN.GB18030 GB18030  
zh_CN.GBK GBK  
zh_CN.UTF-8 UTF-8  
zh_CN GB2312  
#zh_HK.UTF-8 UTF-8  
#zh_HK BIG5-HKSCS  
#zh_SG.UTF-8 UTF-8  
#zh_SG.GBK GBK  
#zh_SG GB2312  
#zh_TW.EUC-TW EUC-TW  
#zh_TW.UTF-8 UTF-8  
#zh_TW BIG5

I noticed that sort -u works as expected when setting LANG set to zh_CN.GBK or zh_CN.GB18030. I solved this issue by using either zh_CN.GBK or zh_CN.GB18030. Here's the proof that it works as expected when setting LANG to zh_CN.GBK.

export LANG=zh_CN.GBK
sort -u main.txt

䔍
𠵇
𠳐
䏝

Here's the proof that it works as expected when setting LANG to zh_CN.GB18030.

export LANG=zh_CN.GB18030
sort -u main.txt

䔍
𠵇
𠳐
䏝

#### Issue no. 3 Let's suppose that we add two more lines to our file main.txt

䔍
䏝
𠵇
𠳐
书
乙

If I use zh_CN.GB18030 or zh_CN.GBK, I lose line no. 6.

export LANG=zh_CN.GB18030
sort -u main.txt

书
䔍
𠵇
𠳐
䏝

export LANG=zh_CN.GBK
sort -u main.txt

书
䔍
𠵇
𠳐
䏝

### The question Which value for $LANG should I use for making sort actually keep the unique lines in the following file?

䔍
䏝
𠵇
𠳐
书
乙

I previously showed that using zh_CN.GB18030, zh_CN.GBK, zh_CN.UTF-8 or zh_CN doesn't seem to work.

I am looking for a cross platform way to check if I am using a terminal emulator (with support for unicode characters) or a TTY session (with only support for ASCII chars). I initially tried to use if [[ $(tty) =~ /dev/tty ]]. This worked well on Linux; I was able to distinguish between the TTY session and a terminal emulator. The issue arises with Mac. All Mac terminal emulators return /dev/tty... from the tty command. I then saw suggestions to use $WINDOWID, $DISPLAY and $TERM_PROGRAM. The issue then becomes that not all terminal emulators set these variables. So my question is this: Is there a surefire way to reliably tell if you are using a terminal emulator or a TTY session? I know that i could theoretically create an if statement that checks for all of those but I am not a big fan of doing that. Is there a cleaner way to do this? (This is for importing my prompt to use Nerd fonts or not)

I sometimes need to type "alt codes" to get symbols and on linux, it can be inefficient to use shift+ctrl+u then type the code, for example shift+ctrl+u+00a7 for § when on Windows it's alt+21. So what I'm asking is, is there any way to basically use alt+... for these symbols as I don't really like using shift+ctrl+u and it doesn't work in some places. I use xfce4 as my desktop environment and under the keyboard settings, I've seen something called a "compose key." As I didn't know what this was, I looked it up and saw it kind of had something to do with these unicode characters. I've also seen somethings along the lines of level 1, 2, 3 and 4 keys but I don't really understand how these work, so if someone could explain these as well, that would be great. (I use arch (manjaro more specifically) if this helps)