Skip to content

Instantly share code, notes, and snippets.

@cfillion

cfillion/ja_JP.UTF-8.md

Last active February 26, 2024 14:09
Show Gist options
  • Star 19 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save cfillion/4394c3b8cd051fb45721187053e92296 to your computer and use it in GitHub Desktop.
Save cfillion/4394c3b8cd051fb45721187053e92296 to your computer and use it in GitHub Desktop.
Download ZIP
Solution for locale and character encoding issues in Japanese games on Steam Deck
Raw
ja_JP.UTF-8.md

Problem 1

Cannot access filenames containing non-ASCII characters.

Steam Deck's SteamOS 3 provides only one locale named en_US.utf8 (as configured in /etc/locale.gen) but /etc/locale.conf sets LANG to en_US.UTF-8. They don't match, so the effective locale for programs is C and the character set becomes limited to ASCII.

Set the game's launch options in Steam to LANG=en_US.utf8 %command% to solve this.

To fix this for all applications running in desktop mode (eg. to correctly extract games), create a file named ~/.config/plasma-workspace/env/utf8.sh containing (restart desktop mode to apply):

#!/bin/sh
export LANG=en_US.utf8

Problem 2

Japanese text displays as mojibake or question marks. Some games may even completely fail to load due to script compilation errors (common in visual novels). The Windows ANSI codepage must be set to SHIFT-JIS to fix this.

Wine uses the current system locale (set via LANG) or the LC_ALL environment variable to determine which ANSI codepage to use. This is equivalent to using Locale Emulator on Windows.

Steam Deck's SteamOS 3 doesn't provide a ja_JP.UTF-8 locale so this leaves LC_ALL.

However Steam may override it, so instead set the game's launch options to HOST_LC_ALL=ja_JP.UTF-8 %command% to have Proton set LC_ALL to the desired value.

Problem 3

The simple solutions shown above to problem 1 and 2 are unfortunately mutually exclusive.

ja_JP.UTF-8 is not provided by SteamOS 3 so requesting it makes the effective Unix locale fallback to C (ASCII) again.

Complete solution

This sets the Wine codepage to Japanese and the Unix character set to UTF-8 without requiring root access or unlocking the readonly system partition of the Steam Deck.

  1. Extract the archive steam-deck-jp-locale-utf8.tar.zst using Ark into the home directory on the Steam Deck (/home/deck).
  2. Double-check that /home/deck/locales/run.sh is executable (right click > properties > permissions).
  3. Set the game's launch options to LANG=ja_JP.UTF-8 ~/locales/run.sh %command% to fix locale and encoding issues.

Alternative steps

Follow these steps instead to generate the files manually. A full install of Arch Linux is required (commands may need adjustments to work in other distributions).

  1. On an Arch Linux install:
    1. Uncomment ja_JP.UTF-8 in /etc/locale.gen
    2. Run the following commands: 
      sudo locale-gen

mkdir locales
export LOCPATH="$HOME/locales"
localedef -f UTF-8 -i ja_JP "$LOCPATH/ja_JP.UTF-8"
  2. Transfer the locales directory created above into /home/deck on your Steam Deck (SteamOS 3).
  3. Create a file named run.sh inside that folder containing: 
    #!/bin/bash
# https://gist.github.com/cfillion/4394c3b8cd051fb45721187053e92296
newlocpath=$(dirname "$(realpath "$0")")
script='
  BEGIN { FS=OFS=" '"'--'"' " }
  NF > 1 { for(i = 1; i < NF; ++i) printf "%s", $i OFS; }
  { print "env LOCPATH='"'"'" newlocpath "'"'"' " $NF }
'
command="$(awk -v newlocpath="$newlocpath" "$script" <<< "${@@Q}")"
eval "$command"
  4. Make run.sh executable: chmod +x ~/locales/run.sh
  5. Set the game's launch options to LANG=ja_JP.UTF-8 ~/locales/run.sh %command% to fix locale and encoding issues.

Sources and tips

@MakandaDEV
Copy link

This doesn't work, ive tried anything in the world possible to fix it. All it does is nothing.

@cfillion
Copy link
Author

cfillion commented Jan 8, 2023 •
edited

All it does is nothing.

As in, no visible effect (the game doesn't work either with or without the modified command line), or that it breaks launching the game at all (aka. "Play" does nothing anymore after setting the command line)?

  • If it's the former, maybe whichever Windows game you're trying to play is not compatible with Proton for a reason unrelated to the missing Japanese locale files. You can check the Proton log for clues as to what is the cause.
  • If it's the latter, double-check that you've extracted the entire archive (preserving all subfolders), that /home/deck/locales/run.sh is executable, and that the game's command line in Steam points to its correct location. Assuming you've already done that, the Proton log will also be the next thing to check for clues.
PROTON_LOG=1 WINEDEBUG=+file,+nls,+msgbox %command%

This creates a file named steam-GAMEID.log in the home folder.

Don't forget to enable Proton in the game's Properties in Steam > Compatibility either. (Also the script provided here is only for launching non-Steam games within Steam – both in Desktop and the Deck's Game mode. Steam store games may work too–haven't tried as I don't have any that require a Japanese locale–but it definitely won't work for launching .exe outside of Steam.)

Can't help much more than that without details (eg. game title, what symptoms of "not working" you're getting, perhaps even some screenshots would be helpful to help you).

@MakandaDEV
Copy link

Hi, first of all thanks for the response, so, the game i am trying to run with your script is Monster hunter Frontier Z, the game itself works on proton. But it needs the Japanese locale to fix missing textures and such.

I've tried everything already, i also found someone that got it to work for exactly that game. So i dont see a reason for it to not work at this point. Been trying for 3 days now

@cfillion
Copy link
Author

cfillion commented Jan 8, 2023 •
edited

Thanks, I've tried it (first result on Nyaa). Out of the box, without the script, I got this after logging in:

Without script

With LANG=ja_JP.UTF-8 ~/locales/run.sh %command%:

With locale script

So my locale script does appear to be working with this game. However textures are indeed broken with Proton. The logos were corrupted and the title screen was a complete mess:

Title screen broken textures

I saw this line in the Proton log that seemed like it could be related:

02d8:fixme:d3dx:D3DXCreateTextureFromFileInMemoryEx Unrecognized file format, returning failure.

Googled it, and as per this post I tried installing d3dx9 using ProtonTricks. It fixed the logos and title screen:

Fixed title screen

Hope this helps a little bit. (I tried starting a game, but it failed after character creation due to the server connection getting lost. Not sure if it's another Proton problem or just me not knowing how to connect to the community-run servers.)

@MakandaDEV
Copy link

Hm weird, the problems you have i never had seen before, the actual problems happen after login (aka when you are on a server)
The game got revived by the community a while back, do you have discord by any chance? Would be easier to communicate

@cfillion
Copy link
Author

cfillion commented Jan 8, 2023 •
edited

do you have discord by any chance? Would be easier to communicate

Sure: <EDIT: removed>. Won't spend too much time this but sounds like it might be a fun troubleshoot for a little while.

@MakandaDEV
Copy link

Amazing, thanks alot, friend request sent

@usaginoshoujo
Copy link

ありがとう

@juri0n
Copy link

juri0n commented Jul 2, 2023

Is there a Solution for zh_CN.utf8 ?

@cfillion
Copy link
Author

cfillion commented Jul 2, 2023

Yes, the manual steps can be used to generate the locale files for any language on an Arch Linux install then transfer them to your Steam Deck.

Also SteamOS 3.5 is apparently going to include multiple locales, so hopefully the solutions described in this page won't be needed anymore in the future! source

@juri0n
Copy link

juri0n commented Jul 5, 2023 via email

@jasicarose75
Copy link

agree, not working

@cfillion
Copy link
Author

cfillion commented Aug 20, 2023 •
edited

Which game (and what is the problem with it)? So far it has been working with all Japanese visual novels I've tried.

@PunkyBust
Copy link

PunkyBust commented Oct 12, 2023 •
edited

Thanks you verry much!! 😀
On my Steamdeck each Unity games has this problem.
And now all is working so well to get my prefered language (maternal)
I've just put on steamlauch options: HOST_LC_ALL=fr_FR.UTF-8
And before that off course a little clean on locale.gen follow by sudo locale-gen

I will forward your operationnal mode if I see somes posts on that issue

@bertogg
Copy link

bertogg commented Oct 18, 2023

Hello, as mentioned in an earlier comment SteamOS 3.5 will ship a bunch of locales in addition to en_US.UTF-8 (here is the list). You can try installing it from the preview channel. If you do and notice any issues please let me know.

@sudocurse
Copy link

sudocurse commented Feb 9, 2024 •
edited

Noticing that the steam menu in some games also garbles japanese characters e.g. when copy-pasting.

Steam Deck's SteamOS 3 provides only one locale named en_US.utf8 (as configured in /etc/locale.gen) but /etc/locale.conf sets LANG to en_US.UTF-8. They don't match, so the effective locale for programs is C and the character set becomes limited to ASCII.

I'm curious about why this arose to begin with; glibc should be normalizing UTF-8 references to utf8 making them functionally equivalent. Not sure if that could lead to my issue but it does sound like OP's first problem/fix suggests there may be a bug? See also ValveSoftware/SteamOS#1178

@denilsonsa
Copy link

Steam Deck's SteamOS 3 provides only one locale named en_US.utf8 (as configured in /etc/locale.gen) but /etc/locale.conf sets LANG to en_US.UTF-8.

This might have been true in the past, but it's not true anymore. Multiple locales are configured in /etc/locale.gen and they all have the UTF-8 suffix, instead of the mentioned utf8.

@cfillion
Copy link
Author

This might have been true in the past, but it's not true anymore. Multiple locales are configured in /etc/locale.gen and they all have the UTF-8 suffix, instead of the mentioned utf8.

Indeed, this only applies to SteamOS prior to 3.5.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment