Home/ Questions/Q 7495033
In Process

The Archive Base Latest Questions

Editorial Team
  • 0
Asked: 2026-05-29T17:56:03+00:00

I’m working with a binary file that references another file using absolute paths. The

  • 0

I’m working with a binary file that references another file using absolute paths.
The path contains both japanese and ascii characters.

The length of the string is given, so I can just read that many bytes and convert it into a string.

However the problem is trying to convert the string. If I specify the encoding as ascii, it’ll fail on the japanese characters. If I specify it as japanese encoding (shift-jis or something), it won’t read the english characters properly.

One byte is used for each ascii character, while two bytes are used for each japanese character.

What is the fastest and cleanest way to convert these bytes into a string? The encodings are known. Will the same technique work in older versions of python.

Share

Leave an answer

You must login to add an answer.

Need An Account,

1 Answer

  1. Editorial Team

    This sounds like you have fallen victim for a misunderstand the basics of Unicode and encodings. It may be that you have not, but misunderstandnings are common and understandable, while the situation you describe are not.

    A string of bytes that contains mixed encodings are, per definition, invalid in any of these encodings. If this really was the case, you would have to split the bytes string into it’s parts, and decode every part separately. In this case it would probably mean splitting on the path separators, so it would be reasonably easy, but in other cases it would not. However, I serously doubt that this is the case, as it would mean that your source is insane. That happens, but it is unlikely. 🙂

    If the source gives you one path as a bytes string, it is most likely that this string uses only one encoding. It may contain both Japanese and ASCII-characters and still be using one encoding. The most common encodings that can handle both Japanese and ASCII are UTF-8 and UTF-16. My guess is that your source uses one of those. In fact, since you write “One byte is used for each ascii character, while two bytes are used for each japanese character” it is probably UTF-8. It could also be Shift JIS, but it seems you already tried that.

    If not, please explain what your source is, and give examples of the byte strings (in ASCII/HEX) that you are given.

    • 0