Ytdyklly

I'm running this code in Windows Powershell and it includes this file called languages.txt where I'm trying to convert between bytes to strings:

Here is languages.txt:

Afrikaans

አማርኛ

Аҧсшәа

العربية

Aragonés

Arpetan

Azərbaycanca

Bamanankan

বাংলা

Bân-lâm-gú

Беларуская

Български

Boarisch

Bosanski

Буряад

Català

Чӑвашла

Čeština

Cymraeg

Dansk

Deutsch

Eesti

Ελληνικά

Español

Esperanto

فارسی

Français

Frysk

Gaelg

Gàidhlig

Galego

한국어

Հայերեն

हिन्दी

Hrvatski

Ido

Interlingua

Italiano

עברית

ಕನ್ನಡ

Kapampangan

ქართული

Қазақша

Kreyòl ayisyen

Latgaļu

Latina

Latviešu

Lëtzebuergesch

Lietuvių

Magyar

Македонски

Malti

मराठी

მარგალური

مازِرونی

Bahasa Melayu

Монгол

Nederlands

नेपाल भाषा

日本語

Norsk bokmål

Nouormand

Occitan

Oʻzbekcha/ўзбекча

ਪੰਜਾਬੀ

پنجابی

پښتو

Plattdüütsch

Polski

Português

Română

Romani

Русский

Seeltersk

Shqip

Simple English

Slovenčina

کوردیی ناوەندی

Српски / srpski

Suomi

Svenska

Tagalog

தமிழ்

ภาษาไทย

Taqbaylit

Татарча/tatarça

తెలుగు

Тоҷикӣ

Türkçe

Українська

اردو

Tiếng Việt

Võro

文言

吴语

ייִדיש

中文

Then, here is the code I used:

import sys

script, input_encoding, error = sys.argv





def main(language_file, encoding, errors):

    line = language_file.readline()



    if line:

        print_line(line, encoding, errors)

        return main(language_file, encoding, errors)





def print_line(line, encoding, errors):

    next_lang = line.strip()

    raw_bytes = next_lang.encode(encoding, errors=errors)

    cooked_string = raw_bytes.decode(encoding, errors=errors)



    print(raw_bytes, "<===>", cooked_string)





languages = open("languages.txt", encoding="utf-8")



main(languages, input_encoding, error)

Here's the output (credit: Learn Python 3 the Hard Way by Zed A. Shaw):

I don't know why it doesn't upload the characters and shows question blocks instead. Can anyone help me?

The first string which fails is አማርኛ. The first character, አ is in unicode 12A0 (see here). In UTF-8, that is b'xe1x8axa0'. So, that part is obviously fine. The file really is UTF-8.

Printing did not raise an exception, so your output encoding can handle all of the characters. Everything is fine.

The only remaining reason I see for it to fail is that the font used in the console does not support all of the characters.

If it is just for play, you should not worry about it. Consider it working correctly.

On the other hand, I would suggest changing some things in your code:

1. You are running main recursively for each line. There is absolutely no need for that and it would run into recursion depth limit on a longer file. User a for loop instead.

for line in lines:

    print_line(line, encoding, errors)

2. You are opening the file as UTF-8, so reading from it automatically decodes UTF-8 into Unicode, then you encode it back into row_bytes and then encode again into cooked_string, which is the same as line. It would be a better exercise to read the file as raw binary, split it on newlines and then decode. Then you'd have a clearer picture of what is going on.

with open("languages.txt", 'rb') as f:

    raw_file_contents = f.read()