Byte Strings Are Decoded To ASCII In IO By Default In Python

1. encode and decode the terminology

  • string = <class 'str'> = decoded byte string → that is executed by the brain
  • byte string = <class 'bytes'> = encoded string → that is executed by the machine
  • by default, python decodes byte strings to ASCII when they are printed — this is confusing unless known
  • to remove b' pre-fix from the string you need to run decode() method on it
  • but if there are non-ASCII characters, you will notice/feel the pain immediately even when accessing/printing the variable
>>> a = 'š'.encode('utf-8')
>>> a
b'\xc5\xa1'
>>> a.decode('ascii')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128)
'ascii' codec can't decode byte 0xc5 in position 0: ordinal not in range(128)
>>> a.decode('utf-8')
'š'
>>> a
b'\xc5\xa1'
>>> print(a)
b'\xc5\xa1'
>>> a = 'š'.encode('ascii')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
UnicodeEncodeError: 'ascii' codec can't encode character '\u0161' in position 0: ordinal not in range(128)
'ascii' codec can't encode character '\u0161' in position 0: ordinal not in range(128)

5. links

Infrastructure Support Engineer/Technical Writer (Snowplow Analytics) with a passion for Python/writing documentation. More about me: https://pavol.kutaj.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Azure Data Explorer에 대량 CSV 파일 ingest

How to make your Raspberry Pi file system read-only (Raspbian Stretch)

OOP: 3 types of inheritance

Porting Python 2.7 Code to Python 3.X

Gain valuable traction for your web app with OEmbed

How Creating Own Password Manager Can Ease Your Pain.

Flow Of The App

9 Useful Programming Tips from a Senior Programmer

What are RESTful Web Services?

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Pavol Kutaj

Pavol Kutaj

Infrastructure Support Engineer/Technical Writer (Snowplow Analytics) with a passion for Python/writing documentation. More about me: https://pavol.kutaj.com

More from Medium

How To Write Into A Single Variable From Different Modules In Python

Introduction To Python Part 6

How To Friendly Inform Python Users To Install Missing Modules With Pip.Txt

How To Write Multiline Format Strings In Python