How to Debug Python Scripts running in Docker Container

The aim of this page📝 is to provide a fast&repeatable setup of

1. debugging experience for
2. …python code that is located in
3. …docker container which contains
4. …bunch of python scripts

…in my case in the context of infrastructure as code → infra/SRE/DevOps stuff → not a container running only a dedicated python app. I am using a GitHub repo containing hundreds of python scripts. I am on Windows, my fellow_colleagues not (theoretically not a problem, right? :) I am decidedly not using VS CODE Docker Extension — all that the following is relying on is a bit of custom configuration of the default VS CODE debugpy module and running a container with a config that makes it attachable.

• NOTE: the prerequisite is that all <script>.py dependencies (i.e. imports) are properly defined/built/found in dockerfile/image/container

The Aim → VS CODE debugging session successfully attached to a script running in a Docker container (in ~60 seconds)

INSTRUCTIONS

1. HOST: run the container with debug_optimized configuration
2. CONTAINER: get the absolute path of the script you want to run

/root/infra/scripts/

3. HOST > VSCODE: open launch.json and paste the launch configuration to attach Python debugger to running program in a Docker container.

4. HOST > VSCODE: still in launch.json, update the localRoot to point to the file dirname on the host and remoteRoot key with the hardcoded container absolute path of the script you want to debug

• NOTE: can be skipped, if you want to be breaking for Uncaught Exceptions _

"pathMappings": [
                {
                    "localRoot": "${fileDirname}",
                    "remoteRoot": "/root/infra/scripts/" /* <<< UPDATE */
                }

5. HOST > VSCODE: open the <script>.py and paste the following to the top of the script (could be modularized into single import )

import subprocess
import sys
try:
    import debugpy
except ModuleNotFoundError:
    subprocess.check_call([sys.executable, "-m", "pip", "install", "debugpy"])
    import debugpy
debugpy.listen(("0.0.0.0", 5678))
print("Waiting for client to attach...")
debugpy.wait_for_client()

6. CONTAINER: Call the script. The script should import the debugpy module and then pause with Waiting for client to attach...

root@1b9a14236f9b:~/infra/scripts/#python script.py
Collecting debugpy
  Downloading debugpy-1.5.1-cp36-cp36m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (1.9 MB)
     |████████████████████████████████| 1.9 MB 1.2 MB/s
Installing collected packages: debugpy
Successfully installed debugpy-1.5.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
Waiting for client to attach...

7. HOST > VSCODE: If needed, set a breakpoint in script.py (or keep a default breaking on Uncaught exceptions only to catch errors only)

8. HOST > VSCODE: Start the debugger with F5 on Python: Docker Attach config. The debugger that should attach!

SETUP#1: DOCKER RUN COMMAND

docker run --rm -it `
    --name "<container_name>" `
    -v "<host_repo_path>:<container_repo_path>" `
    -e LANG='en_US.UTF-8' `
    -e PYTHONIOENCODING='UTF-8' `
    -e PYTHONPATH="<container path to dependencies>" `
    -p 5678:5678 `
    -w $project_container_folder `
    <docker image>

SETUP#2: LAUNCH.JSON IN VS CODE

• NOTE: I am also activating debugger logging to files, see https://github.com/microsoft/debugpy/wiki/Enable-debugger-logs for more

# launch.json
[
{
            "name": "Python: Docker Attach",
            "type": "python",
            "request": "attach",
            "connect": {
                "host": "localhost",
                "port": 5678
            },
            "pathMappings": [
                {
                    "localRoot": "${workspaceFolder}",
                    "remoteRoot": "<absolute_path_to_script_folder_on_CONTAINER>"
                }
            ],
            "logToFile": true
        }
]

OPTIMIZATION: CREATING A DEDICATED DEBUG_DOCKER MODULE

• HOST > DOCKER RUN: You could of course create a dedicated module (debug_docker.py, etc.) → map it to the container

# DOCKER RUN COMMAND EXAMPLE: MAPPING HOST MODULE TO DEDICATED FOLDER (NOT EXISTING!) 
-v C:\Users\Admin\python\debug_docker:"tfm/lib/python/debug_docker/" `

• HOST > DOCKER RUN: ...pass the container path into PYTHONPATH environmental variable just run something like

# DOCKER RUN COMMAND EXAMPLE: MAKING MODULES ACCESSIBLE FOR IMPORT STATEMENTS
-e PYTHONPATH="${project_container_folder}/lib/python" `

• HOST > VS CODE > script.py ...and import the mapped module into the script.py

# SCRIPT.PY
from debug_docker import debug_docker

LINKS

• https://towardsdatascience.com/debugging-for-dockerized-ml-applications-in-python-2f7dec30573d
• https://docs.docker.com/engine/reference/run
• https://github.com/microsoft/debugpy/wiki/Enable-debugger-logs
• https://code.visualstudio.com/docs/editor/variables-reference