CPython and Frozen modules

Frozen Modules

It is common for Firmwares to include a few (or many) python modules as ‘frozen’ modules. ‘Freezing’ modules is a way to pre-process .py modules so they’re ‘baked-in’ to MicroPython’ s firmware and use less memory. Once the code is frozen it can be quickly loaded and interpreted by MicroPython without as much memory and processing time.

Most OSS firmwares store these frozen modules as part of their repository, which allows us to:

  1. Download the *.py from the (github) repo using git clone or a direct download

  2. Extract and store the ‘unfrozen’ modules (ie the *.py files) in a _Frozen folder. if there are different port / boards or releases defined , there may be multiple folders such as:

    • stubs/micropython_1_12_frozen

      • /esp32

        • /GENERIC

        • /RELEASE

        • /TINYPICO

      • /stm32

        • /GENERIC

        • /PYBD_SF2

  3. generate typeshed stubs of these files. (the .pyi files will be stored alongside the .py files)

  4. Include/use them in the configuration

ref: https://learn.adafruit.com/micropython-basics-loading-modules/frozen-modules

Collect Frozen Stubs (micropython)

This is run daily though the github action workflow : get-all-frozen in the micropython-stubs repo.

If you want to run this manually

  • Check out repos side-by-side:

    • micropython-stubs

    • micropython-stubber

    • micropython

    • micropython-lib

  • link repos using all_stubs symlink

  • checkout tag / version in the micropython folder
    (for most accurate results should checkout micropython-lib for the same date)

  • run src/get-frozen.py

  • run src/update-stubs.py

  • create a PR for changes to the stubs repo

Postprocessing

You can run postprocessing for all stubs by running either of the two scripts. There is an optional parameter to specify the location of the stub folder. The default path is ./all_stubs

Powershell:

./scripts/updates_stubs.ps1 [-path ./mystubs]

or python

python ./src/update_stubs.py [./mystubs]

This will generate or update the .pyi stubs for all new (and existing) stubs in the ./all_stubs or specified folder.

From version ‘1.3.8’ the .pyi stubs are generated using stubgen, before that the make_stub_files.py script was used.

Stubgen is run on each ‘collected stub folder’ (that contains a modules.json manifest) using the options : --ignore-errors --include-private and the resulting .pyi files are stored in the same folder (foo.py and foo.pyi are stored next to each other).

In some cases stubgen detects duplicate modules in a ‘collected stub folder’, and subsequently does not generate any stubs for any .py module or script. then Plan B is to run stubgen for each separate *.py file in that folder. THis is significantly slower and according to the stubgen documentation the resulting stubs may of lesser quality, but that is better than no stubs at all.

Note: In several cases stubgen creates folders in inappropriate locations (reason undetermined), which would cause issues when re-running stubgen at a later time. to compensate for this behaviour the known-incorrect .pyi files are removed before and after stubgen is run see: cleanup(modules_folder) in utils.py

Repo structure

This and sister repos

repo

Why

Where

example

micropython-stubber

needed to make stubs

in your source folder

develop/micropython-stubber

micropython

to collect frozen modules

submodule of micropython-stubber

develop/micropython-stubber/micropython

micropython-lib

to collect frozen modules

submodule of micropython-stubber

develop/micropython-stubber/micropython-lib

micropython-stubs

stores collected stubs

next to the stubber

develop/micropython-stubs

Note

  • recommended is to create a symlink from develop/micropython-stubber\all-stubs to develop/micropython-stubs

Note

  • For Git submodules please refer to https://git-scm.com/book/en/v2/Git-Tools-Submodules

Structure of this repo

The file structure is based on my personal windows environment, but you should be able to adapt that without much hardship to you own preference and OS.

What

Details

Where

stub root

symlink to connect the 2 sister-repos

all_stubs

firmware stubber

MicroPython

board/createstubs.py

minified firmware stubber

MicroPython

minified/createstubs.py

PC based scripts

CPython

src/*

PC based scripts

CPython

process.py

pytest tests

test/*

Naming Convention and Stub folder structure

What

Why

Where

stub root

connect the 2 repos

all_stubs

cpython stubs for micropython core

adapt for differences between CPython and MicroPython

stubs/cpython-core

generated stub files

needed to use stubs

stubs/{firmware}-{port}-{version}-frozen

Frozen stub files

better code intellisense

stubs/{firmware}-{version}-frozen

Note: I found that, for me, using submodules caused more problems than it solved. So instead I link the two main repo’s using a symlink.

Note: I in the repo tests I have used the folders TESTREPO-micropython and TESTREPO-micropython-lib to avoid conflicts with any development that you might be doing on similar micropython repos at the potential cost of a little disk space.

cd /develop 

git clone  https://github.com/josverl/micropython-stubber.git 
git clone  https://github.com/josverl/micropython-stubs.git 
git clone  https://github.com/micropython/micropython.git 
git clone  https://github.com/micropython/micropython.git

Stubs

Initially I also stored all the generated subs in the same repo. That turned out to be a bit of a hassle and since then I have moved all the stubs to the micropython-stubs repo

Below are the most relevant stub sources referenced in this project.

Firmware and libraries

MicroPython firmware and frozen modules [MIT]

https://github.com/micropython/micropython

https://github.com/micropython/micropython-lib

Pycopy firmware and frozen modules [MIT]

https://github.com/pfalcon/pycopy

https://github.com/pfalcon/pycopy-lib

LoBoris ESP32 firmware and frozen modules [MIT, Apache 2]

https://github.com/loboris/MicroPython_ESP32_psRAM_LoBo

Included custom stubs

Github repo

Contributions

License

pfalcon/micropython-lib

CPython backports

MIT

dastultz/micropython-pyb

a pyb.py file for use with IDEs in developing a project for the Pyboard

Apache 2

Stub source: MicroPython-lib > CPython backports [MIT, Python]

While micropython-lib focuses on MicroPython, sometimes it may be beneficial to run MicroPython code using CPython, e.g. to use code coverage, debugging, etc. tools available for it. To facilitate such usage, micropython-lib also provides re-implementations (“backports”) of MicroPython modules which run on CPython. https://github.com/pfalcon/micropython-lib#cpython-backports

micropython_pyb [Apache 2]

This project provides a pyb.py file for use with IDEs in developing a project for the Pyboard. https://github.com/dastultz/micropython-pyb

References

Inspiration

Thonny - MicroPython _cmd_dump_api_info [MIT License]

The createstubs.py script to create the stubs is based on the work of Aivar Annamaa and the Thonny crew. It is somewhere deep in the code and is apparently only used during the development cycle but it showed a way how to extract/generate a representation of the MicroPython modules written in C

While the concepts remain, the code has been rewritten to run on a micropython board, rather than on a connected PC running CPython. Please refer to : Thonny code sample

MyPy Stubgen

MyPy stubgen is used to generate stubs for the frozen modules and for the *.py stubs that were generated on a board.

make_stub_files [Public Domain]

https://github.com/edreamleo/make-stub-files

This script make_stub_files.py makes a stub (.pyi) file in the output directory for each source file listed on the command line (wildcard file names are supported).

The script does no type inference. Instead, the user supplies patterns in a configuration file. The script matches these patterns to: The names of arguments in functions and methods and The text of return expressions. Return expressions are the actual text of whatever follows the “return” keyword. The script removes all comments in return expressions and converts all strings to “str”. This preprocessing greatly simplifies pattern matching.

Note

It was found that the stubs / prototypes of some functions with complex arguments were not handled correctly, resulting in incorrectly formatted stubs (.pyi)
Therefore this functionality has been replaced by MyPy stubgen

Documentation on Type hints