CPython and Frozen Modules

Frozen Modules

It is common for firmwares to include a few (or many) Python modules as ‘frozen modules’. ‘Freezing’ modules is a way to pre-process .py modules so they’re ‘baked-in’ to MicroPython’s firmware and use less memory. Once the code is frozen it can be quickly loaded and interpreted by MicroPython without as much memory and processing time.

Most OSS firmwares store these frozen modules as part of their repository, which allows us to:

  1. Download the *.py from the (GitHub) repo using git clone or a direct download.

  2. Extract and store the ‘unfrozen’ modules (i.e. the *.py files) in a _Frozen folder. If there are different port / boards or releases defined, there may be multiple folders such as:

    • stubs/micropython_1_12_frozen

      • /esp32

        • /GENERIC

        • /RELEASE

        • /TINYPICO

      • /stm32

        • /GENERIC

        • /PYBD_SF2

  3. Generate typeshed stubs of these files. const pre-processing: As the mypy.stubgen tool is not able to incur the correct types from the MicroPython foo = const(1) syntax, the ‘to be frozen’ modules are pre-processed using a regular expression to replace the foo = const(1) with foo = 1. If the .py files contain any docstrings, they are preserved. However this is uncommon as most micropython-lib modules have no docstrings to save space.

    Addition of docstrings: Then the docstring to modules, classes and methods are added by merging the docstrings based on the docstubs generated from the MicroPython documentation.

    Finally the stubs are generated using the stubgen tool. The resulting .pyi files are stored alongside the .py files

  4. Include/use them in the configuration

ref: https://learn.adafruit.com/micropython-basics-loading-modules/frozen-modules

Collect Frozen Stubs (micropython)

stubber frozen (alias: frozen-stubs) pulls frozen modules from the selected MicroPython version and turns them into stubs. For a complete end-to-end walkthrough see the Quick Start guide.

Typical local workflow:

  1. Initialize your workspace (from an empty working directory): stubber init.

  2. Enter the workspace created by init: cd micropython-stubs.

  3. Optional: switch to a different version/tag: stubber switch v1.22.2 (or preview, custom tag).

  4. Generate frozen stubs: stubber frozen --version stable --enrich.

  5. Combine with doc-based stubs and build wheels as needed:

    • stubber merge --port esp32 --board ESP32_GENERIC --version stable

    • stubber build --port esp32 --board ESP32_GENERIC --version stable

CI in micropython-stubs automates a similar flow via the get-all-frozen workflow.

Postprocessing

You can run postprocessing for all stubs by running either of the two scripts. There is an optional parameter to specify the location of the stub folder. The default path is ./all_stubs

update_stubs [./mystubs]

This will generate or update the .pyi stubs for all new (and existing) stubs in the ./all_stubs or specified folder.

From version ‘1.3.8’ the .pyi stubs are generated using stubgen, before that the make_stub_files.py script was used.

Stubgen is run on each ‘collected stub folder’ (that contains a modules.json manifest) using the options : --ignore-errors --include-private and the resulting .pyi files are stored in the same folder (foo.py and foo.pyi are stored next to each other).

In some cases stubgen detects duplicate modules in a ‘collected stub folder’, and subsequently does not generate any stubs for any .py module or script. then Plan B is to run stubgen for each separate *.py file in that folder. While this is significantly slower and according to the stubgen documentation the resulting stubs may of lesser quality, but that is better than no stubs at all.

Note: In several cases stubgen creates folders in inappropriate locations (reason undetermined), which would cause issues when re-running stubgen at a later time. to compensate for this behaviour the known-incorrect .pyi files are removed before and after stubgen is run see: cleanup(modules_folder) in utils.py