pw_tokenizer: Replace string literals with tokens

pw_tokenizer provides macros that replace printf-style string literals
with 32-bit hashes at compile time. The string literals are removed
from the resulting binary, which dramatically reduces the binary size.
Like any printf-style string, binary versions of the strings can be
formatted with arguments and then transmitted or stored.

The pw_tokenizer module is general purpose, but its most common use case
is binary logging. In binary logging, human-readable text logs are
replaced with binary tokens. These are decoded off-device.

This commit includes the C and C++ code for tokenizing strings. It also
includes a C++ library for decoding tokenized strings.

Change-Id: I6d5737ab2d6dfdd76dcf70c852b547fdcd68d683
40 files changed
tree: 12036046cca9ceb239ab4e6e9ec26a896ab3dd80
  1. docs/
  2. env_setup/
  3. pw_base64/
  4. pw_bloat/
  5. pw_build/
  6. pw_cli/
  7. pw_cpu_exception/
  8. pw_cpu_exception_armv7m/
  9. pw_docgen/
  10. pw_doctor/
  11. pw_dumb_io/
  12. pw_dumb_io_baremetal_stm32f429/
  13. pw_dumb_io_stdio/
  14. pw_module/
  15. pw_preprocessor/
  16. pw_presubmit/
  17. pw_protobuf/
  18. pw_protobuf_compiler/
  19. pw_span/
  20. pw_status/
  21. pw_string/
  22. pw_target_runner/
  23. pw_tokenizer/
  24. pw_toolchain/
  25. pw_unit_test/
  26. pw_varint/
  27. pw_watch/
  28. targets/
  29. .clang-format
  30. .gitignore
  31. .gn
  32. .pylintrc
  33. AUTHORS
  34. BUILD
  35. BUILD.gn
  36. BUILDCONFIG.gn
  37. CMakeLists.txt
  38. CONTRIBUTING.md
  39. gn_defaults.gni
  40. LICENSE
  41. modules.gni
  42. pw_vars_default.gni
  43. README.md
  44. WORKSPACE
README.md

Pigweed Embedded Oriented Software Libraries

Pigweed is a collection of embedded-focused libraries, which we call "modules". These modules are designed for small-footprint MMU-less microcontrollers like the ST Micro STM32L452 or the Nordic NRF82832. The modules are designed to facilitate easy integration into existing codebases.

Pigweed is in the early stages of development.

Getting Started

$ git clone sso://pigweed.googlesource.com/pigweed/pigweed ~/pigweed
$ cd ~/pigweed
$ env_setup/cipd/cipd.py auth-login  # Once per machine.
$ . env_setup/bootstrap.sh

You can use . env_setup/env_setup.sh in place of . env_setup/bootstrap.sh. Both should work every time, but bootstrap.sh tends to remove and reinstall things at the expense of time whereas env_setup.sh tends to do basic checks to see if time can be saved by skipping expensive operations.

If you're using Homebrew and you get an error saying module 'http.client' has no attribute 'HTTPSConnection' then your Homebrew Python was not set up to support SSL. Ensure it's installed with brew install openssl and then run brew uninstall python && brew install python. After that things should work.

The environment setup script will pull down the versions of tools necessary to build Pigweed and add them to your environment. You can then build with GN, CMake, or Bazel. You can also confirm you're getting the right versions of tools—they should be installed under env_setup/.

Build for the host with GN

$ which gn
~/pigweed/.cipd/pigweed.ensure/gn
$ gn gen out/host
$ ninja -C out/host

Build for the host with CMake

$ which cmake
~/pigweed/.cipd/pigweed.ensure/bin/cmake
$ cmake -B out/cmake-host -S . -G Ninja
$ ninja -C out/cmake-host

Build for the host with Bazel

$ which bazel
~/pigweed/.cipd/pigweed.ensure/bazel
$ bazel test //...

Build for the STM32F429 Discovery board

$ gn gen --args='pw_target_config = "//targets/stm32f429i-disc1/target_config.gni"' out/disco
$ ninja -C out/disco
$ pw test --root out/disco/ --runner stm32f429i_disc1_unit_test_runner

The CMake and Bazel builds do not yet support building for hardware.

To flash firmware to an STM32 Discovery development board (and run pw test) from macOS, you need to install OpenOCD. Install Homebrew using the latest instructions at https://brew.sh/, then install OpenOCD with brew install openocd.

If any of this doesn't work please file a bug.