| Remote Processor Framework |
| |
| 1. Introduction |
| |
| Modern SoCs typically have heterogeneous remote processor devices in asymmetric |
| multiprocessing (AMP) configurations, which may be running different instances |
| of operating system, whether it's Linux or any other flavor of real-time OS. |
| |
| OMAP4, for example, has dual Cortex-A9, dual Cortex-M3 and a C64x+ DSP. |
| In a typical configuration, the dual cortex-A9 is running Linux in a SMP |
| configuration, and each of the other three cores (two M3 cores and a DSP) |
| is running its own instance of RTOS in an AMP configuration. |
| |
| The remoteproc framework allows different platforms/architectures to |
| control (power on, load firmware, power off) those remote processors while |
| abstracting the hardware differences, so the entire driver doesn't need to be |
| duplicated. In addition, this framework also adds rpmsg virtio devices |
| for remote processors that supports this kind of communication. This way, |
| platform-specific remoteproc drivers only need to provide a few low-level |
| handlers, and then all rpmsg drivers will then just work |
| (for more information about the virtio-based rpmsg bus and its drivers, |
| please read Documentation/rpmsg.txt). |
| |
| 2. User API |
| |
| int rproc_boot(struct rproc *rproc) |
| - Boot a remote processor (i.e. load its firmware, power it on, ...). |
| If the remote processor is already powered on, this function immediately |
| returns (successfully). |
| Returns 0 on success, and an appropriate error value otherwise. |
| Note: to use this function you should already have a valid rproc |
| handle. There are several ways to achieve that cleanly (devres, pdata, |
| the way remoteproc_rpmsg.c does this, or, if this becomes prevalent, we |
| might also consider using dev_archdata for this). See also |
| rproc_get_by_name() below. |
| |
| void rproc_shutdown(struct rproc *rproc) |
| - Power off a remote processor (previously booted with rproc_boot()). |
| In case @rproc is still being used by an additional user(s), then |
| this function will just decrement the power refcount and exit, |
| without really powering off the device. |
| Every call to rproc_boot() must (eventually) be accompanied by a call |
| to rproc_shutdown(). Calling rproc_shutdown() redundantly is a bug. |
| Notes: |
| - we're not decrementing the rproc's refcount, only the power refcount. |
| which means that the @rproc handle stays valid even after |
| rproc_shutdown() returns, and users can still use it with a subsequent |
| rproc_boot(), if needed. |
| - don't call rproc_shutdown() to unroll rproc_get_by_name(), exactly |
| because rproc_shutdown() _does not_ decrement the refcount of @rproc. |
| To decrement the refcount of @rproc, use rproc_put() (but _only_ if |
| you acquired @rproc using rproc_get_by_name()). |
| |
| struct rproc *rproc_get_by_name(const char *name) |
| - Find an rproc handle using the remote processor's name, and then |
| boot it. If it's already powered on, then just immediately return |
| (successfully). Returns the rproc handle on success, and NULL on failure. |
| This function increments the remote processor's refcount, so always |
| use rproc_put() to decrement it back once rproc isn't needed anymore. |
| Note: currently rproc_get_by_name() and rproc_put() are not used anymore |
| by the rpmsg bus and its drivers. We need to scrutinize the use cases |
| that still need them, and see if we can migrate them to use the non |
| name-based boot/shutdown interface. |
| |
| void rproc_put(struct rproc *rproc) |
| - Decrement @rproc's power refcount and shut it down if it reaches zero |
| (essentially by just calling rproc_shutdown), and then decrement @rproc's |
| validity refcount too. |
| After this function returns, @rproc may _not_ be used anymore, and its |
| handle should be considered invalid. |
| This function should be called _iff_ the @rproc handle was grabbed by |
| calling rproc_get_by_name(). |
| |
| 3. Typical usage |
| |
| #include <linux/remoteproc.h> |
| |
| /* in case we were given a valid 'rproc' handle */ |
| int dummy_rproc_example(struct rproc *my_rproc) |
| { |
| int ret; |
| |
| /* let's power on and boot our remote processor */ |
| ret = rproc_boot(my_rproc); |
| if (ret) { |
| /* |
| * something went wrong. handle it and leave. |
| */ |
| } |
| |
| /* |
| * our remote processor is now powered on... give it some work |
| */ |
| |
| /* let's shut it down now */ |
| rproc_shutdown(my_rproc); |
| } |
| |
| 4. API for implementors |
| |
| struct rproc *rproc_alloc(struct device *dev, const char *name, |
| const struct rproc_ops *ops, |
| const char *firmware, int len) |
| - Allocate a new remote processor handle, but don't register |
| it yet. Required parameters are the underlying device, the |
| name of this remote processor, platform-specific ops handlers, |
| the name of the firmware to boot this rproc with, and the |
| length of private data needed by the allocating rproc driver (in bytes). |
| |
| This function should be used by rproc implementations during |
| initialization of the remote processor. |
| After creating an rproc handle using this function, and when ready, |
| implementations should then call rproc_register() to complete |
| the registration of the remote processor. |
| On success, the new rproc is returned, and on failure, NULL. |
| |
| Note: _never_ directly deallocate @rproc, even if it was not registered |
| yet. Instead, if you just need to unroll rproc_alloc(), use rproc_free(). |
| |
| void rproc_free(struct rproc *rproc) |
| - Free an rproc handle that was allocated by rproc_alloc. |
| This function should _only_ be used if @rproc was only allocated, |
| but not registered yet. |
| If @rproc was already successfully registered (by calling |
| rproc_register()), then use rproc_unregister() instead. |
| |
| int rproc_register(struct rproc *rproc) |
| - Register @rproc with the remoteproc framework, after it has been |
| allocated with rproc_alloc(). |
| This is called by the platform-specific rproc implementation, whenever |
| a new remote processor device is probed. |
| Returns 0 on success and an appropriate error code otherwise. |
| Note: this function initiates an asynchronous firmware loading |
| context, which will look for virtio devices supported by the rproc's |
| firmware. |
| If found, those virtio devices will be created and added, so as a result |
| of registering this remote processor, additional virtio drivers might get |
| probed. |
| Currently, though, we only support a single RPMSG virtio vdev per remote |
| processor. |
| |
| int rproc_unregister(struct rproc *rproc) |
| - Unregister a remote processor, and decrement its refcount. |
| If its refcount drops to zero, then @rproc will be freed. If not, |
| it will be freed later once the last reference is dropped. |
| |
| This function should be called when the platform specific rproc |
| implementation decides to remove the rproc device. it should |
| _only_ be called if a previous invocation of rproc_register() |
| has completed successfully. |
| |
| After rproc_unregister() returns, @rproc is _not_ valid anymore and |
| it shouldn't be used. More specifically, don't call rproc_free() |
| or try to directly free @rproc after rproc_unregister() returns; |
| none of these are needed, and calling them is a bug. |
| |
| Returns 0 on success and -EINVAL if @rproc isn't valid. |
| |
| 5. Implementation callbacks |
| |
| These callbacks should be provided by platform-specific remoteproc |
| drivers: |
| |
| /** |
| * struct rproc_ops - platform-specific device handlers |
| * @start: power on the device and boot it |
| * @stop: power off the device |
| * @kick: kick a virtqueue (virtqueue id given as a parameter) |
| */ |
| struct rproc_ops { |
| int (*start)(struct rproc *rproc); |
| int (*stop)(struct rproc *rproc); |
| void (*kick)(struct rproc *rproc, int vqid); |
| }; |
| |
| Every remoteproc implementation should at least provide the ->start and ->stop |
| handlers. If rpmsg functionality is also desired, then the ->kick handler |
| should be provided as well. |
| |
| The ->start() handler takes an rproc handle and should then power on the |
| device and boot it (use rproc->priv to access platform-specific private data). |
| The boot address, in case needed, can be found in rproc->bootaddr (remoteproc |
| core puts there the ELF entry point). |
| On success, 0 should be returned, and on failure, an appropriate error code. |
| |
| The ->stop() handler takes an rproc handle and powers the device down. |
| On success, 0 is returned, and on failure, an appropriate error code. |
| |
| The ->kick() handler takes an rproc handle, and an index of a virtqueue |
| where new message was placed in. Implementations should interrupt the remote |
| processor and let it know it has pending messages. Notifying remote processors |
| the exact virtqueue index to look in is optional: it is easy (and not |
| too expensive) to go through the existing virtqueues and look for new buffers |
| in the used rings. |
| |
| 6. Binary Firmware Structure |
| |
| At this point remoteproc only supports ELF32 firmware binaries. However, |
| it is quite expected that other platforms/devices which we'd want to |
| support with this framework will be based on different binary formats. |
| |
| When those use cases show up, we will have to decouple the binary format |
| from the framework core, so we can support several binary formats without |
| duplicating common code. |
| |
| When the firmware is parsed, its various segments are loaded to memory |
| according to the specified device address (might be a physical address |
| if the remote processor is accessing memory directly). |
| |
| In addition to the standard ELF segments, most remote processors would |
| also include a special section which we call "the resource table". |
| |
| The resource table contains system resources that the remote processor |
| requires before it should be powered on, such as allocation of physically |
| contiguous memory, or iommu mapping of certain on-chip peripherals. |
| Remotecore will only power up the device after all the resource table's |
| requirement are met. |
| |
| In addition to system resources, the resource table may also contain |
| resource entries that publish the existence of supported features |
| or configurations by the remote processor, such as trace buffers and |
| supported virtio devices (and their configurations). |
| |
| Currently the resource table is just an array of: |
| |
| /** |
| * struct fw_resource - describes an entry from the resource section |
| * @type: resource type |
| * @id: index number of the resource |
| * @da: device address of the resource |
| * @pa: physical address of the resource |
| * @len: size, in bytes, of the resource |
| * @flags: properties of the resource, e.g. iommu protection required |
| * @reserved: must be 0 atm |
| * @name: name of resource |
| */ |
| struct fw_resource { |
| u32 type; |
| u32 id; |
| u64 da; |
| u64 pa; |
| u32 len; |
| u32 flags; |
| u8 reserved[16]; |
| u8 name[48]; |
| } __packed; |
| |
| Some resources entries are mere announcements, where the host is informed |
| of specific remoteproc configuration. Other entries require the host to |
| do something (e.g. reserve a requested resource) and possibly also reply |
| by overwriting a member inside 'struct fw_resource' with info about the |
| allocated resource. |
| |
| Different resource entries use different members of this struct, |
| with different meanings. This is pretty limiting and error-prone, |
| so the plan is to move to variable-length TLV-based resource entries, |
| where each resource will begin with a type and length fields, followed by |
| its own specific structure. |
| |
| Here are the resource types that are currently being used: |
| |
| /** |
| * enum fw_resource_type - types of resource entries |
| * |
| * @RSC_CARVEOUT: request for allocation of a physically contiguous |
| * memory region. |
| * @RSC_DEVMEM: request to iommu_map a memory-based peripheral. |
| * @RSC_TRACE: announces the availability of a trace buffer into which |
| * the remote processor will be writing logs. In this case, |
| * 'da' indicates the device address where logs are written to, |
| * and 'len' is the size of the trace buffer. |
| * @RSC_VRING: request for allocation of a virtio vring (address should |
| * be indicated in 'da', and 'len' should contain the number |
| * of buffers supported by the vring). |
| * @RSC_VIRTIO_DEV: announces support for a virtio device, and serves as |
| * the virtio header. 'da' contains the virtio device |
| * features, 'pa' holds the virtio guest features (host |
| * will write them here after they're negotiated), 'len' |
| * holds the virtio status, and 'flags' holds the virtio |
| * device id (currently only VIRTIO_ID_RPMSG is supported). |
| */ |
| enum fw_resource_type { |
| RSC_CARVEOUT = 0, |
| RSC_DEVMEM = 1, |
| RSC_TRACE = 2, |
| RSC_VRING = 3, |
| RSC_VIRTIO_DEV = 4, |
| RSC_VIRTIO_CFG = 5, |
| }; |
| |
| Most of the resource entries share the basic idea of address/length |
| negotiation with the host: the firmware usually asks for memory |
| of size 'len' bytes, and the host needs to allocate it and provide |
| the device/physical address (when relevant) in 'da'/'pa' respectively. |
| |
| If the firmware is compiled with hard coded device addresses, and |
| can't handle dynamically allocated 'da' values, then the 'da' field |
| will contain the expected device addresses (today we actually only support |
| this scheme, as there aren't yet any use cases for dynamically allocated |
| device addresses). |
| |
| We also expect that platform-specific resource entries will show up |
| at some point. When that happens, we could easily add a new RSC_PLAFORM |
| type, and hand those resources to the platform-specific rproc driver to handle. |
| |
| 7. Virtio and remoteproc |
| |
| The firmware should provide remoteproc information about virtio devices |
| that it supports, and their configurations: a RSC_VIRTIO_DEV resource entry |
| should specify the virtio device id, and subsequent RSC_VRING resource entries |
| should indicate the vring size (i.e. how many buffers do they support) and |
| where should they be mapped (i.e. which device address). Note: the alignment |
| between the consumer and producer parts of the vring is assumed to be 4096. |
| |
| At this point we only support a single virtio rpmsg device per remote |
| processor, but the plan is to remove this limitation. In addition, once we |
| move to TLV-based resource table, the plan is to have a single RSC_VIRTIO |
| entry per supported virtio device, which will include the virtio header, |
| the vrings information and the virtio config space. |
| |
| Of course, RSC_VIRTIO resource entries are only good enough for static |
| allocation of virtio devices. Dynamic allocations will also be made possible |
| using the rpmsg bus (similar to how we already do dynamic allocations of |
| rpmsg channels; read more about it in rpmsg.txt). |