1Raspberry Pi 3
2==============
3
4The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four
5Arm Cortex-A53 cores.
6
7The following instructions explain how to use this port of the TF-A with the
8default distribution of `Raspbian`_ because that's the distribution officially
9supported by the Raspberry Pi Foundation. At the moment of writing this, the
10officially supported kernel is a AArch32 kernel. This doesn't mean that this
11port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by
12the Foundation can be compiled for AArch64 by following the steps in
13`AArch64 kernel build instructions`_.
14
15**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM,
16which is available from both the Non-secure and Secure worlds. This port
17shouldn't be considered more than a prototype to play with and implement
18elements like PSCI to support the Linux kernel.
19
20Design
21------
22
23The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a
24VideoCore IV that acts as primary processor (and loads everything from the SD
25card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi
263 documentation`_ for more information.
27
28This explains why it is possible to change the execution state (AArch64/AArch32)
29depending on a few files on the SD card. We only care about the cases in which
30the cores boot in AArch64 mode.
31
32The rules are simple:
33
34- If a file called ``kernel8.img`` is located on the ``boot`` partition of the
35  SD card, it will load it and execute in EL2 in AArch64. Basically, it executes
36  a `default AArch64 stub`_ at address **0x0** that jumps to the kernel.
37
38- If there is also a file called ``armstub8.bin``, it will load it at address
39  **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All
40  the cores are powered on at the same time and start at address **0x0**.
41
42This means that we can use the default AArch32 kernel provided in the official
43`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and
44anything else we need is in ``armstub8.bin``. This way we can forget about the
45default bootstrap code. When using a AArch64 kernel, it is only needed to make
46sure that the name on the SD card is ``kernel8.img``.
47
48Ideally, we want to load the kernel and have all cores available, which means
49that we need to make the secondary cores work in the way the kernel expects, as
50explained in `Secondary cores`_. In practice, a small bootstrap is needed
51between TF-A and the kernel.
52
53To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode
54in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The
55architecture specifies that AArch32 Hypervisor mode isn't present when AArch64
56is used for EL2. When using a AArch64 kernel, it should simply start in EL2.
57
58Placement of images
59~~~~~~~~~~~~~~~~~~~
60
61The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding
62between them so that the addresses they are loaded to match the ones specified
63when compiling TF-A. This is done automatically by the build system.
64
65The device tree block is loaded by the VideoCore loader from an appropriate
66file, but we can specify the address it is loaded to in ``config.txt``.
67
68The file ``kernel8.img`` contains a kernel image that is loaded to the address
69specified in ``config.txt``. The `Linux kernel tree`_ has information about how
70a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``:
71
72::
73
74    The zImage may also be placed in system RAM and called there.  The
75    kernel should be placed in the first 128MiB of RAM.  It is recommended
76    that it is loaded above 32MiB in order to avoid the need to relocate
77    prior to decompression, which will make the boot process slightly
78    faster.
79
80There are no similar restrictions for AArch64 kernels, as specified in the file
81``Documentation/arm64/booting.txt``.
82
83This means that we need to avoid the first 128 MiB of RAM when placing the
84TF-A images (and specially the first 32 MiB, as they are directly used to
85place the uncompressed AArch32 kernel image. This way, both AArch32 and
86AArch64 kernels can be placed at the same address.
87
88In the end, the images look like the following diagram when placed in memory.
89All addresses are Physical Addresses from the point of view of the Arm cores.
90Again, note that this is all just part of the same DRAM that goes from
91**0x00000000** to **0x3F000000**, it just has different names to simulate a real
92secure platform!
93
94::
95
96    0x00000000 +-----------------+
97               |       ROM       | BL1
98    0x00020000 +-----------------+
99               |       FIP       |
100    0x00200000 +-----------------+
101               |                 |
102               |       ...       |
103               |                 |
104    0x01000000 +-----------------+
105               |       DTB       | (Loaded by the VideoCore)
106               +-----------------+
107               |                 |
108               |       ...       |
109               |                 |
110    0x02000000 +-----------------+
111               |     Kernel      | (Loaded by the VideoCore)
112               +-----------------+
113               |                 |
114               |       ...       |
115               |                 |
116    0x10000000 +-----------------+
117               |   Secure SRAM   | BL2, BL31
118    0x10100000 +-----------------+
119               |   Secure DRAM   | BL32 (Secure payload)
120    0x11000000 +-----------------+
121               | Non-secure DRAM | BL33
122               +-----------------+
123               |                 |
124               |       ...       |
125               |                 |
126    0x3F000000 +-----------------+
127               |       I/O       |
128    0x40000000 +-----------------+
129
130The area between **0x10000000** and **0x11000000** has to be manually protected
131so that the kernel doesn't use it. The current port tries to modify the live DTB
132to add a memreserve region that reserves the previously mentioned area.
133
134If this is not possible, the user may manually add ``memmap=16M$256M`` to the
135command line passed to the kernel in ``cmdline.txt``. See the `Setup SD card`_
136instructions to see how to do it. This system is strongly discouraged.
137
138The last 16 MiB of DRAM can only be accessed by the VideoCore, that has
139different mappings than the Arm cores in which the I/O addresses don't overlap
140the DRAM. The memory reserved to be used by the VideoCore is always placed at
141the end of the DRAM, so this space isn't wasted.
142
143Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for
144TF-A, there are 880 MiB available for Linux.
145
146Boot sequence
147~~~~~~~~~~~~~
148
149The boot sequence of TF-A is the usual one except when booting an AArch32
150kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it
151can jump to the kernel in the same mode and let it take over that privilege
152level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of
153TF-A) it could only jump to the kernel in AArch32 in Supervisor mode.
154
155The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel
156in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The
157bootstrap should take care of this.
158
159This port support a direct boot of the Linux kernel from the firmware (as a BL33
160image). Alternatively, U-Boot or other bootloaders may be used.
161
162Secondary cores
163~~~~~~~~~~~~~~~
164
165This port of the Trusted Firmware-A supports ``PSCI_CPU_ON``,
166``PSCI_SYSTEM_RESET`` and ``PSCI_SYSTEM_OFF``. The last one doesn't really turn
167the system off, it simply reboots it and asks the VideoCore firmware to keep it
168in a low power mode permanently.
169
170The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to
171use mailboxes to trap the secondary cores until they are ready to jump to the
172kernel. This mailbox is located at a different address in the AArch32 default
173kernel than in the AArch64 kernel.
174
175Kernels with PSCI support can use the PSCI calls instead for a cleaner boot.
176
177Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During
178cold boot, all secondary cores wait in a loop until they are given given an
179address to jump to in this Mailbox (``bl31_warm_entrypoint``).
180
181Once BL31 has finished and the primary core has jumped to the BL33 payload, it
182has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop.
183The payload then makes them wait in another waitloop listening from messages
184from the kernel. When the primary CPU jumps into the kernel, it will send an
185address to the mailbox so that the secondary CPUs jump to it and are recognised
186by the kernel.
187
188Build Instructions
189------------------
190
191To boot a AArch64 kernel, only the AArch64 toolchain is required.
192
193To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The
194AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit
195kernel.
196
197The build system concatenates BL1 and the FIP so that the addresses match the
198ones in the memory map. The resulting file is ``armstub8.bin``, located in the
199build folder (e.g. ``build/rpi3/debug/armstub8.bin``). To know how to use this
200file, follow the instructions in `Setup SD card`_.
201
202The following build options are supported:
203
204- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image.
205  By default this option is 0, which means that TF-A will jump to BL33 in EL2
206  in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32
207  mode.
208
209- ``PRELOADED_BL33_BASE``: Used to specify the address of a BL33 binary that has
210  been preloaded by any other system than using the firmware. ``BL33`` isn't
211  needed in the build command line if this option is used. Specially useful
212  because the file ``kernel8.img`` can be loaded anywhere by modifying the file
213  ``config.txt``. It doesn't have to contain a kernel, it could have any
214  arbitrary payload.
215
216- ``RPI3_DIRECT_LINUX_BOOT``: Disabled by default. Set to 1 to enable the direct
217  boot of the Linux kernel from the firmware. Option ``RPI3_PRELOADED_DTB_BASE``
218  is mandatory when the direct Linux kernel boot is used. Options
219  ``PRELOADED_BL33_BASE`` will most likely be needed as well because it is
220  unlikely that the kernel image will fit in the space reserved for BL33 images.
221  This option can be combined with ``RPI3_BL33_IN_AARCH32`` in order to boot a
222  32-bit kernel. The only thing this option does is to set the arguments in
223  registers x0-x3 or r0-r2 as expected by the kernel.
224
225- ``RPI3_PRELOADED_DTB_BASE``: Auxiliary build option needed when using
226  ``RPI3_DIRECT_LINUX_BOOT=1``. This option allows to specify the location of a
227  DTB in memory.
228
229- ``RPI3_RUNTIME_UART``: Indicates whether the UART should be used at runtime
230  or disabled. ``-1`` (default) disables the runtime UART. Any other value
231  enables the default UART (currently UART1) for runtime messages.
232
233- ``RPI3_USE_UEFI_MAP``: Set to 1 to build ATF with the altername memory
234  mapping required for an UEFI firmware payload. These changes are needed
235  to be able to run Windows on ARM64. This option, which is disabled by
236  default, results in the following memory mappings:
237
238::
239
240    0x00000000 +-----------------+
241               |       ROM       | BL1
242    0x00010000 +-----------------+
243               |       DTB       | (Loaded by the VideoCore)
244    0x00020000 +-----------------+
245               |       FIP       |
246    0x00030000 +-----------------+
247               |                 |
248               |  UEFI PAYLOAD   |
249               |                 |
250    0x00200000 +-----------------+
251               |   Secure SRAM   | BL2, BL31
252    0x00300000 +-----------------+
253               |   Secure DRAM   | BL32 (Secure payload)
254    0x00400000 +-----------------+
255               |                 |
256               |                 |
257               | Non-secure DRAM | BL33
258               |                 |
259               |                 |
260    0x01000000 +-----------------+
261               |                 |
262               |       ...       |
263               |                 |
264    0x3F000000 +-----------------+
265               |       I/O       |
266
267- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional.
268  Please use the code from `here <https://github.com/OP-TEE/optee_os>`__.
269  Build the Trusted Firmware with option ``BL32=tee-header_v2.bin
270  BL32_EXTRA1=tee-pager_v2.bin  BL32_EXTRA2=tee-pageable_v2.bin``
271  to put the binaries into the FIP.
272
273  .. warning::
274     If OP-TEE is used it may be needed to add the following options to the
275     Linux command line so that the USB driver doesn't use FIQs:
276     ``dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0``.
277     This will unfortunately reduce the performance of the USB driver. It is
278     needed when using Raspbian, for example.
279
280- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option to 1 to enable
281  it. In order to use TBB, you might want to set ``GENERATE_COT=1`` to let the
282  contents of the FIP automatically signed by the build process. The ROT key
283  will be generated and output to ``rot_key.pem`` in the build directory. It is
284  able to set ROT_KEY to your own key in PEM format.  Also in order to build,
285  you need to clone mbed TLS from `here <https://github.com/ARMmbed/mbedtls>`__.
286  ``MBEDTLS_DIR`` must point at the mbed TLS source directory.
287
288- ``ENABLE_STACK_PROTECTOR``: Disabled by default. It uses the hardware RNG of
289  the board.
290
291The following is not currently supported:
292
293- AArch32 for TF-A itself.
294
295- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any
296  address by changing the file ``armstub8.bin``, so there's no point in using
297  TF-A in this case.
298
299- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the
300  crash console uses the internal 16550 driver functions directly in order to be
301  able to print error messages during early crashes before setting up the
302  multi console API.
303
304Building the firmware for kernels that don't support PSCI
305~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
306
307This is the case for the 32-bit image of Raspbian, for example. 64-bit kernels
308always support PSCI, but they may not know that the system understands PSCI due
309to an incorrect DTB file.
310
311First, clone and compile the 32-bit version of the `Raspberry Pi 3 TF-A
312bootstrap`_. Choose the one needed for the architecture of your kernel.
313
314Then compile TF-A. For a 32-bit kernel, use the following command line:
315
316.. code:: shell
317
318    CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3             \
319    RPI3_BL33_IN_AARCH32=1                                      \
320    BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin
321
322For a 64-bit kernel, use this other command line:
323
324.. code:: shell
325
326    CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3             \
327    BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin
328
329However, enabling PSCI support in a 64-bit kernel is really easy. In the
330repository `Raspberry Pi 3 TF-A bootstrap`_ there is a patch that can be applied
331to the Linux kernel tree maintained by the Raspberry Pi foundation. It modifes
332the DTS to tell the kernel to use PSCI. Once this patch is applied, follow the
333instructions in `AArch64 kernel build instructions`_ to get a working 64-bit
334kernel image and supporting files.
335
336Building the firmware for kernels that support PSCI
337~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
338
339For a 64-bit kernel:
340
341.. code:: shell
342
343    CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3             \
344    PRELOADED_BL33_BASE=0x02000000                              \
345    RPI3_PRELOADED_DTB_BASE=0x01000000                          \
346    RPI3_DIRECT_LINUX_BOOT=1
347
348For a 32-bit kernel:
349
350.. code:: shell
351
352    CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3             \
353    PRELOADED_BL33_BASE=0x02000000                              \
354    RPI3_PRELOADED_DTB_BASE=0x01000000                          \
355    RPI3_DIRECT_LINUX_BOOT=1                                    \
356    RPI3_BL33_IN_AARCH32=1
357
358AArch64 kernel build instructions
359---------------------------------
360
361The following instructions show how to install and run a AArch64 kernel by
362using a SD card with the default `Raspbian`_ install as base. Skip them if you
363want to use the default 32-bit kernel.
364
365Note that this system won't be fully 64-bit because all the tools in the
366filesystem are 32-bit binaries, but it's a quick way to get it working, and it
367allows the user to run 64-bit binaries in addition to 32-bit binaries.
368
3691. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To
370   speed things up, do a shallow clone of the desired branch.
371
372.. code:: shell
373
374    git clone --depth=1 -b rpi-4.18.y https://github.com/raspberrypi/linux
375    cd linux
376
3772. Configure and compile the kernel. Adapt the number after ``-j`` so that it is
378   1.5 times the number of CPUs in your computer. This may take some time to
379   finish.
380
381.. code:: shell
382
383    make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig
384    make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu-
385
3863. Copy the kernel image and the device tree to the SD card. Replace the path
387   by the corresponding path in your computers to the ``boot`` partition of the
388   SD card.
389
390.. code:: shell
391
392    cp arch/arm64/boot/Image /path/to/boot/kernel8.img
393    cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/
394    cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtb /path/to/boot/
395
3964. Install the kernel modules. Replace the path by the corresponding path to the
397   filesystem partition of the SD card on your computer.
398
399.. code:: shell
400
401    make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \
402    INSTALL_MOD_PATH=/path/to/filesystem modules_install
403
4045. Follow the instructions in `Setup SD card`_ except for the step of renaming
405   the existing ``kernel7.img`` (we have already copied a AArch64 kernel).
406
407Setup SD card
408-------------
409
410The instructions assume that you have an SD card with a fresh install of
411`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly
412untouched). They have been tested with the image available in 2018-03-13.
413
4141. Insert the SD card and open the ``boot`` partition.
415
4162. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore
417   bootloader into booting the Arm cores in AArch64 mode, like TF-A needs,
418   even though the kernel is not compiled for AArch64.
419
4203. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore
421   bootloader will look for a file called ``armstub8.bin`` and load it at
422   address **0x0** instead of a predefined one.
423
4244. To enable the serial port "Mini UART" in Linux, open ``cmdline.txt`` and add
425   ``console=serial0,115200 console=tty1``.
426
4275. Open ``config.txt`` and add the following lines at the end (``enable_uart=1``
428   is only needed to enable debugging through the Mini UART):
429
430::
431
432    enable_uart=1
433    kernel_address=0x02000000
434    device_tree_address=0x01000000
435
436If you connect a serial cable to the Mini UART and your computer, and connect
437to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some
438text. In the case of an AArch32 kernel, you should see something like this:
439
440::
441
442    NOTICE:  Booting Trusted Firmware
443    NOTICE:  BL1: v1.4(release):v1.4-329-g61e94684-dirty
444    NOTICE:  BL1: Built : 00:09:25, Nov  6 2017
445    NOTICE:  BL1: Booting BL2
446    NOTICE:  BL2: v1.4(release):v1.4-329-g61e94684-dirty
447    NOTICE:  BL2: Built : 00:09:25, Nov  6 2017
448    NOTICE:  BL1: Booting BL31
449    NOTICE:  BL31: v1.4(release):v1.4-329-g61e94684-dirty
450    NOTICE:  BL31: Built : 00:09:25, Nov  6 2017
451    [    0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517
452
453    Raspbian GNU/Linux 9 raspberrypi ttyS0
454    raspberrypi login:
455
456Just enter your credentials, everything should work as expected. Note that the
457HDMI output won't show any text during boot.
458
459.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S
460.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S
461.. _Linux kernel tree: https://github.com/torvalds/linux
462.. _Linux tree fork: https://github.com/raspberrypi/linux
463.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/
464.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap
465.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/
466.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/
467