1Raspberry Pi 3 2============== 3 4The `Raspberry Pi 3`_ is an inexpensive single-board computer that contains four 5Arm Cortex-A53 cores. 6 7The following instructions explain how to use this port of the TF-A with the 8default distribution of `Raspbian`_ because that's the distribution officially 9supported by the Raspberry Pi Foundation. At the moment of writing this, the 10officially supported kernel is a AArch32 kernel. This doesn't mean that this 11port of TF-A can't boot a AArch64 kernel. The `Linux tree fork`_ maintained by 12the Foundation can be compiled for AArch64 by following the steps in 13`AArch64 kernel build instructions`_. 14 15**IMPORTANT NOTE**: This port isn't secure. All of the memory used is DRAM, 16which is available from both the Non-secure and Secure worlds. This port 17shouldn't be considered more than a prototype to play with and implement 18elements like PSCI to support the Linux kernel. 19 20Design 21------ 22 23The SoC used by the Raspberry Pi 3 is the Broadcom BCM2837. It is a SoC with a 24VideoCore IV that acts as primary processor (and loads everything from the SD 25card) and is located between all Arm cores and the DRAM. Check the `Raspberry Pi 263 documentation`_ for more information. 27 28This explains why it is possible to change the execution state (AArch64/AArch32) 29depending on a few files on the SD card. We only care about the cases in which 30the cores boot in AArch64 mode. 31 32The rules are simple: 33 34- If a file called ``kernel8.img`` is located on the ``boot`` partition of the 35 SD card, it will load it and execute in EL2 in AArch64. Basically, it executes 36 a `default AArch64 stub`_ at address **0x0** that jumps to the kernel. 37 38- If there is also a file called ``armstub8.bin``, it will load it at address 39 **0x0** (instead of the default stub) and execute it in EL3 in AArch64. All 40 the cores are powered on at the same time and start at address **0x0**. 41 42This means that we can use the default AArch32 kernel provided in the official 43`Raspbian`_ distribution by renaming it to ``kernel8.img``, while TF-A and 44anything else we need is in ``armstub8.bin``. This way we can forget about the 45default bootstrap code. When using a AArch64 kernel, it is only needed to make 46sure that the name on the SD card is ``kernel8.img``. 47 48Ideally, we want to load the kernel and have all cores available, which means 49that we need to make the secondary cores work in the way the kernel expects, as 50explained in `Secondary cores`_. In practice, a small bootstrap is needed 51between TF-A and the kernel. 52 53To get the most out of a AArch32 kernel, we want to boot it in Hypervisor mode 54in AArch32. This means that BL33 can't be in EL2 in AArch64 mode. The 55architecture specifies that AArch32 Hypervisor mode isn't present when AArch64 56is used for EL2. When using a AArch64 kernel, it should simply start in EL2. 57 58Placement of images 59~~~~~~~~~~~~~~~~~~~ 60 61The file ``armstub8.bin`` contains BL1 and the FIP. It is needed to add padding 62between them so that the addresses they are loaded to match the ones specified 63when compiling TF-A. This is done automatically by the build system. 64 65The device tree block is loaded by the VideoCore loader from an appropriate 66file, but we can specify the address it is loaded to in ``config.txt``. 67 68The file ``kernel8.img`` contains a kernel image that is loaded to the address 69specified in ``config.txt``. The `Linux kernel tree`_ has information about how 70a AArch32 Linux kernel image is loaded in ``Documentation/arm/Booting``: 71 72:: 73 74 The zImage may also be placed in system RAM and called there. The 75 kernel should be placed in the first 128MiB of RAM. It is recommended 76 that it is loaded above 32MiB in order to avoid the need to relocate 77 prior to decompression, which will make the boot process slightly 78 faster. 79 80There are no similar restrictions for AArch64 kernels, as specified in the file 81``Documentation/arm64/booting.txt``. 82 83This means that we need to avoid the first 128 MiB of RAM when placing the 84TF-A images (and specially the first 32 MiB, as they are directly used to 85place the uncompressed AArch32 kernel image. This way, both AArch32 and 86AArch64 kernels can be placed at the same address. 87 88In the end, the images look like the following diagram when placed in memory. 89All addresses are Physical Addresses from the point of view of the Arm cores. 90Again, note that this is all just part of the same DRAM that goes from 91**0x00000000** to **0x3F000000**, it just has different names to simulate a real 92secure platform! 93 94:: 95 96 0x00000000 +-----------------+ 97 | ROM | BL1 98 0x00020000 +-----------------+ 99 | FIP | 100 0x00200000 +-----------------+ 101 | | 102 | ... | 103 | | 104 0x01000000 +-----------------+ 105 | DTB | (Loaded by the VideoCore) 106 +-----------------+ 107 | | 108 | ... | 109 | | 110 0x02000000 +-----------------+ 111 | Kernel | (Loaded by the VideoCore) 112 +-----------------+ 113 | | 114 | ... | 115 | | 116 0x10000000 +-----------------+ 117 | Secure SRAM | BL2, BL31 118 0x10100000 +-----------------+ 119 | Secure DRAM | BL32 (Secure payload) 120 0x11000000 +-----------------+ 121 | Non-secure DRAM | BL33 122 +-----------------+ 123 | | 124 | ... | 125 | | 126 0x3F000000 +-----------------+ 127 | I/O | 128 0x40000000 +-----------------+ 129 130The area between **0x10000000** and **0x11000000** has to be manually protected 131so that the kernel doesn't use it. The current port tries to modify the live DTB 132to add a memreserve region that reserves the previously mentioned area. 133 134If this is not possible, the user may manually add ``memmap=16M$256M`` to the 135command line passed to the kernel in ``cmdline.txt``. See the `Setup SD card`_ 136instructions to see how to do it. This system is strongly discouraged. 137 138The last 16 MiB of DRAM can only be accessed by the VideoCore, that has 139different mappings than the Arm cores in which the I/O addresses don't overlap 140the DRAM. The memory reserved to be used by the VideoCore is always placed at 141the end of the DRAM, so this space isn't wasted. 142 143Considering the 128 MiB allocated to the GPU and the 16 MiB allocated for 144TF-A, there are 880 MiB available for Linux. 145 146Boot sequence 147~~~~~~~~~~~~~ 148 149The boot sequence of TF-A is the usual one except when booting an AArch32 150kernel. In that case, BL33 is booted in AArch32 Hypervisor mode so that it 151can jump to the kernel in the same mode and let it take over that privilege 152level. If BL33 was running in EL2 in AArch64 (as in the default bootflow of 153TF-A) it could only jump to the kernel in AArch32 in Supervisor mode. 154 155The `Linux kernel tree`_ has instructions on how to jump to the Linux kernel 156in ``Documentation/arm/Booting`` and ``Documentation/arm64/booting.txt``. The 157bootstrap should take care of this. 158 159This port support a direct boot of the Linux kernel from the firmware (as a BL33 160image). Alternatively, U-Boot or other bootloaders may be used. 161 162Secondary cores 163~~~~~~~~~~~~~~~ 164 165This port of the Trusted Firmware-A supports ``PSCI_CPU_ON``, 166``PSCI_SYSTEM_RESET`` and ``PSCI_SYSTEM_OFF``. The last one doesn't really turn 167the system off, it simply reboots it and asks the VideoCore firmware to keep it 168in a low power mode permanently. 169 170The kernel used by `Raspbian`_ doesn't have support for PSCI, so it is needed to 171use mailboxes to trap the secondary cores until they are ready to jump to the 172kernel. This mailbox is located at a different address in the AArch32 default 173kernel than in the AArch64 kernel. 174 175Kernels with PSCI support can use the PSCI calls instead for a cleaner boot. 176 177Also, this port of TF-A has another Trusted Mailbox in Shared BL RAM. During 178cold boot, all secondary cores wait in a loop until they are given given an 179address to jump to in this Mailbox (``bl31_warm_entrypoint``). 180 181Once BL31 has finished and the primary core has jumped to the BL33 payload, it 182has to call ``PSCI_CPU_ON`` to release the secondary CPUs from the wait loop. 183The payload then makes them wait in another waitloop listening from messages 184from the kernel. When the primary CPU jumps into the kernel, it will send an 185address to the mailbox so that the secondary CPUs jump to it and are recognised 186by the kernel. 187 188Build Instructions 189------------------ 190 191To boot a AArch64 kernel, only the AArch64 toolchain is required. 192 193To boot a AArch32 kernel, both AArch64 and AArch32 toolchains are required. The 194AArch32 toolchain is needed for the AArch32 bootstrap needed to load a 32-bit 195kernel. 196 197The build system concatenates BL1 and the FIP so that the addresses match the 198ones in the memory map. The resulting file is ``armstub8.bin``, located in the 199build folder (e.g. ``build/rpi3/debug/armstub8.bin``). To know how to use this 200file, follow the instructions in `Setup SD card`_. 201 202The following build options are supported: 203 204- ``RPI3_BL33_IN_AARCH32``: This port can load a AArch64 or AArch32 BL33 image. 205 By default this option is 0, which means that TF-A will jump to BL33 in EL2 206 in AArch64 mode. If set to 1, it will jump to BL33 in Hypervisor in AArch32 207 mode. 208 209- ``PRELOADED_BL33_BASE``: Used to specify the address of a BL33 binary that has 210 been preloaded by any other system than using the firmware. ``BL33`` isn't 211 needed in the build command line if this option is used. Specially useful 212 because the file ``kernel8.img`` can be loaded anywhere by modifying the file 213 ``config.txt``. It doesn't have to contain a kernel, it could have any 214 arbitrary payload. 215 216- ``RPI3_DIRECT_LINUX_BOOT``: Disabled by default. Set to 1 to enable the direct 217 boot of the Linux kernel from the firmware. Option ``RPI3_PRELOADED_DTB_BASE`` 218 is mandatory when the direct Linux kernel boot is used. Options 219 ``PRELOADED_BL33_BASE`` will most likely be needed as well because it is 220 unlikely that the kernel image will fit in the space reserved for BL33 images. 221 This option can be combined with ``RPI3_BL33_IN_AARCH32`` in order to boot a 222 32-bit kernel. The only thing this option does is to set the arguments in 223 registers x0-x3 or r0-r2 as expected by the kernel. 224 225- ``RPI3_PRELOADED_DTB_BASE``: Auxiliary build option needed when using 226 ``RPI3_DIRECT_LINUX_BOOT=1``. This option allows to specify the location of a 227 DTB in memory. 228 229- ``RPI3_RUNTIME_UART``: Indicates whether the UART should be used at runtime 230 or disabled. ``-1`` (default) disables the runtime UART. Any other value 231 enables the default UART (currently UART1) for runtime messages. 232 233- ``RPI3_USE_UEFI_MAP``: Set to 1 to build ATF with the altername memory 234 mapping required for an UEFI firmware payload. These changes are needed 235 to be able to run Windows on ARM64. This option, which is disabled by 236 default, results in the following memory mappings: 237 238:: 239 240 0x00000000 +-----------------+ 241 | ROM | BL1 242 0x00010000 +-----------------+ 243 | DTB | (Loaded by the VideoCore) 244 0x00020000 +-----------------+ 245 | FIP | 246 0x00030000 +-----------------+ 247 | | 248 | UEFI PAYLOAD | 249 | | 250 0x00200000 +-----------------+ 251 | Secure SRAM | BL2, BL31 252 0x00300000 +-----------------+ 253 | Secure DRAM | BL32 (Secure payload) 254 0x00400000 +-----------------+ 255 | | 256 | | 257 | Non-secure DRAM | BL33 258 | | 259 | | 260 0x01000000 +-----------------+ 261 | | 262 | ... | 263 | | 264 0x3F000000 +-----------------+ 265 | I/O | 266 267- ``BL32``: This port can load and run OP-TEE. The OP-TEE image is optional. 268 Please use the code from `here <https://github.com/OP-TEE/optee_os>`__. 269 Build the Trusted Firmware with option ``BL32=tee-header_v2.bin 270 BL32_EXTRA1=tee-pager_v2.bin BL32_EXTRA2=tee-pageable_v2.bin`` 271 to put the binaries into the FIP. 272 273 .. warning:: 274 If OP-TEE is used it may be needed to add the following options to the 275 Linux command line so that the USB driver doesn't use FIQs: 276 ``dwc_otg.fiq_enable=0 dwc_otg.fiq_fsm_enable=0 dwc_otg.nak_holdoff=0``. 277 This will unfortunately reduce the performance of the USB driver. It is 278 needed when using Raspbian, for example. 279 280- ``TRUSTED_BOARD_BOOT``: This port supports TBB. Set this option to 1 to enable 281 it. In order to use TBB, you might want to set ``GENERATE_COT=1`` to let the 282 contents of the FIP automatically signed by the build process. The ROT key 283 will be generated and output to ``rot_key.pem`` in the build directory. It is 284 able to set ROT_KEY to your own key in PEM format. Also in order to build, 285 you need to clone mbed TLS from `here <https://github.com/ARMmbed/mbedtls>`__. 286 ``MBEDTLS_DIR`` must point at the mbed TLS source directory. 287 288- ``ENABLE_STACK_PROTECTOR``: Disabled by default. It uses the hardware RNG of 289 the board. 290 291The following is not currently supported: 292 293- AArch32 for TF-A itself. 294 295- ``EL3_PAYLOAD_BASE``: The reason is that you can already load anything to any 296 address by changing the file ``armstub8.bin``, so there's no point in using 297 TF-A in this case. 298 299- ``MULTI_CONSOLE_API=0``: The multi console API must be enabled. Note that the 300 crash console uses the internal 16550 driver functions directly in order to be 301 able to print error messages during early crashes before setting up the 302 multi console API. 303 304Building the firmware for kernels that don't support PSCI 305~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 306 307This is the case for the 32-bit image of Raspbian, for example. 64-bit kernels 308always support PSCI, but they may not know that the system understands PSCI due 309to an incorrect DTB file. 310 311First, clone and compile the 32-bit version of the `Raspberry Pi 3 TF-A 312bootstrap`_. Choose the one needed for the architecture of your kernel. 313 314Then compile TF-A. For a 32-bit kernel, use the following command line: 315 316.. code:: shell 317 318 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 319 RPI3_BL33_IN_AARCH32=1 \ 320 BL33=../rpi3-arm-tf-bootstrap/aarch32/el2-bootstrap.bin 321 322For a 64-bit kernel, use this other command line: 323 324.. code:: shell 325 326 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 327 BL33=../rpi3-arm-tf-bootstrap/aarch64/el2-bootstrap.bin 328 329However, enabling PSCI support in a 64-bit kernel is really easy. In the 330repository `Raspberry Pi 3 TF-A bootstrap`_ there is a patch that can be applied 331to the Linux kernel tree maintained by the Raspberry Pi foundation. It modifes 332the DTS to tell the kernel to use PSCI. Once this patch is applied, follow the 333instructions in `AArch64 kernel build instructions`_ to get a working 64-bit 334kernel image and supporting files. 335 336Building the firmware for kernels that support PSCI 337~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 338 339For a 64-bit kernel: 340 341.. code:: shell 342 343 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 344 PRELOADED_BL33_BASE=0x02000000 \ 345 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 346 RPI3_DIRECT_LINUX_BOOT=1 347 348For a 32-bit kernel: 349 350.. code:: shell 351 352 CROSS_COMPILE=aarch64-linux-gnu- make PLAT=rpi3 \ 353 PRELOADED_BL33_BASE=0x02000000 \ 354 RPI3_PRELOADED_DTB_BASE=0x01000000 \ 355 RPI3_DIRECT_LINUX_BOOT=1 \ 356 RPI3_BL33_IN_AARCH32=1 357 358AArch64 kernel build instructions 359--------------------------------- 360 361The following instructions show how to install and run a AArch64 kernel by 362using a SD card with the default `Raspbian`_ install as base. Skip them if you 363want to use the default 32-bit kernel. 364 365Note that this system won't be fully 64-bit because all the tools in the 366filesystem are 32-bit binaries, but it's a quick way to get it working, and it 367allows the user to run 64-bit binaries in addition to 32-bit binaries. 368 3691. Clone the `Linux tree fork`_ maintained by the Raspberry Pi Foundation. To 370 speed things up, do a shallow clone of the desired branch. 371 372.. code:: shell 373 374 git clone --depth=1 -b rpi-4.18.y https://github.com/raspberrypi/linux 375 cd linux 376 3772. Configure and compile the kernel. Adapt the number after ``-j`` so that it is 378 1.5 times the number of CPUs in your computer. This may take some time to 379 finish. 380 381.. code:: shell 382 383 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- bcmrpi3_defconfig 384 make -j 6 ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- 385 3863. Copy the kernel image and the device tree to the SD card. Replace the path 387 by the corresponding path in your computers to the ``boot`` partition of the 388 SD card. 389 390.. code:: shell 391 392 cp arch/arm64/boot/Image /path/to/boot/kernel8.img 393 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b.dtb /path/to/boot/ 394 cp arch/arm64/boot/dts/broadcom/bcm2710-rpi-3-b-plus.dtb /path/to/boot/ 395 3964. Install the kernel modules. Replace the path by the corresponding path to the 397 filesystem partition of the SD card on your computer. 398 399.. code:: shell 400 401 make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- \ 402 INSTALL_MOD_PATH=/path/to/filesystem modules_install 403 4045. Follow the instructions in `Setup SD card`_ except for the step of renaming 405 the existing ``kernel7.img`` (we have already copied a AArch64 kernel). 406 407Setup SD card 408------------- 409 410The instructions assume that you have an SD card with a fresh install of 411`Raspbian`_ (or that, at least, the ``boot`` partition is untouched, or nearly 412untouched). They have been tested with the image available in 2018-03-13. 413 4141. Insert the SD card and open the ``boot`` partition. 415 4162. Rename ``kernel7.img`` to ``kernel8.img``. This tricks the VideoCore 417 bootloader into booting the Arm cores in AArch64 mode, like TF-A needs, 418 even though the kernel is not compiled for AArch64. 419 4203. Copy ``armstub8.bin`` here. When ``kernel8.img`` is available, The VideoCore 421 bootloader will look for a file called ``armstub8.bin`` and load it at 422 address **0x0** instead of a predefined one. 423 4244. To enable the serial port "Mini UART" in Linux, open ``cmdline.txt`` and add 425 ``console=serial0,115200 console=tty1``. 426 4275. Open ``config.txt`` and add the following lines at the end (``enable_uart=1`` 428 is only needed to enable debugging through the Mini UART): 429 430:: 431 432 enable_uart=1 433 kernel_address=0x02000000 434 device_tree_address=0x01000000 435 436If you connect a serial cable to the Mini UART and your computer, and connect 437to it (for example, with ``screen /dev/ttyUSB0 115200``) you should see some 438text. In the case of an AArch32 kernel, you should see something like this: 439 440:: 441 442 NOTICE: Booting Trusted Firmware 443 NOTICE: BL1: v1.4(release):v1.4-329-g61e94684-dirty 444 NOTICE: BL1: Built : 00:09:25, Nov 6 2017 445 NOTICE: BL1: Booting BL2 446 NOTICE: BL2: v1.4(release):v1.4-329-g61e94684-dirty 447 NOTICE: BL2: Built : 00:09:25, Nov 6 2017 448 NOTICE: BL1: Booting BL31 449 NOTICE: BL31: v1.4(release):v1.4-329-g61e94684-dirty 450 NOTICE: BL31: Built : 00:09:25, Nov 6 2017 451 [ 0.266484] bcm2835-aux-uart 3f215040.serial: could not get clk: -517 452 453 Raspbian GNU/Linux 9 raspberrypi ttyS0 454 raspberrypi login: 455 456Just enter your credentials, everything should work as expected. Note that the 457HDMI output won't show any text during boot. 458 459.. _default Arm stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub7.S 460.. _default AArch64 stub: https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S 461.. _Linux kernel tree: https://github.com/torvalds/linux 462.. _Linux tree fork: https://github.com/raspberrypi/linux 463.. _Raspberry Pi 3: https://www.raspberrypi.org/products/raspberry-pi-3-model-b/ 464.. _Raspberry Pi 3 TF-A bootstrap: https://github.com/AntonioND/rpi3-arm-tf-bootstrap 465.. _Raspberry Pi 3 documentation: https://www.raspberrypi.org/documentation/ 466.. _Raspbian: https://www.raspberrypi.org/downloads/raspbian/ 467