1--------------------------------------------------------------------------
2Disclaimer: This branch contains prototype code which is for demonstrative
3purposes only and to serve as proof of concept. It is designed to allow
4prototyping of new features and any productization paths if taken
5forward would be delivered through the master branch.
6--------------------------------------------------------------------------
7
8Getting Started
9===============
10
11These instructions will help to recreate the FF-A prototype running the SPMC in EL3.
12
13The end goal is to run a set of tests from a FF-A Test Driver via a userspace application
14to demonstrate functionality of the SPMC with the help of a single bare metal MP
15Partition running in S-EL1 and a logical partition running in EL3 alongside the SPMC on
16the AEM FVP Platform.
17
18Steps to clone the relevant repos
19=================================
20
21Note: These set of instructions assume you have a "Base Directory" where the
22various repositories can be checked out to.
23
24BASE_DIR=$(pwd)
25mkdir $BASE_DIR/out
26
27ATF_FFA_REL_PROTO_REPO=https://git.trustedfirmware.org/TF-A/trusted-firmware-a.git
28ATF_FFA_EL3_SPMC_PROTO_BRANCH=topics/ffa_el3_spmc
29
30LINUX_FFA_EL3_SPMC_PROTO_REPO=https://git.kernel.org/pub/scm/linux/kernel/git/sudeep.holla/linux.git
31LINUX_FFA_EL3_SPMC_PROTO_BRANCH=ffa_el3_spmc
32
33
34# Clone Trusted Firmware-A Repo:
35git clone $ATF_FFA_REL_PROTO_REPO trusted-firmware-a
36cd $BASE_DIR/trusted-firmware-a
37git checkout $ATF_FFA_EL3_SPMC_PROTO_BRANCH
38cd $BASE_DIR
39
40# Clone Linux Kernel
41git clone $LINUX_FFA_EL3_SPMC_PROTO_REPO linux_kernel
42cd $BASE_DIR/linux_kernel
43git checkout $LINUX_FFA_EL3_SPMC_PROTO_BRANCH
44cd $BASE_DIR
45
46
47
48Prerequisites
49=============
50To exercise the tests in this prototype we require some prebuilt binaries. In testing these binaries have
51been sourced from the OP-TEE build system however any such images should work. For the purposes of these
52instructions they assume the following binaries are available in a $PREBUILTS_PATH directory:
53
54- EDK2 binary (FVP_AARCH64_EFI.fd)
55- Linux filesystem (rootfs.cpio.gz)
56- UEFI ram disk (boot-fat.uefi.img)
57
58Download FVP
59------------
60
61Download Armv8-A Base RevC AEM FVP from:
62https://developer.arm.com/tools-and-software/simulation-models/fixed-virtual-platforms/arm-ecosystem-models
63
64Extract into the `fvp` directory (the final file path should be
65$BASE_DIR/fvp/Base_RevC_AEMv8A_pkg/...)
66
67Toolchains
68----------
69The instructions expects aarch64 toolchain to be make available with the use of the CROSS_COMPILE variable e.g.
70export CROSS_COMPILE=aarch64-linux-gnu-
71
72Tested with gcc version 9.2.1 20191025
73
74
75Build Instructions
76==================
77
78TF-A SPMC / TSP
79---------------
80# Set our file paths appropriately.
81export TF_A_PATH=$BASE_DIR/trusted-firmware-a
82export OUT=$BASE_DIR/out
83
84export BL33_BIN=$PREBUILTS_PATH/FVP_AARCH64_EFI.fd
85export SPMC_MANIFEST_FILE=$TF_A_PATH/plat/arm/board/fvp/fdts/fvp_spmc_el3_manifest.dts
86
87export LOG_LEVEL=40 (Default info output, additional logging e.g. SMC prints available at 50 (Verbose))
88
89make realclean
90
91make -j16 CROSS_COMPILE=$CROSS_COMPILE \
92SPD=spmd \
93SPMC_AT_EL3=1 \
94CTX_INCLUDE_EL2_REGS=0 \
95SPMD_SPM_AT_SEL2=0 \
96EL3_EXCEPTION_HANDLING=1 \
97PLAT=fvp \
98BL33=$BL33_BIN \
99DEBUG=1 \
100LOG_LEVEL=$LOG_LEVEL \
101ARM_SPMC_MANIFEST_DTS=$SPMC_MANIFEST_FILE \
102ARM_ARCH_MINOR=4 \
103FVP_DT_PREFIX=fvp-base-gicv3-psci-1t \
104all fip
105
106# Copy our binaries and DTB to our output folder
107cp $TF_A_PATH/build/fvp/debug/fip.bin $OUT/fip.bin
108cp $TF_A_PATH/build/fvp/debug/bl1.bin $OUT/bl1.bin
109cp $TF_A_PATH/build/fvp/debug/bl31.bin $OUT/TSP.bin
110cp $$TF_A_PATH/build/fvp/debug/fdts/fvp-base-gicv3-psci-1t.dtb fvp-base-gicv3-psci-1t.dtb $OUT/fvp-base-gicv3-psci-1t.dtb
111
112
113Linux
114------
115Ensure that FFA support is enabled in the kernel config (CONFIG_ARM_FFA_TRANSPORT=y).
116
117make ARCH=arm64 CROSS_COMPILE=$CROSS_COMPILE defconfig -j$NO_CPUS
118make ARCH=arm64 CROSS_COMPILE=$CROSS_COMPILE -j$NO_CPUS
119
120cp arch/arm64/boot/Image $OUT/
121
122Assembling Images
123-----------------
124 - Copy the prebuilt userspace test binary ($TF_A_PATH/ioctl) to your filesystem. This is requried to interact
125   with the test FFA driver.
126 - Copy the modified filesystem image, linux kernel and FVP DTB to the UEFI ram disk ($OUT/boot-fat.uefi.img)
127
128
129Launching the FVP
130=================
131The FVP can then be launched with the following command. You should be able to observe output from the
132normal world, trusted firmware and S-EL1 TSP each on terminals 0, 1 & 2 respectively.
133
134$BASE_DIR/fvp/Base_RevC_AEMv8A_pkg/models/Linux64_GCC-6.4/FVP_Base_RevC-2xAEMvA \
135-C pctl.startup=0.0.0.0 \
136-C bp.secure_memory=1 \
137-C cluster0.NUM_CORES=4 \
138-C cluster1.NUM_CORES=4 \
139-C cache_state_modelled=0 \
140-C bp.pl011_uart0.untimed_fifos=1 \
141-C bp.pl011_uart0.unbuffered_output=1 \
142-C bp.pl011_uart1.untimed_fifos=1 \
143-C bp.pl011_uart1.unbuffered_output=1 \
144-C bp.pl011_uart0.out_file=$OUT/uart0.log \
145-C bp.pl011_uart1.out_file=$OUT/uart1.log \
146-C bp.pl011_uart2.out_file=$OUT/uart2.log \
147-C bp.pl011_uart3.out_file=$OUT/uart3.log \
148-C bp.vis.disable_visualisation=0 \
149-C bp.secureflashloader.fname=$OUT/bl1.bin \
150-C bp.flashloader0.fname=$OUT/fip.bin \
151-C bp.ve_sysregs.mmbSiteDefault=0 \
152-C bp.ve_sysregs.exit_on_shutdown=1 \
153-C cluster0.has_arm_v8-4=1 \
154-C cluster1.has_arm_v8-4=1 \
155-C bp.virtioblockdevice.image_path=$OUT/boot-fat.uefi.img
156
157
158Running the tests
159=================
160Once a shell has been reached the `ioctl` userspace application can be used to run some basic tests to verify
161the system has started correctly.
162
163The following tests are available:
164
165    1) Echo a message to the Physical SP and Back using direct req/resp
166    2) Send a direct request to the physical SP and ask it to send it's
167       own direct_req to the Logical SP and relay the message back again.
168    3) Share a single page of memory with the SP and ensure the contents
169       can be modified.
170    4) Share multiple pages requiring the memory descriptor to be split
171       over multiple invocations.
172    5) Lend a single page of memory with the SP and ensure the contents
173       can be modified.
174    6) Lend multiple pages requiring the memory descriptor to be split
175       over multiple invocations.
176    255) Run all of the above test cases.
177
178Usage: /path/to/ioctl <test_num>
179
180
181Example Output
182==============
183
184Booting
185-------
186Expected output from TF-A on uart0:
187```
188INFO:    BL31: Initialising Exception Handling Framework
189INFO:    BL31: Initializing runtime services
190INFO:    Logical Secure Partition init start.
191INFO:    LSP: Init function called.
192INFO:    Secure Partition initialized.
193INFO:    Secure Partition context setup start.
194INFO:    Secure Partition setup done.
195INFO:    BL31: Initializing logical partition structures
196INFO:    BL31: Initializing BL32
197INFO:    Secure Partition (0x8001) init start.
198INFO:    S-EL1 SP context on core0 is in 1 state
199INFO:    S-EL1 SP context on core0 is in 0 state
200INFO:    Secure Partition initialized.
201INFO:    BL31: Preparing for EL3 exit to normal world
202INFO:    Entry point address = 0x88000000
203INFO:    SPSR = 0x3c9
204UEFI firmware (version  built at 08:58:20 on Oct  1 2019)
205...
206```
207Expected output from the SPMC on uart1:
208```
209INFO:    SP (0x8001) init start on core1.
210INFO:    SP (0x8001) init start on core2.
211INFO:    SP (0x8001) init start on core3.
212INFO:    SP (0x8001) init start on core4.
213INFO:    SP (0x8001) init start on core5.
214INFO:    SP (0x8001) init start on core6.
215INFO:    SP (0x8001) init start on core7.
216```
217
218
219Expected output from the TSP on uart2:
220```
221NOTICE:  TSP: v2.5(debug):v2.5-453-gc6a2750df
222NOTICE:  TSP: Built : 17:23:51, Oct  5 2021
223INFO:    TSP: Total memory base : 0xff200000
224INFO:    TSP: Total memory size : 0x1b000 bytes
225INFO:    TSP FF-A endpoint id = 0x8001
226INFO:    TSP: cpu 0x81000000: 1 smcs, 1 erets 1 cpu on requests
227INFO:    TSP: cpu 0x81000100 turned on
228INFO:    TSP: cpu 0x81000100: 1 smcs, 1 erets 1 cpu on requests
229INFO:    TSP: cpu 0x81000200 turned on
230INFO:    TSP: cpu 0x81000200: 1 smcs, 1 erets 1 cpu on requests
231INFO:    TSP: cpu 0x81000300 turned on
232INFO:    TSP: cpu 0x81000300: 1 smcs, 1 erets 1 cpu on requests
233INFO:    TSP: cpu 0x81010000 turned on
234INFO:    TSP: cpu 0x81010000: 1 smcs, 1 erets 1 cpu on requests
235INFO:    TSP: cpu 0x81010100 turned on
236INFO:    TSP: cpu 0x81010100: 1 smcs, 1 erets 1 cpu on requests
237INFO:    TSP: cpu 0x81010200 turned on
238INFO:    TSP: cpu 0x81010200: 1 smcs, 1 erets 1 cpu on requests
239INFO:    TSP: cpu 0x81010300 turned on
240INFO:    TSP: cpu 0x81010300: 1 smcs, 1 erets 1 cpu on requests
241```
242
243Test Cases
244----------
245Linux Output:
246```
247# ./ioctl 255
248[15284.959068] Completed Test Case: 1
249[15284.959155] Test Executed Successfully
250[15284.959155]
251[15284.959355] Completed Test Case: 2
252[15284.959477] Test Executed Successfully
253[15284.959477]
254[15284.961055] Verified 1 constituents successfully
255[15284.961155] Completed Test Case: 3
256[15284.961255] Test Executed Successfully
257[15284.961255]
258[15285.219655] Verified 256 constituents successfully
259[15285.219841] Completed Test Case: 4
260[15285.220055] Test Executed Successfully
261[15285.220055]
262[15285.221510] Verified 1 constituents successfully
263[15285.221621] Completed Test Case: 5
264[15285.221750] Test Executed Successfully
265[15285.221750]
266[15285.479955] Verified 256 constituents successfully
267[15285.480088] Completed Test Case: 6
268[15285.480255] Test Executed Successfully
269[15285.480255]
270[15285.480355] 0 Tests Failed
271[15285.480355]
272[15285.480555] Exiting Test Application -  Total Failures: 0
273```
274
275S-EL1 SP Output:
276```
277INFO:    TSP Tests: echo message--
278INFO:    TSP Tests: Relaying message--
279INFO:    TSP Tests: Memory Share Request--
280INFO:    TSP Tests: Memory Share Request--
281INFO:    TSP Tests: Memory Lend Request--
282INFO:    TSP Tests: Memory Lend Request--
283```
284
285