Date Archives February 2019

Trusted Thread Scheduling in OP-TEE

Trusted thread for standard services (standard SMC)

  • Terminates when optee_os returns to the normal world with a service completion status.
  • Can be interrupted by
    • Foreign interrupt (non-secure): optee_os suspends the trusted thread and invokes the normal world through the Monitor (RPC services). The trusted threads will resume only once normal world invokes the optee_os with the RPC service status.
    • Native interrupt (secure): Native interrupt is handled by the interrupt exception handler. Once served, optee_os then returns to the execution of trusted thread.
  • Can lead optee_os to invoke a service in normal world (e.g., access a file, get the REE current time, etc.).
    • Trusted thread is suspended/resumed for the remote service execution.


Scheduling consideration

  • When interrupted by foreign interrupt or optee_os invokes a normal world service, the normal world gets the opportunity to reschedule the running application. The trusted thread can resume only once the client application is scheduled back. That means a trusted thread execution follows the scheduling of the normal world.
  • optee_os does not implement any thread scheduling. Each trusted thread is expected to track a service that is invoked from the normal world and should return to it with an execution status.
  • Linux thread invoking OP-TEE gets assigned a trusted thread on TEE. So trusted threads are scheduled by the linux kernel.


Trusted thread constraints

  • TEE handles a static number of trusted thread (CFG_NUM_THREADS).
  • Trusted threads are only expensive on memory constrained system, mainly regarding the execution stack size.
  • On SMP (Systematic Multi-Processing) system, optee_os can execute several trusted threads in parallel if the normal world supports scheduling of processes. UP system, supporting several trusted threads in optee_os, also helps normal world scheduler to be efficient.




Interrupt Handling in OP-TEE


  • World switches through Secure Monitor Call (SMC).
  • A secure interrupt is signaled by the ARM GIC (Generic Interrupt Controller).
  • There are two types of interrupts.
    • IRQ: Foreign interrupt (non-secure)
    • FIQ: Native interrupt (secure)
  • Each world has own interrupt exception vector.
  • When an interrupt is in same world, it directly handles the interrupt.
  • When an interrupt is in different world, it switches a context (by Monitor vector) to the corresponding world first and then handles the interrupt.

Overview of Interrupt Handling


Standard SMC

Normal world invokes optee_os using SMC

  • On every context switching, it saves a state of current world and restores a previous state of an other world. (normal – secure)
  • Fast SMC: Blocks all IRQ/FIQ exception until it returns back to normal world.
  • Standard SMC: After assigning a trusted thread (core/arch/arm/kernel/thread.c),
  • Both fast SMC and standard SMC end on the entry stack with IRQ/FIQ blocked.

SMC entry to secure world


Multiple cases of IRQ & FIQ

  • Explanations on this section is based on GICv2, and details about GICv3 are discussed on the bottom of this post.

Non-secure interrupts (IRQ) on Secure World (SCR_NS = 0)

  1. Saves trusted thread context
  2. Blocks all interrupt (IRQ and FIQ)
  3. Switches to entry stack
  4. Restores normal world context with a return code indicating that an IRQ is about to be delivered
  5. After handling IRQ, normal world issues a new SMC to return and to finish last SMC.

IRQ received in secure world and forwarded to normal world


Non-secure interrupts (IRQ) on Normal World (SCR_NS = 1)

  • IRQ will be delivered using the state vector (VBAR) in the normal world.
  • The monitor and the Trusted OS are not involved at all.


Secure interrupts (FIQ) on Normal World (SCR_NS = 1)

  1. Saves normal world context and restores previous secure world context
  2. Clears SCR_FIQ when clearing SCR_NS
  3. Sets “FIQ” as parameter to secure world entry and returns to secure world
  4. Secure world unmasks FIQs because of the “FIQ” parameter.
  5. FIQ is received as an exception in the state vector, and the state vector handles and returns the exception.
  6. Secure world issues an SMC to return to normal world
  7. Monitor saves secure world context and restores normal world context.
  8. Return from exception to normal world

FIQ received when SCR_NS is set


Secure interrupts (FIQ) on Secure World (SCR_NS = 0)

  • FIQ will be delivered using the state vector (VBAR) in the secure world.
  • The monitor is not involved at all.


Secure Interrupts (FIQ) received while processing Non-secure Interrupts (IRQ) forwarded from Secure World

  • FIQ has higher priority than IRQ.
  • While processing IRQ, the context switches back to secure world to handle FIQ. After FIQ is completely handled, it switches back to normal world to finish IRQ.

FIQ received while processing an IRQ forwarded from secure world


Generic Interrupt Controller (GIC)

GIC is an architected resource that supports and controls interrupts.

  • GICv2: supports ARMv7
  • GICv3: supports ARMv8-A (Hikey960)
    • New features added to scale a large system (GICv2 handles only 8 processing elements)
      • Affinity routing (Interrupt routing)
      • Redistributor (Interrupt distribution)
  • GICv4: extension of GICv3


Important differences between GICv2 and GICv3

  • GICv2: native interrupt is sent as FIQ and foreign interrupt is sent as IRQ.
  • GICv3: foreign interrupt is sent as FIQ which could be handled by either secure world (aarch32 Monitor mode or aarch64 EL3) or normal world. ARM GICv3 mode can be enabled by setting (CFG_ARM_CIV3=y).