TfLiteContext Struct Reference

# TfLiteContext

`#include <common.h>`

`TfLiteContext` allows an op to access the tensors.

## Summary

`TfLiteContext` is a struct that is created by the TF Lite runtime and passed to the "methods" (C function pointers) in the `TfLiteRegistration` struct that are used to define custom ops and custom delegate kernels. It contains information and methods (C function pointers) that can be called by the code implementing a custom op or a custom delegate kernel. These methods provide access to the context in which that custom op or custom delegate kernel occurs, such as access to the input and output tensors for that op, as well as methods for allocating memory buffers and intermediate tensors, etc.

See also `TfLiteOpaqueContext`, which is an more ABI-stable equivalent.

### Public attributes
`AcquireSubgraphContext)(struct TfLiteContext *context, int subgraph_index, struct TfLiteContext **acquired_context)` | `TfLiteStatus(*` Retrieves the corresponding TfLiteContext of a subgraph that the given subgraph_index points to and switches to the delegate context for that subgraph.
`AddTensors)(struct TfLiteContext *, int tensors_to_add, int *first_new_tensor_index)` | `TfLiteStatus(*` Add `tensors_to_add` tensors, preserving pre-existing Tensor entries.
`AllocateBufferForEval)(struct TfLiteContext *ctx, size_t bytes, void **ptr)` | `TfLiteStatus(*` Allocate a buffer which will be deallocated right after invoke phase.
`AllocatePersistentBuffer)(struct TfLiteContext *ctx, size_t bytes)` | `void *(*` Allocate persistent buffer which has the same life time as the interpreter.
`GetEvalTensor)(const struct TfLiteContext *context, int tensor_idx)` | `TfLiteEvalTensor *(*` Returns a TfLiteEvalTensor struct for a given index.
`GetExecutionPlan)(struct TfLiteContext *context, TfLiteIntArray **execution_plan)` | `TfLiteStatus(*` The execution plan contains a list of the node indices in execution order.
`GetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType)` | `TfLiteExternalContext *(*` Access external contexts by type.
`GetModelMetadata)(const struct TfLiteContext *context, const char *name, const char **ptr, size_t *bytes)` | `TfLiteStatus(*` Retrieves named metadata buffer from the TFLite model.
`GetNodeAndRegistration)(struct TfLiteContext *, int node_index, TfLiteNode **node, struct TfLiteRegistration **registration)` | `TfLiteStatus(*` Get a Tensor node by node_index.
`GetScratchBuffer)(struct TfLiteContext *ctx, int buffer_idx)` | `void *(*` Get the scratch buffer pointer.
`GetTensor)(const struct TfLiteContext *context, int tensor_idx)` | `TfLiteTensor *(*` Returns a TfLiteTensor struct for a given index.
`PreviewDelegatePartitioning)(struct TfLiteContext *context, const TfLiteIntArray *nodes_to_replace, TfLiteDelegateParams **partition_params_array, int *num_partitions)` | `TfLiteStatus(*` This method provides a preview of post-delegation partitioning.
`ReleaseSubgraphContext)(struct TfLiteContext *context, int subgraph_index)` | `TfLiteStatus(*` Releases the subgraph context by switching back to the TFLite kernel context for the subgraph that the given subgraph_index points to.
`ReplaceNodeSubsetsWithDelegateKernels)(struct TfLiteContext *, struct TfLiteRegistration registration, const TfLiteIntArray *nodes_to_replace, struct TfLiteDelegate *delegate)` | `TfLiteStatus(*` Replace ops with one or more stub delegate operations.
`ReportError)(struct TfLiteContext *, const char *msg,...)` | `void(*` Request that an error be reported with format string msg.
`RequestScratchBufferInArena)(struct TfLiteContext *ctx, size_t bytes, int *buffer_idx)` | `TfLiteStatus(*` Request a scratch buffer in the arena through static memory planning.
`ResizeTensor)(struct TfLiteContext *, TfLiteTensor *tensor, TfLiteIntArray *new_size)` | `TfLiteStatus(*` Request memory pointer be resized.
`ResizeTensorExplicit)(struct TfLiteContext *ctx, TfLiteTensor *tensor, int dims, const int *shape)` | `TfLiteStatus(*` Resize the memory pointer of the `tensor`.
`SetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType, TfLiteExternalContext *)` | `void(*` Set the value of a external context.
`allow_fp32_relax_to_fp16` | `bool` Flag for allowing float16 precision for FP32 calculation.
`impl_` | `void *` opaque full context ptr (an opaque c++ data structure)
`profiler` | `void *` Pointer to the op-level profiler, if set; nullptr otherwise.
`recommended_num_threads` | `int` Number of threads that are recommended to subsystems like gemmlowp and eigen.
`tensors` | `TfLiteTensor *` An array of tensors in the interpreter context (of length `tensors_size`)
`tensors_size` | `size_t` Number of tensors in the context.

## Public attributes

### AcquireSubgraphContext

    TfLiteStatus(* TfLiteContext::AcquireSubgraphContext)(struct TfLiteContext *context, int subgraph_index, struct TfLiteContext **acquired_context)

Retrieves the corresponding TfLiteContext of a subgraph that the given subgraph_index points to and switches to the delegate context for that subgraph.

If an invalid subgraph index is given, returns kTfLiteError.

NOTE: This function is expected to be paired with ReleaseSubgraphContext() once the delegate preparation is done and/or the delegate context functions are no longer needed.

WARNING: This is an experimental interface that is subject to change.

### AddTensors

    TfLiteStatus(* TfLiteContext::AddTensors)(struct TfLiteContext *, int tensors_to_add, int *first_new_tensor_index)

Add `tensors_to_add` tensors, preserving pre-existing Tensor entries.

If non-null, the value pointed to by `first_new_tensor_index` will be set to the index of the first new tensor.

### AllocateBufferForEval

    TfLiteStatus(* TfLiteContext::AllocateBufferForEval)(struct TfLiteContext *ctx, size_t bytes, void **ptr)

Allocate a buffer which will be deallocated right after invoke phase.

The memory is allocated from heap in TFL, and from volatile arena in TFLM. This method is only available in invoke stage.

NOTE: If possible use `RequestScratchBufferInArena` method to avoid memory allocation during inference time.

WARNING: This is an experimental interface that is subject to change.

### AllocatePersistentBuffer

    void *(* TfLiteContext::AllocatePersistentBuffer)(struct TfLiteContext *ctx, size_t bytes)

Allocate persistent buffer which has the same life time as the interpreter.

Returns `nullptr` on failure. The memory is allocated from heap for TFL, and from tail in TFLM. This method is only available in `Init` or `Prepare` stage.

WARNING: This is an experimental interface that is subject to change.

### GetEvalTensor

    TfLiteEvalTensor *(* TfLiteContext::GetEvalTensor)(const struct TfLiteContext *context, int tensor_idx)

Returns a TfLiteEvalTensor struct for a given index.

WARNING: This is an experimental interface that is subject to change.

WARNING: This method may not be available on all platforms.

### GetExecutionPlan

    TfLiteStatus(* TfLiteContext::GetExecutionPlan)(struct TfLiteContext *context, TfLiteIntArray **execution_plan)

The execution plan contains a list of the node indices in execution order.

execution_plan->size is the current number of nodes. And, execution_plan->data[0] is the first node that needs to be run. TfLiteDelegates can traverse the current execution plan by iterating through each member of this array and using GetNodeAndRegistration() to access details about a node. i.e.

    TfLiteIntArray* execution_plan;
    TF_LITE_ENSURE_STATUS(context->GetExecutionPlan(context,
                                                    &execution_plan));
    for (int exec_index = 0; exec_index < execution_plan->size;
          exec_index++) {
       int node_index = execution_plan->data[exec_index];
       TfLiteNode* node;
       TfLiteRegistration* reg;
       context->GetNodeAndRegistration(context, node_index, &node, ®);
    }

Note: the memory pointed by '`*execution_plan` is OWNED by TfLite runtime. Future calls to GetExecutionPlan invalidates earlier outputs. The following code snippet shows the issue of such an invocation pattern. After calling CheckNode, subsequent access to `plan_1st` is undefined.

    void CheckNode(const TfLiteNode* node) {
      ...
      TfLiteIntArray* plan_2nd;
      TF_LITE_ENSURE_STATUS(
          context->GetExecutionPlan(context, &plan_2nd)
      );
      ...
    }

    TfLiteIntArray* plan_1st;
    TF_LITE_ENSURE_STATUS(context->GetExecutionPlan(context, &plan_1st));
    for (int exec_index = 0; exec_index < plan_1st->size; exec_index++) {
       int node_index = plan_1st->data[exec_index];
       TfLiteNode* node;
       TfLiteRegistration* reg;
       context->GetNodeAndRegistration(context, node_index, &node, ®);
       CheckNode(node);
    }

WARNING: This is an experimental interface that is subject to change.

### GetExternalContext

    TfLiteExternalContext *(* TfLiteContext::GetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType)

Access external contexts by type.

WARNING: This is an experimental interface that is subject to change.

### GetModelMetadata

    TfLiteStatus(* TfLiteContext::GetModelMetadata)(const struct TfLiteContext *context, const char *name, const char **ptr, size_t *bytes)

Retrieves named metadata buffer from the TFLite model.

Returns kTfLiteOk if metadata is successfully obtained from the flatbuffer Model: that is, there exists a `metadata` entry with given `name` string. (see TFLite's schema.fbs). The corresponding `buffer` information is populated in `ptr` & `bytes`. The data from `ptr` is valid for the lifetime of the Interpreter.

WARNING: This is an experimental interface that is subject to change.

### GetNodeAndRegistration

    TfLiteStatus(* TfLiteContext::GetNodeAndRegistration)(struct TfLiteContext *, int node_index, TfLiteNode **node, struct TfLiteRegistration **registration)

Get a Tensor node by node_index.

WARNING: This is an experimental interface that is subject to change.

### GetScratchBuffer

    void *(* TfLiteContext::GetScratchBuffer)(struct TfLiteContext *ctx, int buffer_idx)

Get the scratch buffer pointer.

This method is only available in Eval stage.

WARNING: This is an experimental interface that is subject to change.

### GetTensor

    TfLiteTensor *(* TfLiteContext::GetTensor)(const struct TfLiteContext *context, int tensor_idx)

Returns a TfLiteTensor struct for a given index.

WARNING: This is an experimental interface that is subject to change.

WARNING: This method may not be available on all platforms.

### PreviewDelegatePartitioning

    TfLiteStatus(* TfLiteContext::PreviewDelegatePartitioning)(struct TfLiteContext *context, const TfLiteIntArray *nodes_to_replace, TfLiteDelegateParams **partition_params_array, int *num_partitions)

This method provides a preview of post-delegation partitioning.

Each TfLiteDelegateParams in the referenced array corresponds to one instance of the delegate kernel. Example usage:

    TfLiteIntArray* nodes_to_replace = ...;
    TfLiteDelegateParams* params_array;
    int num_partitions = 0;
    TF_LITE_ENSURE_STATUS(context->PreviewDelegatePartitioning(
       context, delegate, nodes_to_replace, ¶ms_array,
       &num_partitions));
    for (int idx = 0; idx < num_partitions; idx++) {
       const auto& partition_params = params_array[idx];
       ...
    }

NOTE: The context owns the memory referenced by partition_params_array. It will be cleared with another call to PreviewDelegatePartitioning, or after TfLiteDelegateParams::Prepare returns.

WARNING: This is an experimental interface that is subject to change.

### ReleaseSubgraphContext

    TfLiteStatus(* TfLiteContext::ReleaseSubgraphContext)(struct TfLiteContext *context, int subgraph_index)

Releases the subgraph context by switching back to the TFLite kernel context for the subgraph that the given subgraph_index points to.

NOTE: This function is expected to be used after AcquireSubgraphContext() once the delegate preparation is done and/or the delegate context functions are no longer needed.

WARNING: This is an experimental interface that is subject to change.

### ReplaceNodeSubsetsWithDelegateKernels

    TfLiteStatus(* TfLiteContext::ReplaceNodeSubsetsWithDelegateKernels)(struct TfLiteContext *, struct TfLiteRegistration registration, const TfLiteIntArray *nodes_to_replace, struct TfLiteDelegate *delegate)

Replace ops with one or more stub delegate operations.

This function does not take ownership of `nodes_to_replace`.

### ReportError

    void(* TfLiteContext::ReportError)(struct TfLiteContext *, const char *msg,...)

Request that an error be reported with format string msg.

### RequestScratchBufferInArena

    TfLiteStatus(* TfLiteContext::RequestScratchBufferInArena)(struct TfLiteContext *ctx, size_t bytes, int *buffer_idx)

Request a scratch buffer in the arena through static memory planning.

This method is only available in `Prepare` stage and the buffer is allocated by the interpreter between Prepare and Eval stage. In `Eval` stage, `GetScratchBuffer` API can be used to fetch the address.

WARNING: This is an experimental interface that is subject to change.

### ResizeTensor

    TfLiteStatus(* TfLiteContext::ResizeTensor)(struct TfLiteContext *, TfLiteTensor *tensor, TfLiteIntArray *new_size)

Request memory pointer be resized.

Updates dimensions on the tensor. NOTE: ResizeTensor takes ownership of newSize.

### ResizeTensorExplicit

    TfLiteStatus(* TfLiteContext::ResizeTensorExplicit)(struct TfLiteContext *ctx, TfLiteTensor *tensor, int dims, const int *shape)

Resize the memory pointer of the `tensor`.

This method behaves the same as `ResizeTensor`, except that it makes a copy of the shape array internally so the shape array could be deallocated right afterwards.

WARNING: This is an experimental interface that is subject to change.

### SetExternalContext

    void(* TfLiteContext::SetExternalContext)(struct TfLiteContext *, TfLiteExternalContextType, TfLiteExternalContext *)

Set the value of a external context.

Does not take ownership of the pointer.

WARNING: This is an experimental interface that is subject to change.

### allow_fp32_relax_to_fp16

    bool TfLiteContext::allow_fp32_relax_to_fp16

Flag for allowing float16 precision for FP32 calculation.

default: false.

WARNING: This is an experimental API and subject to change.

### impl_

    void * TfLiteContext::impl_

opaque full context ptr (an opaque c++ data structure)

### profiler

    void * TfLiteContext::profiler

Pointer to the op-level profiler, if set; nullptr otherwise.

### recommended_num_threads

    int TfLiteContext::recommended_num_threads

Number of threads that are recommended to subsystems like gemmlowp and eigen.

### tensors

    TfLiteTensor * TfLiteContext::tensors

An array of tensors in the interpreter context (of length `tensors_size`)

### tensors_size

    size_t TfLiteContext::tensors_size

Number of tensors in the context.