Skip to content

Latest commit

 

History

History
265 lines (203 loc) · 7.58 KB

architecture.md

File metadata and controls

265 lines (203 loc) · 7.58 KB
Raw

OpenVINO Architecture

This guide encompasses existed architectural ideas and guidelines for architecture development.

Main architectual design concepts

OpenVINO development is based on next principals:

  • Performance and Scalability - OpenVINO should provide the optimal performance in different customer cases
  • Availability and Resilience - OpenVINO is high quality product and we should demonstrate this high level in different cases
  • Security - OpenVINO is applied in different applications, it is why we have a strict validation pipelines which allow to find vulnerabilities on the development stage.

Each OpenVINO component is projected with using DOTADIW (Do One Thing And Do It Well) approach, it means that OpenVINO is highly granularity product and we are avoiding mixing of responsibilities for different components.

OpenVINO Component responsibilities

flowchart TB
    subgraph tools [Tools]
        mo{{Model Optimizer}}
        pot{{PoT}}
    
        style mo fill:#6c9f7f
        style pot fill:#6c9f7f
    end
    subgraph tutorials [Tutorials]
        samples[Samples]
        demos[Demos]
        notebooks[Notebooks]
    end
    subgraph api [API]
        cpp[C++]
        subgraph bindings [Bindings]
            c_api[C]
            python_api[Python]
        end
    end
    
    subgraph plugins [OV Plugins]
        auto(["AUTO"])
        cpu(["Intel_CPU"])
        gpu(["Intel_GPU"])
    end
    subgraph frontends [OV Frontends]
        direction TB
        ir_fe["IR Frontend"]
        onnx_fe["ONNX Frontend"]
        paddle_fe["Paddle Frontend"]
    end
    openvino(openvino library)
    
    frontends--Read model--->openvino
    openvino--API--->api
    openvino--infer--->plugins
Loading

All OpenVINO components can be logicaly divided to several groups:

  • OpenVINO library is a core library which provides the OpenVINO API.
  • Bindings are libraries which provide OpenVINO API for different languages.
  • Frontends are libraries which converts model from framework representation to OpenVINO format.
  • Plugins are components which allow to run models on different accelerators.
  • Tools a set of different components which provide additional opportunities for OpenVINO.
  • Tutorials different applications which show how user can work with OpenVINO.

OpenVINO Inference pipeline

OpenVINO inference pipeline has several objects which are responsible for different stages of model inference.

  • Core object is a global context for Model inference, allows to create the unique environment for inference pipeline.
    • read_model() finds the right frontend and converts Framework model to OpenVINO representation.
    • compile_model() loads hardware plugin and compiles model on the device.
  • CompiledModel represents the hardware specified model, this model can contain additional plugin optimization and on this stage model already was compiled for the specific device.
  • InferRequest provides an interface for running inference.
flowchart TB
    subgraph core [ov::Core core]
        read_model["core.read_model()"]
        compile_model["core.compile_model()"]
    end
    fw_model[(Framework model)]
    ov_model[ov::Model]
    
    subgraph compiled_models [Compiled models]
        cpu_model["CPU"]
        gpu_model["GPU"]
        auto_model["AUTO"]
    end
    subgraph infer_reguests [InferRequests]
        cpu_req["CPU"]
        gpu_req["GPU"]
        auto_req["AUTO"]
    end
   
    result[Results]
    
    fw_model--->read_model
    read_model--->ov_model
    ov_model-->compile_model
    
    fw_model--->compile_model
    
    compile_model--"compile_model(CPU)"--->cpu_model
    compile_model--"compile_model(GPU)"--->gpu_model
    compile_model--"compile_model(AUTO)"--->auto_model
    
    cpu_model--"create_infer_request()"--->cpu_req
    gpu_model--"create_infer_request()"--->gpu_req
    auto_model--"create_infer_request()"--->auto_req
    
    cpu_req--"infer()"-->result
    gpu_req--"infer()"-->result
    auto_req--"infer()"-->result
Loading

OpenVINO Components Relationships

The diagram below shows internal components and their relationships inside the OpenVINO dynamic library.

flowchart LR
    subgraph legend [Legend]
    interface_lib[Interface library]
    style interface_lib fill:#af8401

    object_lib[Object library]
    style object_lib fill:#6e9f01

    static_lib[(Static library)]
    style static_lib fill:#01429f
    dynamic_lib([Dynamic library])
    style dynamic_lib fill:#cb6969

    some_api{{Some library API}}
    some_dev_api{{Some developer API}}

    object_lib ===>|Link with| interface_lib

    some_api --->|Depends on headers only| static_lib
    some_api -.->|Include headers| some_dev_api
    end
Loading 
flowchart TB
    util[(openvino::util)]
    style util fill:#01429f

    itt[(openvino::itt)]
    style itt fill:#01429f

    pugixml[(openvino::pugixml)]
    style pugixml fill:#01429f

    builders[(builders)]
    style builders fill:#01429f

    reference[(reference implementations)]
    style reference fill:#01429f

    shape_inference[(shape inference)]
    style shape_inference fill:#01429f

    subgraph core [core]
        style core fill:#6e9f01
        core_api{{Core public API}}

        core_api--->builders
        core_api--->reference
        core_api--->shape_inference
    end

    subgraph core_dev [openvino::core::dev]
        style core_dev fill:#af8401
        core_dev_api{{Core developer API}}
    end

    core ===> core_dev

    core_dev_api-.->openvino_dev_api

    pugixml ===> core
    builders ===> core
    reference ===> core
    shape_inference ===> core
    util ===> core
    itt ===> core

    subgraph inference [inference]
        style inference fill:#6e9f01
        inference_api{{Inference public API}}
        inference_dev_api{{Infernce developer API}}

        inference_api-.->inference_dev_api
    end

    core ===> inference
    util ===> inference
    itt ===> inference

    inference_dev_api-.->openvino_dev_api

    subgraph frontend_common [frontend_common]
        style frontend_common fill:#6e9f01
        frontend_api{{Frontend API}}
    end

    core_dev ===> frontend_common
    util ===> frontend_common


    frontend_common ---> core_dev


    subgraph transformations [OpenVINO transformations]
        style transformations fill:#6e9f01
        transformations_api{{Transformations API}}
    end

    reference ===> transformations
    builders ===> transformations
    shape_inference ===> transformations
    itt ===> transformations
    core_dev ===> transformations

    transformations ---> core_dev

    subgraph lp_transformations [OpenVINO LP transformations]
        style lp_transformations fill:#6e9f01
        lp_transformations_api{{LP transformations API}}
    end

    itt ===> lp_transformations
    transformations ---> lp_transformations
    core ---> lp_transformations


    subgraph openvino [openvino::runtime]
        style openvino fill:#cb6969
        openvino_api{{OpenVINO Public API}}
    end

    transformations ===> openvino
    lp_transformations ===> openvino
    frontend_common ===> openvino
    reference ===> openvino
    builders ===> openvino
    shape_inference ===> openvino
    core ===> openvino

    frontend_api-.->openvino_api
    inference_api-.->openvino_api
    core_api-.->openvino_api

    subgraph openvino_dev [openvino::runtime::dev]
        style openvino_dev fill:#af8401
        openvino_dev_api{{OpenVINO developer API}}
    end

    openvino ===> openvino_dev
    inference_dev_api-.->openvino_dev_api
    lp_transformations_api-.->openvino_dev_api
Loading

See also